ARA Institute of Canterbury, New Zealand
University of Canterbury, New Zealand
University of Canterbury, New Zealand
Education Contractor, New Zealand
Nesbit, T., O'Steen, B., Bell, T., & Martin, A. (2016). The journey from texting to applications on personally owned devices to enhance student eEngagement in large lectures: A pilot study. Journal of Applied Computing and Information Technology, 20(1). Retrieved September 22, 2020 from http://www.citrenz.ac.nz/jacit/JACIT2001/2016Nesbit_Engagement.html
Increasing class sizes to gain economies of scale have resulted in less interaction between lecturers and students during lectures. This paper presented the results of a pilot study that set out to examine the use of applications on personally owned devices (APODs) to enhance student interaction, participation and engagement in large lectures. The pilot study commences with the development and trial of a text messaging based application, and after a survey of students regarding ownership levels of mobile devices, concludes with the trial of an application developed for mobile devices. The conclusions of the paper highlight that the use of APODs can significantly increase student interaction, participation and engagement in large lectures and identifies implications and opportunities for further research.
Student engagement, large lectures, mobile devices, APODs
The pressure of increasing class sizes to gain financial economies of scale is seen by some as reducing the quality of the educational outcomes for students, with some of this reduction in quality focussing on the increasing lack of engagement and feedback between students and teachers/lecturers as class sizes increase. One aspect of engagement that diminishes as class size increases is the use of small group discussion around a topic, with one member of each group explaining their findings to the rest of the class with this then allowing the teacher/lecturer to give immediate feedback to the rest of the class.
Much of the literature surrounding the introduction of technology into the learning process has focussed on engaging students outside of the traditional face-face learning environment whether in a full eLearning or pure eLearning sense, or as a supplement to face-face delivery. There has been some attention in the literature paid to the use of technologies such as "clickers", audience response systems (ARS) and the like where technologies such as mobile phones and other wirelessly connected devices have been used with the explicit aim of enhancing student engagement.
This paper presents the results of a four phase pilot study at the University of Canterbury in Christchurch, New Zealand that investigates the use of applications on personally owned devices (APODs) to increase student engagement in large lectures. For the purposes of this pilot study the term Applications on Personally Owned Devices (APODs) has been coined to describe text messaging based systems (such as the one developed in Phase A and used in Phase B, see Table 1) as well as applications running on smart phones, tablets and laptops (such as the one used in Phase D).
The four phases of the pilot study are shown in Table 1.
The literature relating the use of ARS generally informs the study as does the literature on the use of APODs specifically, with this relationship being shown in Figure 1. For the purposes of this study the APODs being investigated are those that can be used as ARS, and as such exclude applications that cannot be used as ARS.
The research method for each phase is described with the results of each phase being presented along with the analysis and discussion sections and conclusions section including directions in which the research can be extended.
The development of a text messaging based system is briefly described. The outcomes of four experiments using the system that were conducted with a first year information systems course at the University of Canterbury with approximately 250 students enrolled are presented, along with implications for later phases of the research.
Students were asked to send a text message containing their name and favourite colour as an initial test to check that the system captured the messages so that they could be displayed on the screen at the front of the class.
Students were asked to discuss in small groups how many entities (between 1 and 6) they thought would be in the entity relationship diagram based on small narrative and to have one student in each group text the number to the system. A graph showing the responses was displayed on the screen at the front of the class.
Students were given another small example to discuss, with this including the need for a bridging entity because of the existence of a many-many relationship. The question asked whether there were 3 or 4 entities and why. One person from each group was asked to text their answer with the lecturer displaying a cross section of the answers.
Students were asked to discuss why some people choose to not purchase things online with one person from each group texting their answer after which the lecturer displayed a cross section of the answers.
Phase B involved surveying the students in a second year accounting information systems course at the University of Canterbury with approximately 170 students enrolled where the system was used in three different ways:
In the survey the students were asked to indicate how often they would be likely to participate in each of the manners of participation shown in Table 2 noting that the contents of the first column are verbal interactions and the second column is based on using the system.
The results of this phase of the study are presented and analysed and the implications for future research are identified.
In the conclusions to Phase B of the study it was identified that a growing number of students owned smart phones, tablets, and/or laptops and that there were some technical issues relating to the use of the texting based system that had been developed. To determine whether the research could move from the texting based system to applications based on devices such as smart phones it was decided to survey students at the University of Canterbury with the aim of measuring the changing patterns of device ownership.
The students were asked which of the devices shown in Table 3 they personally owned at the end of 2010, 2011 and 2012.
The responses where collated and analysed to show the changing patterns and the importance of the analysis for following phases of the research.
An application developed for mobile devices that allows a similar level of interaction between students and lecturers was used during a large first year commerce course at the University of Canterbury in 2013. In a survey at the end of the course students were asked to indicate how often they would be likely to participate in each of the manners of participation shown in Table 4 (based on the questions asked in Phase B.
The literature relating the use of technologies to increase student interaction has most of its origins in the use of clicker technologies. Some of the literature then moved on to the use of texting technologies using mobile phones and more recently to the use of applications on smart phones and tablets (APODs) as they became more widely used. As indicated in the introduction (see Figure 1) the literature on clicker based technologies, while not directly related to this study, can inform aspects of this study.
In a trial that was reported on (Freeman & Blayney, 2005) the same pedagogical approach of small group discussion was used irrespective of whether the feedback mechanism was by show of hands or the handheld keypads (commonly called clickers). This enabled a comparison to be made between the two different feedback mechanisms using the same learning pedagogy. Students involved perceived some advantage when using the handheld keypads when it came to their interaction and understanding. The concept of anonymity was looked at as being one of the driving factors behind this perception, and that this may encourage students who are unsure about their response to engage and interact more than they might with a more public display of a show of hands. It was identified that requiring students to respond to inclass questions during class time can assist learning, but that the use of such devices is a costly alternative to asking students to respond to a question by a show of hands, even though this allows for quick feedback in large classes (Freeman & Blayney, 2005).
In a case study of the use of clickers in an introductory management information systems course (Nelson & Hauck, 2008) it was found that use of classroom response systems like clickers significantly improved students' perception of their performance in the course and that the use of the clickers also met with higher rates of class attendance and of student performance. It was also found that the higher the use of the clickers, the higher the students' perceptions when it came to active learning, motivation and providing feedback.
In the literature review conducted (Nelson & Hauck, 2008) a number of factors that promote effective learning in the classroom were identified including active learning, providing feedback, increasing attention span and motivation, with all of these being seen as being particularly challenging in large lectures (Beatty, 2004) and for Net Generation learners (Robinson & Ritzko, 2006). The importance of feedback was also highlighted (Bangerrt-Downs, Kulik, Kulik, & Morgan, 1991) with the timing of the feedback being seen as especially important (Azevedo & Bernard, 1995; Kulik & Kulik, 1988).
The development and trial of a short-message-service (SMS) based classroom interaction system was trailed in a 2007 study (Scornavacca, Huff & Marshall, 2007). The theoretical background for this work included the use of large lecture theatres to produce effective and scalable approaches to teaching large classes, and how this approach can be used at the expense of student interaction which can result in reduced student engagement, motivation and learning as outlined in other studies (Freeman & Blayney, 2005).
The system developed was based around the assumption that a large proportion of students have SMS enabled mobile phones with them during lectures, and that these could be used to enable the lecturer to receive messages from students during a lecture. The system developed involved the use of a SMS management tool (SMS Studio) and was used in two different ways during lectures. Firstly it was used to allow students to send messages to the lecturer during the lecture that could be responded to during the lecture or afterwards (open channel mode). Secondly it was used to present students with a discussion topic to be discussed in small groups, and then respond to a multi-choice question based on the topic (m-quiz mode). Approximately one quarter of the students participating in open channel mode, and over half of the students responding in m-quiz mode (Scornavacca et al, 2007).
It was concluded that the additional channel of communication was of benefit to both students and lecturers compared with when there was no equivalent channel. There was a perceived increase in the quantity and quality of feedback from the students. It was also identified that an area for further research was related to any change in student learning and performance when using such a system, and whether this would be different across a range of contexts and subject area (Scornavacca et al, 2007). This development and trial provided the initial impetus for the development of the text messaging based system in Phase A of this pilot study.
An initiative analysed involved students being able to use applications on smart phones to participate in lectures (Calma, Webster, Petry & Pesina, 2014). In each lecture students were asked to answer five multiple choice questions using either a smart phone app or using pen and paper. Results indicated that students found the approach useful irrespective of whether they used the app or pen and paper. One of the most useful factors was the immediate feedback that was only really possible when students used the app. The implications highlighted in this study are that there is a significant potential to increase student engagement following this approach.
The study completed by Kay & Le Sage (2009) identified six threads in the literature relating to the use of audience response systems with these being classroom environment benefits, learning benefits, assessment benefits, technology based challenges, teacher (or lecturer) based challenges and student based challenges. Two additional threads were identified with these being pedagogical issues (Flies & Marshall, 2006; Beatty, Gerace, Leonard & Dufresne, 2006; Blood & Gluchak, 2013; Brady, Seli & Rosenthal, 2013; Wolter, Lundeberg, Kang & Herreid, 2011; Camacho-Minano & del Campo, 2014; Stewart & Stewart, 2013; Latham & Hill, 2014) and the cost and simplicity of devices (Freeman & Blayney, 2005; Scornavacca, Huff & Marshall, 2007; Chen & Lan, 2013; Blood & Gulchak, 2013).
A summary of the eight threads mentioned above is shown in Table 2, with the aspects under each thread that have been added to or renamed from the study completed by Kay & Le Sage (2009) being marked with an asterisk.
The system allows for text messages to be sent to a mobile phone that is plugged into the USB port of a computer. The computer sees the mobile phone as being a GSM modem and when a text message arrives on the phone an SQL command is executed to insert the contents of the message into an Access database.
The lecturer can select which responses to display to the class as shown in Figure 2 and then display the selected responses on the screen at the front of the class as shown in Figure 3.
It is possible to have more than one mobile phone plugged in to different USB ports and have the system recognise all of the phones and process text messages from any of them at the same time. This allows students a choice of which mobile network to text and would enable more students to participate at no cost.
The purpose of the experiment was to test whether the system was working, and within 2 minutes 148 responses had been inserted into the Access database with some being selected and displayed. The result of this experiment was successful.
This experiment was in essence a multi-choice question with 3 being the correct answer. The graph generated by the system is show in Figure 4 indicating that the majority of the students were correct. The lecturer went on to explain why 3 was the correct answer.
Of the 76 responses there were 20 that included more than just the number of entities and included a reason why with these 20 responses being displayed for the class to see. The correct answer was 4 due to the need for a bridging entity because of a many-many relationship. The lecturer was able to explain why 4 was the correct answer and highlight the valid reasons for this. One response was "3 because James said" and resulted in a light hearted moment. Some students were heard to say "oh, I get it now" when they saw the responses that were displayed and the reasons associated with the correct answer.
This experiment resulted in 48 responses being received with two of them being "U r going to fast [sic]" and "Speak a liltle louder cnt hear at the back [sic]". Of the other 46 responses, 13 were shared with the class. These responses (see Figure 5) indicated that the students collectively had a good understanding and that was able to be built on.
Observations of the lecturers included that many more students would participate by sending text messages than talking out loud. It was also observed by one of the lecturers in the weeks following these experiments that students appeared to interact more verbally at that stage of the course than they had in the previous semesters for that part of the course. This could point to the idea of students sharing anonymously and not being judged as being an approach to increasing the trust, willingness and confidence to share verbally.
At the end of the lecture where experiment #4 was conducted, the students were asked to text in their perceptions of the system. There were 7 responses sent in, which is too small a sample size to make generalisations about, however these responses were all positive and are shown in Table 6.
Of the 170 students enrolled in this course, 63 of the students responded to the survey for a response rate of 37.1%.
The comparison of how willing the students are to interrupt the lecturer to ask questions and how willing they are to text questions to the lecturer is shown in Table 7 and indicates a significant increase in willingness to text questions than interrupt the lecturer to ask verbally.
The comparison of how willing the students are to answer a question asked by the lecturer and how willing they are to text answers to questions asked by the lecturer is shown in Table 8 and indicates a significant increase in willingness to answer questions by sending a text message as opposed to answering verbally.
The comparison of how willing the students are to tell the rest of the class what was talked about in their small group discussion, and how willing they are to text what their small group had talked about is shown in Table 9. This indicates a significant increase in willingness to send a text message to share the group's response than to share it verbally.
The comparison of how willing the students are to ask questions in a test/exam review session and how willing they are to text questions during a text/exam review session is shown in Table 10. This indicates a significant increase in willingness to send text messages to ask questions about a test/exam than to ask the questions verbally.
Of the 63 students who responded to the survey, 44 (68.9%) responded to the first question about how they felt about the use of the system and 57 (90.5%) responded to the second question about the future use of the system giving a total of 101 open ended responses about the use of the system. 42 (66.7%) of the students responded to both questions, with 59 (93.7%) responding to at least one of the two questions.
The analysis of these responses is presented in the Analysis and Discussion section of this paper.
The students were asked which of the devices shown in Table 11 they personally owned at the end of 2010, 2011 and 2012, with the numbers of responses for each device also being shown. The data shows significant increases in the ownership of the smart phones, tablets and laptops.
Of the 380 students enrolled in this course, 55 of the students responded to the survey with valid responses for a response rate of 14.5%.
The comparison of how willing the students are to interrupt the lecturer to ask questions and how willing they are to use the application to ask questions of the lecturer is shown in Table 12.
The comparison of how willing the students are to answer a question asked by the lecturer and how willing they are to use the application to send answers to questions asked by the lecturer is shown in Table 13.
The comparison of how willing the students are to tell the rest of the class what was talked about in their small group discussion and how willing they are to send what their small group had talked about while using about the Application is shown in Table 14.
In Phase A of the research there was clear evidence of a number of the threads and aspects from the literature review (see Table 5). These include what was observed to be increased attendance, participation, engagement and enjoyment of learning relating to classroom environment benefits. There was also increased interaction and discussion, and the scope for approaches such as contingent teaching and question driven instruction from the learning benefits thread (Kay & LeSage, 2009). When it came to the assessment benefits thread there were aspects of formative assessment, feedback, and the ability to compare responses present.
In the technology based challenges thread, the issue of not all students having a device with them was addressed through the students working in small groups, and the issue of the technology not functioning was addressed by two lecturers being present as the system was seen as being somewhat "clunky". Issues related to lecture based challenges were not experienced significantly. When it came to the student based challenges, students appeared to have little problem adapting to using the system as there was a high degree of familiarity with texting.
When it came to pedagogical issues, it appeared from the lecturers' perspective that the approach was a good teaching strategy in that it addressed some of the issues relating to large classes, and had the potential for being used in a contingent teaching and question driven instruction context (Kay & LeSage, 2009; Beatty et al, 2006; Brady et al, 2013; Camacho-Minano & del Campo, 2014; Flies & Marshall, 2006; Latham & Hill, 2014; Stewart & Stewart, 2013; Wolter et al, 2014).
When it came to cost and ease of use for students, there appeared to be few issues, however at the lecturer end the system was seen as being "clunky", and there had been a reasonable amount of time spent developing the system to this point that had been funded by a teaching grant.
The findings from this phase of the pilot study point to some of the issues in the literature relating to the use of ARS have been addressed particularly when it comes to the cost and simplicity of devices (Freeman & Blayney, 2005; Scornavacca et al, 2007; Chen & Lan, 2013; Blood & Gluchak, 2013).
The results shown in Table 7 demonstrate that a much higher proportion of students would use the text messaging system to ask questions (44 out of 63 occasionally or often) as opposed to interrupting the lecturer to ask questions (3 out of 63 occasionally or often).
The results shown in Table 8 demonstrate that a much higher proportion of students would use the text messaging system to answer questions (39 out of 63 occasionally or often) as opposed to verbally answering a question (17 out of 63 occasionally or often).
The results shown in Table 9 demonstrate that a much higher proportion of students would use the text messaging system to share the outcome of their small group discussion (34 out of 63 occasionally or often).
The results shown in Table 10 demonstrate that a much higher proportion of students would use the text messaging system to ask questions during a revision session (29 out of 63 occasionally or often) as opposed to doing so verbally (7 out of 63 occasionally or often).
This analysis is consistent with a number aspects from the classroom environment benefits thread from the literature review (see Table 5). Of particular relevance was that there was clear evidence that the use of the text messaging system resulted in increased student engagement, interaction and participation (Kay & LeSage, 2009; Freeman & Blayney, 2005; Nelson & Hauck, 2008), with these all being aspects of the classroom environment benefits thread, irrespective of the mode that the system was used in.
The perception from the lecturers that the system was somewhat "clunky" for them to use remained an issue, with this relating to the ease of use for lecturers aspect of the cost and simplicity of devices thread from the literature and the technology not functioning aspect of the technology based challenges thread (see Table 5) which was predominantly based on the model developed in Kay & LeSage (2009).
There was a growing awareness that significant numbers of students were owning smartphones and tablets and that some applications were becoming available that would offer similar functionality to the text messaging system without being quite as "clunky" for the lecturers to use. While this would address the technology based challenges aspect and the ease of use for lecturers aspects (Kay & LeSage, 2009), it would bring in to question whether sufficient students owned a device capable of running the application which is consistent with the students not having or bringing a device aspect of the technology based challenges thread from the literature (Kay & LesSage, 2009) and the cost to students aspect of the costs and simplicity of devices thread (Freeman & Blayney, 2005; Scornavacca et al, 2007; Chen & Lan, 2013; Blood & Gluchak, 2013).
Whether sufficient students owned smart phones or tablets was something that would need to be tested. One issue that had been overcome for lecturers relating to the use of clickers was no longer needing to have a process to distribute clickers to students which relates to the technology based challenges of the literature review (Kay & LeSage, 2009).
The results of the survey regarding device ownership in Table 11 were further analysed to determine how many of the students owned at least one of a smart phone, tablet or laptop with this being shown in Table 15. This shows the percentage of students owning smart phones growing from 19.0% to 72.2% and the percentage owning any mobile web enabled device growing from 73.0% to 96.6% from the end of 2010 to the end of 2012, with this 96.6% figure only being marginally behind the 97.5% of students owning a mobile phone that was capable of texting.
A consequence of this was that if there were suitable applications available for mobile web enabled devices that were freely available that it would be possible to use them instead of the text messaging based system. This connects to the aspect of students not having or bringing a device in the technology based challenges thread of the literature and to the costs of devices aspect in the cost and simplicity of devices thread from the literature (Freeman & Blayney, 2005; Scornavacca et al, 2007; Chen & Lan, 2013; Blood & Gluchak, 2013).
The results shown in Table 12 demonstrate that a much higher proportion of students would use the application to ask questions (20 out of 55 often or very often) as opposed to interrupting the lecturer to ask questions (3 out of 55 often or very often).
The results shown in Table 13 demonstrate that a much higher proportion of students would use the application to answer a question (38 out of 55 often or very often) as opposed to verbally answering questions (1 out of 55 often or very often).
The results shown in Table 14 demonstrate that a much higher proportion of students would use the application to share the outcome of their small group discussion (21 out of 55 often or very often) as opposed to sharing the outcome verbally (9 out of 55 often or very often).
This analysis demonstrates that irrespective of which mode the application was used in, that a much higher proportion of students will use it to engage, interact and participate during lectures in comparison to doing so verbally with this being consistent with much of the literature.
This analysis is consistent with a number of aspects from the classroom environment benefits thread from the literature review (Kay & LeSage, 2009). Of particular relevance was that there was clear evidence that the use of the application resulted in increased student engagement, interaction and participation (Freeman & Blayney, 2005; Scornavacca et al, 2007; Chen & Lan, 2013; Blood & Gluchak, 2013), which are all aspects of the classroom environment benefits thread, irrespective of the mode that the application was used in.
Phase A and Phase B of the pilot study showed that the concept of using a text messaging based APOD would allow students to respond anonymously during large lectures and demonstrated some of the key aspects of the literature including: increased interaction, participation and engagement; low cost and simplicity of use for students; and without the need to find a process for distributing devices to students.
Changing the focus of the pilot study from the text messaging system (Phase A and Phase B) to one based on an application running on a mobile web enabled device (Phase D) was partly in response to the ease of use aspect for lecturers. This decision was only made after consideration of the challenges of not all students having or bringing a device and the issue of cost of devices for students. This decision was enabled by the results of the survey regarding ownership of devices (Phase C).
Phase D of the pilot study demonstrated that the classroom environment benefit of increasing student interaction, participation and engagement that was experienced when using the text messaging based system (Phase B) was also present when using an application based on a mobile web enabled device.
Based on the findings of this pilot study the use of APODs in large lectures can result in increased student interaction, participation and engagement.
While the results of the pilot study are successful, there are a number of implications and opportunities for further research including:
These issues need to be explored from a range of perspectives including lecturers (through interviews), students (through surveys and focus group) and those supporting lecturers in the adoption of APODs (through interviews of learning advisers).
Azevedo, R., & Bernard, R. M. (1995). A meta-analysis of the effects of feedback in computer-based instruction. Journal of Educational Computing Research, 13(2), 111-127.
Bangert-Downs, R. L., Kulik, C. C., Kulik, J. A., & Morgan, M. T. (1991). The instructional effect of feedback in test-like events. Review of Educational Research, 61(2), 213-238.
Beatty, I. D. (2004). Transforming student learning with classroom communication systems. Educause Research Bulletin, 3, 2-13.
Beatty, I. D., Gerace, W. J., Leonard, W. J., & Dufresne, R. J. (2006). Designing Effective Questions for Classroom Response System Teaching. American Journal of Physics, 74. 31-39.
Blood, E. & Gulchak, D. (2013). Embedding "clickers" into classroom instruction: benefits and strategies. Intervention in School and Clinic, 48(4). 246-253.
Brady, M., Seli, H. & Rosenthal, J. (2013). "Clickers" at metacognition: A quasi comparative study about metacognitive self-regulation and use of electronic feedback devices. Computers and Education, 65. 56-63
Calma, A., Webster, B., Petry, S. & Pesina, J. (2014). Improving the quality of student experience in large lectures using quick polls. Australian Journal of Adult Learning. 54(1), 114-136.
Camacho-Minano, M. & del Campo, C. (2014). Useful interactive teaching tool: clickers in higher education. Interactive Learning Environments. 1-18
Chen, T. & Lan, Y. (2013). Using a personal response system as an in-class assessment tool in the teaching of basic college chemistry. Australasian Journal of Educational Technology, 29(1). 32-40.
Flies, C., & Marshall, J. (2006). Classroom Response Systems: A Review of the Literature. Journal of Science Education and Technology, 15(1), 101-109.
Freeman, M., & Blayney, P. (2005). Promoting Interactive In-class Learning Environments: A Comparison of an Electronic Response System with a Traditional Alternative. Proceedings of the 11th Australasian Teaching Economics Conference, 23-34.
Kay, R. & LeSage, A. (2009). Examining the benefits and challenges of using audience response systems: A review of the literature. Computers & Education, 53. 819-827.
Kulik, J. A., & Kulik, C. C. (1988). Timing of feedback and verbal learning. Review of Educational Research, 58(1), 79-97.
Latham, A. & Hill, N.S. (2014). Preference for anonymous classroom participation: Linking student characteristics and reactions to electronic response systems. Journal of Management Education, 38(2). 192-215.
Nelson, M. L., & Hauck, R. V. (2008). Clicking to Learn: A Case Study of Embedding Radio Frequency Based Clickers in an Introductory Management Information Systems Course. Journal of Information Systems Education, 19(1), 55-64.
Robinson, S., & Ritzko, J. (2006). Increasing Student Engagement Through Electronic Response Devices. Paper presented at the Allied Academies International Conference, Proceedings of the Academy of Educational Leadership.
Scornavacca, E., Huff, S.L. and Marshall, S. (2007). Developing A SMS-Based Classroom Interaction System. Proceedings of the Conference on Mobile Learning Technologies and Applications, 47-54.
Stewart, S. & Stewart, W. (2013). Taking clickers to the next level: a contingent teaching model. International Journal of Mathematical Education in Science and Technology, 44(8). 1093-1106.
Wolter, B., Lundeberg, M. Kang, H. & Herreid, C. (2011). Students' perceptions of using personal response systems ("clickers") with cases in science. Journal of College Science Teaching, 40(4). 14-19.