Student Preclass Preparation by Both Reading the Textbook and Watching Videos Online Improves Exam Performance in a Partially Flipped Course

INTRODUCTION

In the typical college classroom, the instructor presents material by lecturing, and students apply their knowledge on homework assignments. In the “flipped classroom,” students acquire content outside class, often in the form of video lectures, and do problem solving in the classroom. The flipped classroom is increasingly being used in both K–12 and higher education to engage students in high-level learning during class, when they can get assistance from the instructor and peers (DeLozier and Rhodes, 2017). Several studies on flipped classrooms in science, technology, engineering, and mathematics fields such as calculus, pharmacy, statistics, nursing, chemistry, and engineering reported significantly higher scores on exams or relevant exam questions compared with nonflipped controls (Pierce and Fox, 2012; Mason et al., 2013; Missildine et al., 2013; Wilson, 2013; McLaughlin et al., 2014; Gross et al., 2015; Sahin et al., 2015; Schroeder et al., 2015). These studies also reported additional positive results such as increased attendance and improved performance in a subsequent course (McLaughlin et al., 2014; Schroeder et al., 2015). However, when a flipped class is compared with a lecture class, there may be several different variables involved, and determining which specific changes in course structure or content delivery lead to positive effects is challenging.

Recently, several studies have tried to determine which aspects of the flipped classroom lead to increases in student performance. Jensen et al. (2015) asked whether the instructor presence during the acquisition of knowledge (nonflipped model) was different from the instructor presence during the application of knowledge (flipped model) when both classrooms were active-learning environments. They found no difference between the two active-learning classrooms, contrary to the idea that flipping helps students attain deep conceptual learning because the instructor is present to help during difficult problem-solving activities (Jensen et al., 2015). Another study examining an upper-level biochemistry course with a flipped format found that the increase in exam performance in the flipped course was partly due to students interacting with course material more often and not just immediately before an exam (Gross et al., 2015). Flipping may have other benefits, such as allowing students to practice time management, gain independent learning skills, and develop critical-thinking skills (van Vliet et al., 2015; McLean et al., 2016).

One variable that could affect increased performance of students in flipped courses is the use of online videos to deliver content rather than the assigned textbook reading typical in a lecture course. Several recent student surveys reported low completion of reading assignments by college students in nonflipped classes. One survey of primarily education majors found that only 25% read the assigned textbook before class, and 62% spent an hour or less on average reading assignments (Baier et al., 2011). A survey of students in finance courses found that ∼18% did not read the textbook at all, while 43% spent less than 1 hour per week reading the textbook (Berry et al., 2010). A survey of summer students at a small college found higher rates of reading, with 52% of students reporting reading the assigned textbook (Aagaard et al., 2014). This student survey indicated instructor behaviors most likely to motivate students to read the textbook before class are in-class quizzes, study guides for credit, and not discussing material in class but testing on it (Aagaard et al., 2014). Use of low-stakes, formative assessments to incentivize reading can greatly increase student use of the textbook before class. Quizzes on the reading assignment, both surprise quizzes in class and online reading quizzes, have been shown to be effective (Sappington et al., 2002; Dobson, 2008). Online quizzes led to 80% of physics and biology students reading before class and a positive correlation between number of completed quizzes and final exam score (Heiner et al., 2014).

In order for the flipped classroom to be effective, students must engage with the material before coming to class. Very few studies have directly addressed the question of whether or not students view online videos. Of students in a flipped calculus course, 80% reported that they watched all assigned videos. These students were given a one- or two-question quiz at the beginning of each flipped class (Schroeder et al., 2015). As little as 1% of the student’s total grade in introductory biology for uploading notes taken on videos before class led to 90% participation and increased performance on related exam questions (Moravec et al., 2010). Therefore, similar to assigned textbook reading, incentives may be necessary to attain student compliance with video viewing, even though students report a preference for watching videos to reading the textbook (Sahin et al., 2015).

In a partially flipped cell biology course, students were assigned reading in the textbook 2 days a week before lectures and video lectures to watch before the flipped day. For both assignments, there was a graded assessment to motivate students to complete the work before class. On lecture days, class began with a five-question clicker quiz on the content of the assigned reading. On flipped days, there was an online quiz that covered video content to complete before class. Both clicker and online quiz questions were on material introduced to the student in the assigned reading or videos. The primary questions addressed in this study were: How much of the assigned reading or video viewing did students complete? Did students watch videos more than they read the textbook? Did watching the videos or reading the textbook affect exam performance? Additionally, a survey was given to investigate student attitudes about the partial classroom flip, different aspects of the videos, and the in-class problem set.

METHODS

This study was conducted with approval from the Missouri S&T Institutional Review Board.

Student Demographics

This research was conducted over four semesters in an introductory cell biology course, typically taken by sophomores. The course is required for biological sciences majors, as well as chemical engineering majors with a biochemical emphasis. The school is primarily an engineering school in the rural Midwest, and the student population is majority white (77% white undergraduates in 2015). Although the university itself has predominantly male undergraduates (77%), biological sciences has 62% female undergraduate majors. Average grade point average (GPA) for students in cell biology ranged from 3.1 to 3.3 across years (Table 1). Data were included for all students who completed the course; students who withdrew during the semester were removed from the data set.

Collection of Data on Student Reading and Study Habits

Each semester, four exams were given, and at the end of each exam were optional multiple-choice extra-credit questions asking about the students’ reading and studying habits. The questions were preceded by a consent statement. The question used to generate the “reading score” was:

For the material on this exam, I read the assigned pages in the textbook before class

• A. Always

• B. Usually (only missed one or two days total)

• C. Sometimes (did not read one assignment each week)

• D. Rarely (only read once or twice total)

• E. Never

For data analysis, answer A was assigned a 4, answer B was given 3, choice C was 2, D was 1, and E was scored as 0 and treated as ordinal data. For the response “usually,” this would represent students completing between 66% and 83% of the assigned reading, because they reported missing only 1 or 2 days out of about 6 days total. “Sometimes” represents ∼50% of reading assignments completed, because students did not do one of the two readings per week. “Rarely” would be estimated to be completion of only 16–33% of the readings.

The question used to generate the “study score” was:

How did you prepare for this exam, and how did you feel about your efforts before you took the exam?

• A. I spent several days reviewing the material and I felt well prepared

• B. I spent several days reviewing the material but I felt unprepared

• C. I spent two days reviewing the material and I felt well prepared

• D. I spent two days reviewing the material but I felt unprepared

• E. I studied last night/this morning and I felt well prepared

• F. I studied last night/this morning but I felt unprepared

• G. I did not study

For data analysis, answers A and B were scored as 3, C and D as 2, E and F as 1, and answer G was given a 0 and treated as ordinal data. Feeling prepared or unprepared for the exam was not analyzed in this study, although student confidence may be an important factor in exam performance.

Collection of Video-Viewing Data

Videos were posted on Blackboard, and analytics on video viewing were captured and reported using Kaltura. The “video-viewing score” was calculated by adding up total minutes of video watched by the student before each exam and determining the percentage of total number of minutes of video assigned. For some analyses (Figures 1C and 2), it is helpful to bin the video-viewing scores into groups that roughly correlate with reading scores 0–4. To accomplish this, we binned video viewing into 0–15%, 16–35%, 36–65%, 66–99%, or ≥100% (Figure 1C). For Figure 2, 0–15% was given a score of 0, 16–35% a score of 1, 36–65% a score of 2, 66–99% a score of 3, and 100% or more a score of 4.

Preclass Preparation Assessments

For two of the three class periods each week, students were assigned 10–20 pages to read from the textbook. At the beginning of each class, before any further instruction, students were given a five-question quiz on material from the reading using personal response software (clickers). Clicker points made up 5% of the course grade.

Videos for flipped Fridays typically consisted of three instructor-made videos of voice-over–annotated PowerPoint slides, each video typically being between 8 and 13 min in length. Videos were posted on Blackboard at least 3 days before Friday’s class, and students were required to complete a short online quiz (12–15 questions, true or false and multiple choice) before class that specifically covered material students had only been exposed to by watching the videos. The flipped Friday quiz grade was 10% of the grade in the course. The lowest quiz grade was dropped.

Data Analysis

The significance level used throughout all analyses was at the 0.05 level.

To determine the reliability of the self-reported reading scores, we performed a one-way analysis of variance (ANOVA) using the median of the self-reported reading scores rounded up to the nearest integer on the four exams as the categorical factor and total clicker points for the semester on reading clicker quizzes as the response variable using JMP Pro software. Tukey-Kramer pairwise comparisons were also performed to determine which of the median reading score groups differed significantly from each other in average total clicker points. To determine whether there was a significant difference between the reading scores on exam 1 and exam 4, we performed a Wilcoxon signed-rank test for paired data with a two-tailed hypothesis using an online calculator (www.socscistatistics.com/tests/signedranks/default2.aspx). To determine whether there was a significant difference between video viewing at exam 1 compared with exam 4, we performed a two-tailed paired t test using Excel. Analyses of reading and video watching are represented with bar graphs generated in Excel.

For examining the relationship of the reading, studying, and video-viewing scores together in a single model, a multiple linear regression model (MLR) was fit using JMP Pro software. This analysis allowed us to include GPA as a covariate and separate out effects of different exams all within the same model framework. Additionally, because each student completed four exams, a random effect was included to account for the fact that there are multiple measurements per student included in the analysis. Semester was also included as a random effect, because of the unreplicable combination of students in a class during a particular semester. In the MLR model, exam grade was the response variable, and the predictor variables were exam number (nominal data), student number and semester number (random effects), study score and reading score (ordinal data), and GPA and video views (continuous data).

For investigating the predictive ability of the model, data from Fall 2014 and Spring 2015 were used to fit the model (training set), and the exam score predictions were made on the Fall 2015 and Spring 2016 data (test set). Only variables identified as significant in the MLR (GPA, exam number, reading score, and percent video viewed) were used to generate a predictive formula. Student and semester random effects were also included in the model-building process. A correlation analysis was then performed between the model predictions and actual exam scores for the Fall 2015 and Spring 2016 test semesters.

For examining the pattern of reading and video watching by students, a heat-map graph of reading score and video views was generated in JMP Pro using Graph Builder Heatmap. To facilitate comparison to the reading score, we grouped the percent of videos viewed by the student into categories 0–4 as described in Collection of Video-Viewing Data. Student preclass preparation by reading the textbook and watching videos before class for each exam was plotted. Therefore, the N in each heat block is the percentage of student responses with that combination of reading and video scores out of the total, so that each student is represented four times in the heat map.

Attitude Surveys

To get feedback on aspects of the videos and classroom activities done on the flipped day, students were given a 15-question five-point Likert-scale paper survey and given the option to complete it anonymously in class. No extra credit or other incentive was provided. The survey was conducted near the end of the semester in Fall 2014, Spring 2015, and Fall 2015. No survey was conducted in Spring 2016. Results of the attitude surveys are represented with bar graphs generated in Excel.

RESULTS

Self-Reported Reading Scores Decrease during the Semester and Are Related to Reading Quiz Points

Previous studies showed that students rarely read assigned pages in the textbook unless summative or formative assessments are used. To determine whether a “reading quiz” using clickers was effective in motivating students to complete the reading, we surveyed students on their reading compliance. Although self-reporting is the only way to measure how often students read, there was a possibility that students would overestimate or not answer the survey question honestly because these responses were not anonymous. To validate the reliability of the responses, we totaled the clicker points from the reading quizzes for the semester, and a one-way ANOVA was conducted to see whether this total related to the student’s median reading score group for the four exams. The analysis showed that there were statistically significant differences in the mean clicker points obtained by students among the different self-reported reading score groups (F(4,171) = 10.08, p < 0.001; Figure 1A). Means for the one-way ANOVA and results of the Tukey post hoc test are shown in Table 2. Students with reading scores of 0–2 had the lowest average clicker points and were not significantly different from each other. Students with reading scores of 3 or 4 had higher average clicker points, and students who reported “always” doing the assigned reading had significantly higher clicker points than all other groups (Table 2). This relationship provided some assurance that the students were honest in their answers about reading the textbook. We used the self-reported reading scores in our model, because we were interested in examining the effect of student preclass preparation. Clicker points earned by students on the reading quiz measure both attendance in class and preparation before class.

Analysis of students’ self-reported reading indicated good completion of reading assignments, and showed that students read less at the end of the semester than at the beginning (Figure 1B). For the first exam, 70% of students reported “always” or “usually” reading the assigned pages in the textbook, which dropped to 49% of students by the last exam. Correspondingly, students who reported reading “rarely” or “never” increased from 13% at the start of the semester to 31% by the end of the semester. The percentage of students who gave each answer for their reading on exams 1 and 4 is shown in Figure 1B. We used a Wilcoxon signed-rank test to determine whether the changes in student reading scores from the beginning to the end of the semester were significant. The decrease was found to be significant, z = −6.1422, p < 0.00001.

Student Video Viewing Is Initially High but Also Shows Declines during the Semester

To analyze how much of the assigned videos were actually watched by the students, we calculated the percent of assigned video watched by each student for each exam. Results show that most students watched at least 66% of the assigned videos, suggesting that the online quizzes are an effective motivator for preclass preparation by viewing videos (Figure 1C). Similar to results from reading, students watched more of the videos for exam 1 than for exam 4. For the first exam, 75% of students watched two-thirds or more of the assigned videos, but for the last exam, the percent of students who watched most of the videos had dropped to 60% (Figure 1C). There was a significant decrease in video viewing between exam 1 and exam 4 (p = 0.0001).

Students Usually Both Read and Watch Videos

Because previous research has shown that students rarely read the textbook before class, we hypothesized that more students would watch the videos than read the textbook. To visualize the patterns of students’ reading and video watching, we generated a heat map using a student’s reading score for an exam plotted on the y-axis and the student’s video-viewing score for the same exam on the x-axis (Figure 2). As there were four exams each semester, each student is represented in the graph four times. Most student preclass preparation, 64%, was by both reading and watching videos (both reading and video-viewing scores of 2 or higher; Figure 2). Surprisingly, only 15.9% of student preparation was by watching videos but not reading the textbook, meaning a video-viewing score of 2 or greater with reading score of less than 2, corresponding to the lower right region of the heat map. In fact, about an equal percentage of student preparation showed the opposite pattern, with 14.4% of responses having a high textbook reading score (2 or more) and a low video-watching score (less than 2). The smallest preclass preparation group was neither reading nor watching videos (both reading and video scores of less than 2), 5.5% (Figure 2). Overall, this analysis shows that, most of the time, students use both reading and videos to prepare for class, and it provides no evidence that students will only watch videos and not read the textbook when low-stakes assessments are conducted for both.

Multivariate Analysis Shows Significant Effects of Reading and Watching Videos on Exam Performance

To look at the relationship between multiple variables and exam score while controlling for GPA, we performed an MLR analysis with random subject and semester effects. Data from all four semesters were pooled and analyzed to generate a model to predict the exam score (Figure 3A). This model was able to predict student’s exam score with an R² of 0.71 (p < 0.0001), meaning 71% of the variation in exam scores was explained by the model. In this analysis, GPA had the biggest positive effect. Video views and reading score also had positive significant effects (Figure 3B and Table 3). Exam number was significant, but study score did not significantly affect student performance on the exam (Table 3). This analysis indicates that students who completed both types of preclass preparation—reading the textbook and watching online videos—were expected to have higher exam grades. In the MLR, these effects can be interpreted as being significant even after accounting for the GPA effect.

Model Predicts Exam Performance

For examination of the predictive ability of the model, data from the first two semesters (Fall 2014 and Spring 2015) were used to generate a predictive formula for exam scores. Only the significant variables of exam number, GPA, reading score, and video viewing were used in creating the predictive model, along with random student and semester effects. A correlation between the predictions made by this model and actual exam scores was obtained. For both subsequent semesters, the model fit with data from one academic year was able to predict student exam scores reasonably well with an R² = 0.307 in Fall 2015 (p < 0.0001) and R² = 0.266 in Spring 2016 (p < 0.0001; Figure 4, A and B). This result shows that the MLR model is predictive of student exam performance.

Student Attitudes about Flipping

To get feedback from students about the partial flip, students were given the option to anonymously complete a short five-scale Likert survey with questions about flipping, specific aspects of the videos, and the classroom problem set. Possible responses were strongly disagree to strongly agree. In Fall 2014, the response rate was 88%, and in Spring and Fall 2015, 74% of students completed the survey. These surveys revealed some resistance to flipping, which decreased somewhat over the three semesters the surveys were given. In response to the statement “I would like to have more classes that use flipping,” 49% of students disagreed or strongly disagreed in Fall 2014, but by Fall 2015, only 36% of students disagreed or strongly disagreed (Figure 5A). Students also mostly either disagreed or were neutral on the statement “I learned more from watching the videos than I do in class” (Figure 5B). Students liked controlling the video playback, with 73% or more of students agreeing or strongly agreeing “I liked the online videos because I could watch them at my own pace” (Figure 5C). Many students added written comments that they liked being able to pause or rewind the video lectures. Students also agreed or strongly agreed by 82% or more that “In class problems on Fridays were a good way to practice using the concepts”(Figure 5D). In Fall 2015 only, students were asked on the end of the semester survey to what extent they agreed with the statement “I prefer watching videos to reading the textbook.” Most students agreed (32%) or strongly agreed (29%), while 21% of students disagreed or 5% strongly disagreed, with 13% neutral.

DISCUSSION

Although there are many possible methods of introducing material outside the classroom, for some the flipped classroom by definition uses videos to deliver content (Bishop and Verleger, 2013). While instructor-produced videos have the advantage of being specific to the course, recording and editing the videos requires a large time commitment. There is some evidence that current college students, who are often Millennials or Generation Z, prefer to learn by watching videos (Roehl et al., 2013; Sahin et al., 2015; Seemiller and Grace, 2017). Therefore, we predicted that students would watch videos more than they read the textbook. However, the results did not show a large practice by students of watching videos without also reading (Figure 2). This was surprising, because reading was more frequently assigned (usually twice each week), while videos were assigned only once a week. However, it is true that students watched all assigned minutes of video more than they always read the textbook pages assigned both at the beginning and end of the semester (Figure 1, B and C). Also, when asked, students expressed a preference for watching videos.

Our results suggest that students can be motivated to engage with material by reading or watching videos and that a variety of learning materials may better engage students than use of only one medium. Student attitude surveys showed that students tended to either be neutral or disagree with the statement that they learned more from videos than they do in class (Figure 5B). Therefore, it may be an unfair assumption that “digital natives” strongly prefer learning by watching videos to other methods such as in-class lectures or reading. It also should be noted that our measurement of video viewing is a measure of accessing the videos online, and whether students paid attention to the videos as they played or not is not possible to determine.

The relatively high prevalence of students both reading the textbook and watching the videos may be partly due to the assessments used to motivate students to do both. Although we did not find as high as 80% of students reading before class (Heiner et al., 2014), students reported reading “usually” or “always” 70% of the time at the beginning of the semester and 49% of the time at the end of the semester. This result suggests that the low-stakes clicker quizzes were moderately effective at achieving student compliance with reading assignments. The instructor noted a positive effect on attendance, as previously reported (Freeman et al., 2007), and preventing tardiness, as the class began with the clicker questions. The online quizzes on the videos seemed to motivate most of the class (≥60%) to watch at least 66% of the minutes of video assigned, even at the end of the semester. Of course, one would like for all students to be doing all of the assignments, but perhaps this is not an achievable goal. Even though online quizzes on the videos were weighted more than the clicker quizzes on the reading, we did not see greater compliance with video watching than textbook reading. Our results suggest that when a requirement for before-class preparation was paired with a quiz, either in class using clickers or online before class, students were likely to comply, and we showed that this preparation positively impacted exam scores regardless of a student’s GPA. Although we did not use reading assignments in preparation for the flipped days, it seems unnecessary for instructors to solely use videos to prepare students for a flipped classroom. Assigning a reading relevant to the in-class activity should be effective as long as there is a quiz or other method of ensuring that students come to class prepared. We also found that both reading the textbook and watching videos were significantly higher at the beginning of the semester than at the end of the semester. This is likely due to several factors, such as students beginning the semester with higher levels of enthusiasm and suffering fatigue toward the end of the semester, as well as competing demands on their time throughout the semester, especially during finals week. Therefore, in future studies of student engagement with materials outside the classroom, the timing of measurement is important to note.

The surveys to gauge student attitudes showed some resistance to flipping, as has been previously reported (Missildine et al., 2013; Baker and Hill, 2017). This is likely due to the fact that students regarded watching the videos as “extra” work, as evidenced by frequent comments in student evaluations that this should be a 4 credit-hour course rather than a 3 credit-hour course. It is not clear why students would perceive 30–35 minutes of video as more onerous than reading 20 pages in the textbook, but perhaps taking notes while watching the videos or the pressure of taking the online quiz before class led to their view that the videos were more work. Students liked the ability to pause the videos and watch at their own pace (Figure 5C). They also liked the in-class problem solving, reporting that they thought problem solving was useful for conceptual learning (Figure 5D). Therefore, they responded positively to the active learning done in class, which was the motivation for the partial flip and which has been shown to increase student performance (Freeman et al., 2014).

Our MLR analysis showed that increased engagement with material outside class had a significant impact on exam scores. Both reading the textbook and watching the videos had a positive effect of a similar magnitude on student exam performance. This shows that interacting with the course materials before class, as well as low-stakes assessments on the content of the reading or videos, can lead to greater learning, which may be one reason that active learning leads to gains in exam performance (Gross et al., 2015). Although it is reasonable to assume that high-performing students will do assigned work and also do better on exams, our model shows a positive effect of preclass preparation even after controlling for GPA. This demonstrates that, even though GPA is the single biggest predictor of exam performance, both reading and video viewing can improve learning for all students. This analysis also found no effect of study time on exam scores, indicating that increased time studying did not have a positive impact on exam performance or that students are not good at self-assessing study effort.

It should be noted that this is an observational study, as students chose for themselves the amount of reading and video viewing they would complete; thus there is the potential for confounding variables. We controlled for one natural confounder, GPA, as discussed previously. However, there are obviously other variables in student characteristics and preparation that were not controlled for in this study, such as prior instruction in biology. Students in this course are a mix of biological sciences majors, most of whom have one or two previous semesters of college biology, and chemical engineering majors who have not had biology since high school. The class is also highly structured, with weekly homework and daily clicker questions, and includes active learning in lectures with frequent use of think–pair–share, both of which have been shown to improve student attendance and performance (Freeman et al., 2007; Haak et al., 2011). The MLR model based on variables from the first two semesters provided reasonably accurate predictions of student performance in the next two semesters (Figure 4). This analysis confirms the importance of the increased engagement with course materials outside class on students’ exam scores when the classroom environment is a mix of lectures and active learning.

ACKNOWLEDGMENTS

REFERENCES