Flipped-Class Pedagogy Enhances Student Metacognition and Collaborative-Learning Strategies in Higher Education But Effect Does Not Persist

INTRODUCTION

Large lectures in academia have not changed much in centuries. Calls for change have been made in areas such as science, technology, engineering, and mathematics (STEM; President’s Council of Advisors on Science and Technology, 2012) and medical education (Frenk et al., 2010). Contents of lectures have taken a huge leap, but educational approaches used by the lecturers have not changed much. Despite years of research into active learning and the resulting evidence (Prince, 2004; Michael, 2006; Freeman et al., 2014), the dominant model of lecturing is still a model of “show and tell,” with students as passive recipients of information. Recently, flipped-class (also named flipped-classroom or flipped-lecture) pedagogy has become very popular. During a flipped class, students prepare themselves at home before lectures, often by watching short video clips of the course contents. This saves time during the lecture that can be spent on discussion of the concepts with students. These discussions are focused on improving understanding of the course core issues and sometimes on controversial topics. Table 1 gives an overview of learning activities during a flipped class as compared with a traditional class.

A Google search on “flipped class” gives more than 24 million hits. However, the effects of flipped-class pedagogy on learning are much less reported in higher education. A survey of research by Bishop and Verleger (2013) gives an overview of 24 studies that investigated the effects of the flipped classroom. In these studies, there was only one (Papadopoulos and Santiago-Román, 2010) studying the effects of employing a partial flipped classroom using a matched (within the same group of students) pre- and posttest design. This study showed a gain in student learning in favor of flipped-classroom pedagogy; however, it was not investigated whether the effects of flipped class were long-lasting.

Research shows that increasing interaction with students supports active learning, and thus improves knowledge retention (Hake, 1998; Prince, 2004; Michael, 2006; Freeman et al., 2014; Thloale et al., 2014). Most notable is the concept of peer instruction (Mazur, 1997). During lectures, thought-provoking conceptual questions are discussed in student groups. Through these discussions, interactive engagement of students increases and academic achievement improves (Hake, 1998; Crouch and Mazur, 2001; Thloale et al., 2014). Tune et al. (2013) have shown that flipped classes can improve student performance, since students in these classes had higher exam scores compared with students who did not participate in flipped classes.

Several technological advances are making it easier for lecturers to apply the flipped-class concept. Web lectures and short explanation videos are now available online, making this technology more common in academia (Gorissen, 2013). Web lectures can have the same (limited) effects as live lectures (Wieling and Hofman, 2010), so making long, passive lectures available does not seem to be the answer. Instead, recording short videos with online tools has become more popular. Online learning environments allow lecturers to process student input before a lecture and shape the lecture based on this input. Classroom-response systems, which work via apps or mobile websites, reduce the need to buy costly equipment. But technology alone is not enough. As indicated by Bain (2004), the biggest change is within the lecturer. Instead of delivering topics during the lectures, the lecturer gives structure and facilitates students to reach learning outcomes. Stimulating students to prepare via short videos, asking questions, and organizing discussions allow both lecturer and students to get feedback on progress toward course outcomes. The aim is to stimulate deep learning and change learning strategies via in-class activities, such as discussions.

Therefore, in this study, we used a pre- and posttest design to investigate the effects of flipped-class pedagogy on learning strategies in higher education and study whether the effects of a flipped classroom were persistent. Our aim was to promote active learning with long-lasting effects on student’s metacognition and collaboration strategies.

Pintrich and de Groot (1990) developed a Motivated Strategies for Learning Questionnaire (MSLQ) measuring several learning strategies (Pintrich et al., 1991). The validity of this instrument was investigated by Credé and Phillips (2011). On the basis of the results of the MSLQ, Stegers-Jager et al. (2012) suggested that participation and self-efficacy beliefs are important contributions for the effectiveness of learning in medical education and that encouraging participation and strengthening self-efficacy help to enhance student performance. As described in Pintrich et al. (1991), discussion activities can have effects on metacognitive strategies, such as those measured in the MSLQ—rehearsal of previously read information, organization, and linking to new situations—leading to critical thinking. Learners can test their understanding of previously read materials by comparing themselves with others. Task-value can increase via discussions with peers and explanations of tasks by the lecturers, awakening the intrinsic motivation of the learners for the course topic. When learners find they do well in discussion tasks, self-efficacy may increase as a result of seeing that their efforts from preparing materials beforehand pay off. When students improve on these metacognitive and collaborative strategies, the questions remains whether the effects are persistent. In this study, we wanted to know:

1. Do students improve motivation and learning strategies as a result of flipped-class pedagogy?

2. Do the effects of flipped-class pedagogy on motivation and learning strategies persist for several months?

On the basis of the literature cited, we expected that the flipped-class pedagogy would have positive effects on motivation and learning strategies of students and would thus improve the effectiveness of the class.

METHODS

Participants and Course Design

A cohort of full-time second-year bachelor psychobiology students from the University of Amsterdam, The Netherlands (n = 170), was followed during two compulsory courses (Figure 1). Before and after each course, MSLQ questionnaires were filled out (for details see next paragraph). Five months after the second course was finished, another MSLQ was provided to see whether effects were sustainable. Student attendance of the course sessions was not obligatory in either course.

Course 1 was the compulsory course Higher Mental Processes. The course design was a 4-wk course with daily lectures that lasted 2 h. The lectures were given by various lecturers in a traditional teacher-centric way (“traditional approach”). There was mainly a unidirectional flow of information from the teacher to the students. All the lectures were recorded on video (Web lectures, which included the lecture slides) and were made available in Blackboard (electronic learning platform of the University of Amsterdam) for all the enrolled students during the whole course. There were no explicit assignments for students to prepare for the lectures or to study for the exam.

Course 2 was the compulsory course Pathophysiology of the CNS. The course design was a 4-wk course with daily lectures that lasted 2 h. Four lectures were traditional lectures by various lecturers. One lecture (each Friday) had flipped-class pedagogy in a 2-h session (“traditional + flipped-class approach”) that was given by the same lecturer, who also scored student attendance. As in course 1, the Web lectures were available in Blackboard for all the enrolled students during the whole course. There were no explicit assignments for students to prepare for the traditional lectures. There were assignments for the flipped-class session once a week with the following goals:

1. Goal: Before the session to prepare/start constructing knowledge at home. The assignment was to watch a short video and answer questions about it, prepare conceptual multiple-choice questions at exam level about the material already covered by the traditional lectures, and submit these questions via a discussion forum before class. The video and the questions developed by students were made available online for the entire group of students.

2. Goal: Construct understanding during the session. The assignment was to discuss the questions/answers. A peer-instruction teaching method was used (Mazur, 1997), incorporating the questions created by the students. A clicker voting system was used in the following way: first, the question was posed, and the students had to answer the question within 1–2 min using the understanding they had of the topic so far. Thereafter, students were asked to discuss the possible answers with one another (“buzz groups”) for 3–5 min, thus constructing understanding via collaborative learning. During this time, they could also make use of the Internet or their notes to find the correct answer. Next, they had to answer the same question again. Once students voted, the teacher provided the correct answer and a short explanation.

Motivation and Learning Strategies Tests

The instrument used in this research was a validated MSLQ (Pintrich et al., 1991), which was recently revalidated (Credé and Phillips, 2011). The MSLQ consists of randomized questions and includes 15 different scales (see description in the Supplemental Material). Students rate themselves on a seven-point Likert scale from “not at all true of me” to “very true of me.” The MSLQ was used at the beginning and end of courses 1 and 2 (pretests MSLQ-I and MSLQ-III and posttests MSLQ-II and MSLQ-IV) to measure the changes in motivation and learning strategies of students in relation to the course design (Figure 1). Five months after MSLQ-IV, the MSLQ-V was taken to investigate whether possible effects were sustainable. Within those 5 mo, students had a 2-mo summer holiday followed by two traditional courses that lasted 1 and 2 mo, respectively. Each time an MSLQ was taken, all the students enrolled in the course were asked to participate. Participation was voluntary and no extra credit could be earned for participation or outcome. Between MSLQ-IV and MSLQ-V, there were no other courses that used a flipped-class pedagogy.

Appreciation of the Courses and Quality of Learning

At the end of each course, students were asked to evaluate the course anonymously, using a standard evaluation questionnaire, and to score the course between 1 and 10. In addition, the results of exam questions of course 2 that required a high cognitive level were analyzed to compare the cognitive level reached by the students who participated in all MSLQs and flipped classes with the results of the students who did not. Course exam questions were independently classified according to Bloom’s taxonomy (Anderson and Krathwohl, 2001) by one lecturer and two educational experts. The course exam questions for which the lecturer and the experts agreed that the level was Bloom level 4 (analyze) were taken into account for this research. For these multiple-choice questions students had to examine and break information into parts by identifying motives or causes or making inferences and finding evidence to support generalizations. Each question had four possible answers and only one answer was correct.

Data Analysis

Students were included in group 1 when they completed MSLQ-I to MSLQ-IV and participated in all flipped classes (Figure 1). Students were included in group 2 when they completed fewer than four MSLQs and/or did not participate in all flipped classes (Figure 1). Furthermore, the data obtained from the questionnaire were included in the analysis only when all the questions in the MSLQ were answered.

Per student, the average MSLQ score was calculated per component, as described by Pintrich et al. (1991). Data are reported as mean and SEM. Statistical analysis on MSLQ data for group 1 was performed using repeated-measures analysis of variance (ANOVA) followed by a post hoc Tukey test. Data for group 2 and comparisons between groups 1 and 2 were analyzed using a one-way ANOVA followed by a post hoc Tukey test. The frequencies of correct and incorrect answers for high cognitive-level exam questions were tested between group 1 and 2 using the chi-square test. In all comparisons, p < 0.05 was assumed to indicate a significant difference.

RESULTS

Response Rate and Study Groups

The average response rate for all MSLQs was 48%. Twenty-two students completed MSLQ-I to MSLQ-IV and were present during all flipped classes. These students were included in group 1. Students who completed fewer than four MSLQs and/or were not present during all flipped classes were included in group 2. The number of participants for each MSLQ is indicated in Table 2.

The percentage of students who completed all MSLQs and participated in all flipped classes (group 1) was 39% of the total number of students who participated in this study for MSLQ-I, 20% for MSLQ-II, 50% for MSLQ-III, and 19% for MSLQ-IV.

MSLQ Results for Students Who Participated during All Flipped Classes (Group 1)

MSLQ data were analyzed for students who completed all MSLQs and were present during all flipped classes (group 1), and a within-group comparison was performed for the traditional and traditional + flipped approach.

Traditional Approach

During the traditional approach, the average Likert scale of the various components was not different between the end (MSLQ-II) and the beginning (MSLQ-I) of the course (Figure 2 and Table 3).

Traditional + Flipped Approach

During the traditional + flipped approach, the average Likert scale of component 4 (cognitive and metacognitive strategies component “critical thinking,” p = 0.02), component 8 (value component “task value,” p = 0.001), and component 14 (resource management strategies “peer learning,” p = 0.03) increased at the end of the course (MSLQ-IV) compared with the beginning of this course (MSLQ-III; Figure 2 and Table 3).

To exclude that these changes were due to developments in motivation and learning strategies that would have occurred as part of the regular development of students, we did a comparison for all 15 components between the beginning of the traditional approach (MSLQ-I) and the beginning of the traditional + flipped approach (MSLQ-III). Components 4, 8, and 14 were not different at the beginning of course 1 and at the beginning of the course 2 (MSLQ-I vs. MSLQ-III, Figure 2), suggesting that the changes observed during the traditional + flipped approach are specific and presumably due to the flipped-class pedagogical method. Although these three components were not changed during the traditional class, the following other changes were observed over the 4-wk period between classes when comparing the components of MSLQ-I with MSLQ-III: component 11 (affective component “test anxiety,” 1.09 point decrease, p = 0.001), components 9 and 10 (the expectancy components “control of learning beliefs,” 0.95 point increase, p = 0.003; and “self-efficacy for learning and performance,” 0.66 point increase, p = 0.008).

MSLQ Results for Students Who Did Not Participate in All Flipped Classes (Group 2)

MSLQ data were also analyzed of students who completed fewer than four MSLQs and/or were present during fewer than four flipped classes (group 2). The total number of students who participated ranged between 22 and 91 for the different MSLQs (Table 2).

A between-group comparison showed that MSLQ data at the beginning of the traditional approach and at the beginning of the traditional + flipped approach were not different between group 2 and group 1, indicating that both groups were similar at the start of each course. Also, for group 2, a comparison was made between the traditional and the traditional + flipped approaches, which will be presented in the following paragraphs.

Traditional Approach

During the traditional approach, the average Likert scale of four components decreased at the end of the course (MSLQ-II) compared with the beginning of this course (MSLQ-I; Figure 3 and Table 3): component 1 (“rehearsal,” 0.67 point decrease, p = 0.01), component 5 (“self-regulation,” 0.43 point decrease, p = 0.01), and component 15 (“help seeking,” 0.72 point decrease, p = 0.008). The value component “extrinsic goal orientation” increased (component 7, 0.59 point increase, p = 0.02), as did the expectancy components “control of learning beliefs” and “self-efficacy for learning and performance” (components 9 and 10, respectively: 1.01 point, p = 0.001; and 0.50 point increase, p = 0.02).

Traditional + Flipped Approach

During the traditional + flipped approach, MSLQ component 7 (“extrinsic goal orientation,” 0.61 point increase, p = 0.02) and component 8 (“task value,” 0.65 point increase, p = 0.01) changed at the end of the course (MSLQ-IV) compared with the beginning of this course (MSLQ-III; Figure 3 and Table 3). Because component 7 was also changed during the traditional approach, we excluded this as an effect of flipped class. To exclude that the change of component 8 was due to development in motivation and learning strategies that would have occurred as part of the regular development of students, we did a comparison for all 15 components between the beginning of the traditional approach (MSLQ-I) and the beginning of the traditional + flipped approach (MSLQ-III). Similar changes were observed, as for the data set of the group 1, since components 9–11 were also different between MSLQs (Figure 3). This suggests that the changes that were observed during the traditional + flipped approach of component 8 (task value) were presumably due to the flipped-class pedagogical method, although students did not fully participate during all flipped classes.

Analysis of Exam Questions and Final Grade

To study whether an improvement of MSLQ components would translate into changes in learning outcomes, we specifically chose to evaluate the MSLQ component “critical thinking.” To do so, we studied how often exam questions that required a high cognitive level were answered correctly during course 2 (traditional + flipped approach) by students who participated in all MSLQs and flipped classes (group 1) as compared with students from the same course who did not participate in all MSLQs and/or flipped classes (group 2). Indeed, exam questions that required a high cognitive level were answered correctly by 41% of the students in group 1, which is significantly different from group 2 (23% correct, chi-square, p = 0.04). This is also reflected in the final exam grade, since the grade of students in group 1 (7.6 ± 0.2) was higher compared with group 2 (7.1 ± 0.1, p < 0.05) during this course (traditional + flipped approach). In contrast, the exam grades of group 1 and 2 were not different during the course with the traditional approach (6.8 ± 0.5 vs. 6.3 ± 0.1) or during the course that was given in between the traditional and the traditional + flipped courses (7.2 ± 0.2 vs. 7.1 ± 0.1).

Five-Month Follow-Up

To test whether the effects of the flipped class were long-lasting, we asked all students from group 1 to participate in MSLQ-V, which was given 5 mo after MSLQ-IV. Fifteen out of 22 students who participated in the four MSLQs also completed MSLQ-V. Although the effects of flipped classes seemed long-lasting for some students (Figure 4, A–C, line graphs), the average Likert values for the MSLQ components “critical-thinking,” “task value,” and “peer instruction” were lower for MSLQ-V compared with MSLQ-IV for these 15 students (Figure 4, A–C, bar graphs) and similar to MSLQ-I, II, and III, indicating that long-lasting effects of the flipped class were not evident.

Student’s Appreciation of Flipped-Class Pedagogy

At the end of each course, all students were asked to evaluate the course anonymously using a standard evaluation questionnaire and to score the course between 1 and 10. The students rated the traditional + flipped course with an 8.0 ± 0.7 and the traditional course with a 6.8 ± 0.5.

DISCUSSION

Our study on the effects of flipped-class pedagogy on motivation and strategies for learning shows that the substitution of a flipped-class session for one of the five traditional lecture sessions per week appeared to be sufficient to achieve changes in learning strategies of students toward deep-learning strategies. The main findings are:

1. Flipped-class pedagogy enhanced the MSLQ components “critical thinking,” “task value,” and “peer instruction.”

2. The effects of flipped-class pedagogy were not long-lasting.

Critical thinking is one of the most important strategies for reaching deep knowledge on an academic level. Gorissen (2013) lists 13 reasons to use lectures as an instructional method and gives “to make students think critically about the subject” the highest priority. However, according to constructivist philosophy and extensive research, giving only lectures is not yet enough for most students to achieve deep learning (Biggs and Tang, 2011). Although the constructivist philosophy is already widely accepted in education, and present in many institutional visions, institutions’ educational practices, or the so-called educational culture, are still based on traditional teacher-centric learning beliefs. Many students opt for surface-learning strategies if exams and course pedagogies do not stimulate them to choose higher-order learning (Entwistle and Ramsden, 1982). Faculties are focused on success rates. To diminish high student dropouts, additional teacher-centered sessions such as lectures and tutorials are often organized or measures are taken to efficiently filter out the students who do not achieve the expected number of credits on time. It is most important to stimulate and to encourage individual evidence-based teaching experiments and facilitate them in the professional development programs.

Effects of Flipped-Class Pedagogy on Learning Strategies and Quality of Learning

The goal of the flipped-class course design in our study was to enhance students’ learning activity before the class, to enable them to construct understanding and recognize difficult concepts before they attended the class. This was done by asking students to watch short video clips and submit questions before class, thus identifying gaps in understanding. The peer-instruction method was used during the face-to-face sessions to stimulate active engagement of students and to reach deeper understanding. It is known that learning improves when a peer-instruction approach is used (Mazur, 2009; Crouch and Mazur, 2001). To study the effects of flipped-class pedagogy, we used the MSLQ, since this is a well-validated questionnaire that has been successfully used in previous studies (Pintrich et al., 1991; Credé and Phillips, 2011). In combination with our longitudinal approach, in which the same students were followed, this enabled us to investigate the effects of flipped-class pedagogy. Students who participated in all flipped classes (group 1) showed an improvement in three of the MSLQ components (“critical thinking,” “task value,” and “peer learning”) during the traditional + flipped approach (course 2), while this was not evident in the same students during the traditional approach (course 1). Therefore, we assume that taking part in flipped classes is crucial for changing learning strategies. Although the number of contact hours and the weekly schedule (time on task) were the same for the courses 1 and 2, possible confounding factors include different lecturers and topics between courses. We therefore also performed a within-course comparison between students who participated in all flipped classes and students who did not. This showed that only students who participated in all flipped classes improved on the MSLQ components “critical thinking” and “peer learning.”

When the MSLQ results at the beginning traditional + flipped course (MSLQ-III) were compared with the results obtained at the beginning of the traditional course (MSLQ-I), the component “test anxiety” decreased, while the components “control of learning beliefs” and “self-efficacy for learning and performance” increased for students in both groups 1 and 2. Because students followed one obligatory traditional course (that lasted 4 wk) between course 1 and course 2, we assume that the differences in MSLQ components measured at the beginning of course 1 and at the beginning of the course 2 can be explained by the adaptation of students to the study system as they gained experience with assessments, homework, and/or timetables.

An increase in the MSLQ component “critical thinking” suggests that students were aware of higher cognitive processes such as application of knowledge in new situations, creating questions before they visited the flipped-class session and critically evaluating the statements during these sessions. The MSLQ is a self-evaluation tool, and therefore, this does not necessarily mean that the students with a high score on the MSLQ were indeed better in critical thinking. However, further analysis of the exam results showed that the highest cognitive-level exam questions (Bloom’s taxonomy; Anderson and Krathwohl, 2001) were correctly answered two times more by students who participated in all MSLQs and flipped classes as compared with students from the same course who did not participate in all MSLQs and/or flipped classes. This suggests that students who participated in all flipped classes indeed reached a deeper understanding of concepts.

In addition to “critical thinking,” the “task value” component of the MSLQ was also increased during the traditional + flipped approach. Because this increase was observed in group 1 as well as in group 2, this suggests that the change is due to the flipped-class pedagogical method; however, full participation in flipped classes is not a prerequisite. Task value refers to students’ perceptions of the course material in terms of interest, importance, and utility. This is in line with the results of Smith (2013), who reported a high positive attitude of students toward recorded lectures as used in a flipped class.

Furthermore, the component “peer learning” was increased in students who followed all flipped classes. This may be explained by the method used, as students were asked to discuss concepts with one another (“buzz groups”) during the class. Peer instruction is a powerful method for conceptual learning. Crouch and Mazur (2001) and DeHaan (2011) argued that peer–peer learning assignments stimulate students’ creative thinking. As can be concluded from the set of questions related to critical thinking, the flipped-class learning assignments that the students needed to accomplish before the class also had a very important role in stimulation of deep learning processes. Just-in-time teaching introduced by Novak (2011), which includes preclass Web-based assignments called “warm-ups” led to cognitive gains. This method, supported by feedback loops considering in-class and out-of-class learning experiences, helped students to cope with the material taught (Koopman et al., 2008). On the basis of this, we believe that, in the case of this study, the assignment in which students were asked to design questions before the class, some of which were used during the class sessions in learning activities using voting, is important for the learning process.

Thloaele et al. (2014) showed that learning activities with clickers improved the learning process. In a pre- and posttest control group design, they recorded effects of clicker questions on academic achievement. A group of learners who did interactive activities (using clickers) during lectures scored higher on the final course exam then a more passive group. Similar effects were found in the present study. Students who attended all flipped-classroom sessions (group 1) scored higher on exam questions on the analysis level of Bloom compared with students who did not attend all flipped-classroom sessions (group 2). Although these groups were self-selected (students chose whether or not to attend the lectures), this suggests that flipped-classroom pedagogy supports deep learning.

The effects of flipped classes in this study were not long-lasting. The average values for “critical thinking,” “task value,” and “peer instruction” measured after 5 mo (MSLQ-V) were similar to the values of the MSLQs that obtained before flipped-class pedagogy was introduced. The increase in the MSLQ components “critical thinking,” “task value,” and “peer learning” in course 2 was therefore not sustained. The fact that the values of these specific MSLQ components were lower for the same students during traditional courses given before and after the traditional + flipped course indicates that these students potentially have the ability to function at a higher metacognitive level. Apparently, this ability was not called upon during the traditional courses, and one course with a flipped-class design in a curriculum is not enough to make metacognitive effects sustainable. To achieve sustainability of flipped-class effects, we propose a change of the course design of traditional lecture-based courses. Flipped-class pedagogy should be repeatedly applied in the curriculum to help students develop learning strategies such as critical thinking and apply them to reach deep learning. In our study, this could not be achieved during traditional classes. During these classes, we only observed an increase in the MSLQ value component “extrinsic goal orientation” and in the expectancy components “control of learning beliefs” and “self-efficacy for learning and performance,” while the components “rehearsal,” “self-regulation,” and “help seeking” decreased.

Student’s Grades and Appreciation during Flipped-Class Pedagogy

The exam grades for students in group 1 was higher compared with group 2 during the traditional + flipped course. In contrast, the exam grades of groups 1 and 2 were not different during the course with the traditional approach or during course that was given in between the traditional and the traditional + flipped courses, indicating that students who participated during all flipped classes had benefit from this approach.

The average Likert-scale appreciation of the traditional + flipped course was more than 1 point higher than for the traditional course. Other studies showed similar outcomes. Smith (2013) reports high appreciation values, but some students found watching the videos in the flipped-class approach burdensome in terms of time. Student appreciation is important, since it can increase engagement with the material, thereby enhancing learning. Lage et al. (2000) found that inverted classrooms using educational technology engaged students with different learning styles. Critz and Knight (2013) reported that flipped classrooms had an overwhelming success for the faculty. The flipped-class approach helped students in succeeding to regulate and direct their own learning. Mason et al. (2013) found the flipped-classroom format to be satisfactory and effective. They argue that flipped classes allowed the instructor to cover more material and improved student participation and that students adapted quickly to the method. High dropout rates of students in STEM disciplines caused by the lack of faculty–student interaction can be significantly reduced by an active-learning approach that provides opportunities to think, respond, and interact (Watkins and Mazur, 2013). However, interactive engagement itself is not yet enough, but a combination with deep-learning activities is necessary to effectively increase academic achievement.

CONCLUSIONS

This research shows that flipped-class pedagogy improved metacognition and collaborative-learning strategies, since the MSLQ components “critical thinking,” “task value,” and “peer learning” were increased. This will most likely enhance deep learning and makes flipped class a valuable pedagogy in higher education. We have also demonstrated that flipped-class pedagogy can be used in combination with a traditional class without additional classroom or staff requirements. We have demonstrated that this pedagogy enhanced critical thinking and collaboration strategies that are important for reaching deep learning. However, the effects were not long-lasting. Therefore, we propose repeated use of flipped-class pedagogy in a curriculum to make effects on deep learning sustainable.

ACKNOWLEDGMENTS

REFERENCES