Can I Have Your Recipe? Using a Fidelity of Implementation (FOI) Framework to Identify the Key Ingredients of Formative Assessment for Learning

INTRODUCTION

The formative use of assessment has long been identified as critical to teaching for conceptual understanding (e.g., Sadler, 1989; Black and Wiliam, 1998a; Bell and Cowie, 2001). Formative assessment has been widely cited as a high-impact instructional practice; indeed, it has become an “urban legend” as perhaps the single most effective instructional intervention to increase student learning (Briggs et al., 2012). Effect sizes between 0.4 and 0.7 have historically been reported based on work by Black and Wiliam (1998a,b). However, sparse description of the methodology involved in calculating these statistics has led to recent scrutiny of the reported effect (Dunn and Mulvenon, 2009; Kingston and Nash, 2011; Briggs et al., 2012). A more recent meta-analysis calculates a much smaller effect size, on the order of 0.2 (Kingston and Nash, 2011). Further, the methods by which effect sizes were calculated may have biased the estimate (Briggs et al., 2012).

The empirical evidence is variable, but the link between formative assessment and student learning is intuitive and has strong theoretical support (e.g., see Sadler, 1998; Furtak and Ruiz-Primo, 2008; Black and Wiliam, 2009; Bennett, 2011; Offerdahl and Montplaisir, 2014). The literature is immense, which means that formative assessment may assume different names (i.e., assessment for learning) and definitions that reflect nuanced differences among researchers. For example, some focus on the actions of the instructor (e.g., Ruiz-Primo and Furtak, 2006; Offerdahl and Tomanek, 2011; Sadler and Reimann, 2018), others on the behaviors and predispositions of students (e.g., Gibbs and Simpson, 2004; Clark, 2012; Hepplestone and Chikwa, 2014). Differences notwithstanding, current definitions include representations of formative assessment as a process by which evidence of student learning is used to generate feedback to both the learner and instructor about progress toward desired learning outcomes (e.g., Sadler, 1998; Furtak and Ruiz-Primo, 2008; Bennett, 2011). Thus, formative assessment is thought to positively impact student learning through ongoing diagnosis and feedback about in-progress learning. We propose that, given the strong theoretical support for the efficacy of formative assessment, variation in effect sizes may reflect differences in enactment by instructors “in the wild” and not the efficacy of formative assessment as a high-impact instructional practice (Yin et al., 2008; Stains and Vickrey, 2017). Observed differences in outcomes between the theorized or “ideal” practice and enacted practice can be described in terms of fidelity of implementation (FOI) or “the extent to which an enacted program is consistent with the intended program model” (Century et al., 2010, p. 202).

We adopt an FOI perspective (Mowbray et al., 2003; Century et al., 2010; Stains and Vickrey, 2017) to generate a preliminary hypothesis about the critical components of formative assessment for learning and, in so doing, begin the iterative process through which further research can develop valid and reliable measures of the FOI of formative assessment. Such measures are necessary to empirically determine when, how, and under what conditions formative assessment supports student learning. Accordingly, we begin by defining formative assessment. We then present evidence in support of the hypothesized critical components and discuss potential mediating factors. Finally, we propose potential adaptations of the critical components and discuss the productivity of these adaptations in terms of student learning. We close with a brief discussion of the next steps for research within the context of student-centered undergraduate life sciences instruction. While the primary audience for this work is other discipline-based education researchers, when appropriate, we also include implications for practitioners.

DEFINING FORMATIVE ASSESSMENT

Historically, undergraduate teaching has been predominantly transmissionist in nature; instruction has been structured with the goal of conveying information to students (Barr and Tagg, 1995). Not surprisingly, within the context of these traditional learning environments, formative assessment has been employed as intermittent measurement of learning, an activity whereby the instructor conveys information to the student about the relative correctness, strengths or weaknesses, and appropriateness of the student’s work (Nicol and Macfarlane-Dick, 2006). In transmissionist instructional environments, the locus of control over assessment and feedback resides solely with the instructor. Opportunities for students to develop skills associated with self-regulated learning are therefore limited. Furthermore, while the delivery of feedback to students is accounted for, it largely neglects whether and how students translate feedback into actions that influence learning. As a result, formative assessment in these environments is difficult to distinguish from summative assessment.

With the growing adoption of evidence-based teaching practices and increasingly student-centered instruction, transmissionist instructional models are becoming less common. As a result, the paradigm of formative assessment has necessarily evolved in tandem from a unidirectional instructor monologue to an interactive multidirectional assessment conversation between and among students and members of the instructional team. Accordingly, a more contemporary definition depicts formative assessment as an iterative and reflective process or “dialogue” that creates a feedback loop through which members of the instructional team and students use evidence of student understanding to monitor progress toward learning outcomes and adapt practices to achieve those learning outcomes (Shepard, 2000; Bell and Cowie, 2001; Nicol and Macfarlane-Dick, 2006; Furtak and Ruiz-Primo, 2008).

This dialogue (Figure 1) most often begins with a formative assessment prompt (e.g., clicker question), an opportunity for students to engage with the material and test their understanding, thereby making their thinking explicit (e.g., Allen and Tanner, 2005; Knight and Wood, 2005). Ideally, evidence of student understanding informs both the learner and the instructor about students’ in-progress learning (Sadler, 1998; Furtak and Ruiz-Primo, 2008; Bennett, 2011). Diagnosis of learning progress can influence student and instructor behaviors. For example, productive student behaviors might include engaging in self-regulated learning processes, active reflection on his or her current level of understanding as compared with the desired learning outcome, and internally generated feedback (Nicol and Macfarlane-Dick, 2006), thereby prompting subsequent behaviors such as adopting alternative study strategies or seeking more information from additional external resources (e.g., textbook, peer, teaching assistant).

Evidence of student understanding can also inform instructor behaviors. If a gap between students’ current and desired level of understanding is perceived, the instructor might revise instruction or create new instructional activities (Hattie, 2012). This action could then initiate the start of a new cycle with another formative assessment prompt, with the goal of engaging students and eliciting evolution in student understanding. Alternatively, or additionally, the instructor can provide feedback to students about their in-progress learning. Ideally, this feedback incites further student action (e.g., engaging metacognitive processes, revising study strategies). We note here that, while the cycle frequently but not always begins with an instructor-generated formative assessment prompt, it is possible for an unsolicited student question to reveal evidence of student understanding.

In practice, formative assessment would likely be implemented as a cycle that would be repeated and spiraled forward, supporting students as they move toward the achievement of learning outcomes. Within student-centered learning environments, formative assessment is therefore conceived as both a measure of in-progress learning and a process to scaffold learning.

FOI: A FRAMEWORK FOR UNDERSTANDING THE IMPACT OF FORMATIVE ASSESSMENT

FOI, an approach rooted in the field of program evaluation, has recently been applied to research on the effectiveness of instructional practices in undergraduate science, technology, engineering, and mathematics (STEM). As applied to undergraduate STEM, an FOI perspective would assert that outcomes of common student-centered practices (e.g., peer instruction) are determined by the extent to which the critical components of an intended instructional practice are present during enactment (Stains and Vickrey, 2017). A critical component is defined as a feature or component that is essential to the instructional practice (Ruiz-Primo, 2006). Variations in how the critical components of an instructional practice are put into play are adaptations, the effects of which may be variable on the desired outcome (i.e., student learning). This variation can be explained, in part, by the myriad of moderating variables that are not critical components but still affect implementation and outcomes. While moderating variables by definition have a significant impact on implementation, they are not generally considered part of FOI itself (Ruiz-Primo, 2006).

Stains and Vickrey (2017) examined the potential impact of a well-known instructional practice, peer instruction (PI), through an FOI approach. PI is a student-centered pedagogical approach (Mazur, 1997) whereby students are asked to respond to a conceptual question that targets common student misconceptions. Typically, this response is provided using clickers or voting cards the students hold up. After they commit individually to an answer, students engage in discussion with their peers in an effort to convince them of the correct answer, and then vote individually again (Vickrey et al., 2015). A critical component of PI is question difficulty—only about two-thirds of students should get the right answer on the first try. Another component is discussion with peers following the first vote and before the second vote (Stains and Vickrey, 2017).

In practice, there are a number of ways in which an instructor might implement the critical components that comprise PI. A variation in the cognitive level of the question actually posed by an instructor (i.e., recall vs. application) represents an adaptation of that critical component (Stains and Vickrey, 2017). If an instructor were to implement PI by using predominantly recall questions, the FOI would be considered low (Century et al., 2010) and might result in smaller learning gains than desired. Moreover, comparisons of outcomes between canonical PI and low-­fidelity PI may result in misleading conclusions about the efficacy of the instructional practice.

For this critical component of PI, there is substantial empirical evidence to predict how productive an adaptation might be in terms of student learning; higher-level questions lead to greater learning gains than lower-order questions (e.g., Smith et al., 2009; Knight et al., 2013). Another critical component of PI is interspersing questions throughout lecture. Two possible adaptations could be 1) questions interspersed equally and 2) questions interspersed only in the second half of the lecture. Currently, there is little empirical work to determine the effect of these adaptations on student learning, and it may even be proven that no effect exists. That is to say, adaptations of critical components may be productive, neutral, or unproductive for the desired outcome (Mills and Ragan, 2000; Ruiz-Primo, 2006; Century et al., 2010).

The ways in which an instructor chooses to implement an instructional practice will determine how critical components are adapted. With respect to examining the implementation of instructional practices, potential moderating variables are readily identifiable in the literature. For example, research on the degree to which STEM instructors adopt evidence-based instructional practices (such as PI) are heavily influenced by personal and situational variables such as department reward structure, teaching context, and personal beliefs (Gess-Newsome et al., 2003; Henderson and Dancy, 2007).

In the next section, we continue to build on the vast formative assessment literature base by adopting a “structure and process” approach (Century et al., 2010) to operationalize formative assessment in terms of its critical components. Previous literature has sought to more clearly define formative assessment (e.g., Andrade and Cizek, 2010; Bennett, 2011), distinguish formative from summative assessment (e.g., Harlen and James, 1997; Bell and Cowie, 2001; Garrison and Ehringhaus, 2007), and expressly acknowledge the instructional and disciplinary contexts in which formative assessment is situated (e.g., Yorke, 2003; Coffey et al., 2011). Many authors have taken a holistic approach to formative assessment by identifying overarching characteristics and/or strategies, but fewer have examined the relationships between characteristics (e.g., Bell and Cowie, 2001; Black and Wiliam, 2009). We propose that an FOI approach will expand on this work through a finer-grained, and in some ways mechanistic, look at the components of formative assessment to answer questions about how different patterns of enactment impact student learning outcomes (Ruiz-Primo, 2006).

APPLICATION OF FOI TO IDENTIFY THE CRITICAL COMPONENTS OF FORMATIVE ASSESSMENT

Previous FOI work has applied three main approaches to identifying potential components: 1) leveraging empirical literature, 2) gathering input from experts, and 3) eliciting opinions of users and advocates about what works (Mowbray et al., 2003; Stains and Vickrey, 2017). The first approach consists of identifying the quantifiable evidence of the proven efficacy and effectiveness of formative assessment by reviewing empirical studies and relevant meta-analyses. The second includes conducting expert surveys and/or literature reviews to articulate “key ingredients” that do not yet have operationalized criteria that can be verified empirically. The last is often approached via qualitative research methods that shed light on the questions of “how” emerging from empirical studies. We used all three approaches in a recursive process. We began with a review of the literature and our own experiences as “users” of formative assessment to identify potential critical components. We then consulted the empirical and theoretical literature and applied similar criteria to Stains and Vickrey (2017) and Greenhalgh and colleagues (2004) to identify the level of support for each of the critical components (see Table 1). While leveraging the empirical literature, we reflected on the potential critical components in light of theoretical literature (and vice versa), our own experiences, and the experiences of other experienced instructors to identify potential critical components that also have strong empirical and/or theoretical support (see Table 2 and Supplemental Table S1).

The potential critical components can be further categorized into two groups, structural and instructional (Century et al., 2010). The structural group refers to those components that reflect the composition and organization of an intervention. For example, structural critical components might include the prescribed instructional materials and the order in which they are presented to students. Structural components would also include the requisite knowledge needed by an instructor to enact the materials as intended. The instructional (process) group includes components related to the expected roles and/or behaviors—the human interactions—that take place during the intervention. Century et al. (2010) refer to the process group as instructional critical components, which reflect how instructors and students behave and interact with one another.

While the level of empirical evidence for each of the proposed components varies, each one has substantial theoretical support. The proposed critical components in Table 2 should be viewed as a starting point for measuring the FOI of formative assessment. The FOI literature indicates that initial sets of critical components will be iteratively refined as researchers 1) identify productive and unproductive variations in enactment (adaptations) of each critical component and 2) develop valid and reliable measures of each component (Mills and Ragan, 2000; Ruiz-Primo, 2006). Accordingly, in the next section, we lay the groundwork for future research by predicting potential adaptations—variations in the enactment of one or more critical components—and discuss the relative productivity of these adaptations as they relate to student learning.

POTENTIAL ADAPTATIONS OF THE CRITICAL COMPONENTS OF FORMATIVE ASSESSMENT

FOI has been defined as “the extent to which the critical components of an intended educational program, curriculum, or instructional practice are present when that program, curriculum, or practice is enacted” (Stains and Vickrey, 2017, p. 2). Adaptations in the implementation of each component may positively or negatively impact the predicted outcome of the intended instructional practice or may have no effect at all. We used our own teaching experience, prior research, and the literature to hypothesize about likely adaptations of the critical components. While by no means an exhaustive list, we dedicate the remainder of this section to exploring the productivity of four potential adaptations.

Adaptation 1: Variation in Alignment between Learning Outcomes and Formative Assessment Prompt

The power of formative assessment is in its ability to diagnose students’ current level of understanding relative to the desired level (targeted learning outcome). Without a clear indication of the desired learning outcome—what students need to know or be able to do—it is impossible to diagnose student progress, let alone revise instruction or generate meaningful feedback to improve student learning. It follows, then, that effective formative assessment prompts should first and foremost align with the stated learning outcomes; they must uncover students’ current understanding relative to the desired level of understanding (Wiggins and McTighe, 2005; Handelsman et al., 2007; President’s Council of Advisors on Science and Technology [PCAST], 2012).

Consider the desired learning outcome for students to generate a scientific argument. Asking students to write a “Muddiest point” paper about what they do not understand at the end of class is unlikely to provide evidence about their argumentation skills and represents a potentially unproductive adaptation. On the other hand, a productive adaptation would be modifying a minute-paper approach to ask students to generate and justify one claim from a graph. A minute paper would potentially reveal students’ abilities to interpret data and generate claims, both of which are essential to scientific argumentation. In these two examples, a formative assessment prompt was expressly implemented in response to a desired learning outcome, but the degree of alignment influenced the productivity of the implementation in terms of student learning. Misaligned outcomes and prompts are not likely to produce information that allows for instructors or students to diagnose potential gaps between the present and desired levels of student understanding.

Adaptation 2: Variation in Evidence of Student Understanding Produced by Formative Assessment Prompts

Diagnosis of students’ progress toward intended learning outcomes is dependent on sufficient information about the current state of students’ understanding or abilities. Instructors need to identify not just what students know, but importantly, what students are struggling with, and how prevalent those difficulties are (Furtak and Ruiz-Primo, 2008; Offerdahl and Montplaisir, 2014). In practice, this may be challenging. Large class sizes common to introductory undergraduate biology courses often favor the use of closed-ended prompts, such as forced-choice questions (e.g., clicker questions), due to their ease of application and their capacity to reveal the pervasiveness of ideas within the class (Smith and Tanner, 2010). Clicker questions in particular have the added advantage of creating an opportunity for students to immediately compare their answers with others and receive feedback about the relative correctness of those responses (Preszler et al., 2007). Therefore, the use of clickers to elicit evidence of student understanding is a potentially productive adaptation. But the response options are typically instructor-generated, thereby only providing insights into student understanding about those specific topics selected by the instructor. Closed-ended prompts may fail to reveal unanticipated student ideas not previously documented in the literature or through prior teaching experiences (Offerdahl and Montplaisir, 2014). Use of such prompts could potentially be unproductive for student learning when uncovering diversity of student thinking is appropriate.

Open-ended formative assessment prompts provide a different opportunity for examining students’ understanding. For example, concept maps or student-generated models invite students to represent how they personally make connections between ideas (e.g., Mintzes et al., 2005). The potential merit of open-ended prompts is that they provide an opportunity to reveal the diversity with which students approach and make sense of phenomena and identify overarching trends within the class. On the other hand, if the prompt is too open-ended (i.e., muddiest point) students may provide information that is not directly related to the stated learning outcome or may provide little information—students don’t always know what they don’t know (Tanner and Allen, 2004; Offerdahl and Montplaisir, 2014). Alternatively, the prompt may generate an overwhelmingly large range of student ideas that are difficult for an instructor to summarize and respond to concisely.

Evidence of student understanding is a critical component of formative assessment; the cycle will not work without diagnosing what students know and do not know in relation to the desired learning outcome. Yet there is little empirical evidence to explain how variations in this critical component ultimately influence student learning.

Adaptation 3: Variation in Instructors’ Pedagogical Content Knowledge

We did not explicitly depict pedagogical content knowledge (PCK) in the model (Figure 1), though the literature has often drawn connections between PCK and instructional practice (e.g., Shulman, 1986; Abell, 2008; Etkina, 2010; Van Driel and Berry, 2012). Recently, PCK has been more concisely defined to encompass two related constructs, personal PCK and personal PCK and skills (PCK&S). Personal PCK is an instructor’s “knowledge of, reasoning behind, and planning for teaching a particular topic in a particular way for a particular purpose to particular students” (Gess-Newsome, 2015, p. 36). It can be thought of as the disciplinary and pedagogical knowledge drawn on by an instructor when planning for teaching. As such, it can be characterized as a structural critical component much like other types of instructor knowledge. PCK&S is a closely related construct; it involves activation of personal PCK while engaging in the act of teaching a specific topic in a certain way to achieve a particular outcome. PCK&S is likely an instructional critical component, in that it involves the real-time actions of instructors in response to the present context.

It has been suggested that teachers may need a basic level of personal PCK to enact an instructional practice; yet some teachers may come to the classroom with that knowledge, while others may not (Century et al., 2010). Biology instructors with well-developed personal PCK are readily able to articulate the “big ideas” in biology due to their content expertise and possess the pedagogical skills necessary to generate measurable learning outcomes that reflect what a student should know or be able to do relevant to those big ideas. Similarly, a combination of both content expertise and pedagogical know-how is required for instructors to create formative assessment prompts that not only align with learning outcomes but are likely to produce rich evidence of student understanding, including common reasoning difficulties specific to core ideas in biology.

Both personal PCK and PCK&S are necessary for an instructor to make sense of, respond to, and use evidence of student understanding effectively in real time. For example, responding to students’ misconceptions about the role of noncovalent interactions in protein structure requires knowledge of common student reasoning difficulties about atomic structure, bonding, and polarity (personal PCK) and creating meaningful opportunities in class for students to restructure their mental models to reflect more scientifically accurate understanding of the concept (PCK&S). Similarly, an instructor must have knowledge about effective feedback (i.e., relevant, timely, actionable) and the skills to provide that feedback within the context of a particular topic to guide students in achieving particular outcomes. In each of these examples, well-developed personal PCK and PCK&S are likely a productive adaptation.

K–12 teacher preparation programs have long recognized the importance of instructors’ PCK for effective science instruction (Abell, 2008; Van Driel and Berry, 2012), and further teacher professional development is associated with growth in personal PCK (Daehler et al., 2015). Perhaps unsurprisingly, professional development for life sciences’ instructors similarly incorporates opportunities to learn about and apply personal PCK (e.g., National Academies Summer Institutes on Undergraduate Education, Faculty Institutes for Reforming Science Teaching).

Adaptation 4: Variation in Actionable Instructor-­Generated Feedback

Feedback is a proposed structural critical component that links evidence of student understanding to students’ learning activities (Figure 1). In a meta-analysis of feedback interventions, Kluger and DeNisi (1996) detected an average positive effect (d = 0.41) of feedback on student learning, yet in more than 38% of the studies, a negative effect was measured. These data suggest that there are potentially productive and unproductive adaptations of feedback. Providing timely, relevant, and actionable feedback to students on their learning is likely to be productive for student learning, particularly if that feedback provides answers to the questions of “Where am I going?,” “How am I doing?,” and “Where to next?” Such feedback provides clear guidance for students on their learning (Hattie and Timperley, 2007).

It is intuitive to propose that the converse condition, feedback that is not timely or relevant, is likely an unproductive adaptation. Students are more likely to act on feedback that is relevant (Hepplestone and Chikwa, 2014) and promotes a dialogue and relationship with instructors (Nicol, 2010; Price et al., 2010). Moreover, there are other potentially unproductive adaptations, such as nonspecific praise or simple verification of whether the answer is correct, that have been shown to negatively effect student learning (Hattie and Timperley, 2007).

Implications of an FOI Approach for Undergraduate Biology Instructors and Biology Education Researchers

In the preceding sections, we used an FOI approach to 1) identify potential critical components of formative assessment that must be present to positively impact student learning and 2) describe adaptations of the critical components and their impact on learning. We argue that such an approach provides a useful lens for interpreting the effect sizes associated with formative assessment previously reported in the literature; namely, differences may be attributed to adaptations of the critical components of formative assessment rather than the efficacy of formative assessment as an instructional practice itself. In this section, we briefly discuss implications of the application of an FOI perspective for practitioners and for future research in undergraduate biology.

Current education reform documents call for undergraduate instructors to implement student-centered instructional practices with frequent and ongoing assessment of students’ in-progress learning, and to use assessment data to improve learning (American Association for the Advancement of Science, 2011; PCAST, 2012). Reflecting on one’s teaching has long been recognized as a process through which instructors can turn personal experience into knowledge about teaching that can support further development of the instructor and associated practices (e.g., McAlpine et al., 1999; Kreber, 2005; Andrews and Lemons, 2015). But to maximize the utility of reflection, instructors must know what cues to monitor and how to evaluate them (McAlpine et al., 1999; McAlpine and Weston, 2000). Explicit identification of the critical components of formative assessment makes explicit the potential cues that should be monitored. Similarly, FOI emphasizes noticing the various adaptations in implementation of critical components, and the potential effects on the desired outcome. This approach facilitates systematic reflection on an instructor’s personal formative assessment practice as it relates to student learning. For example, there is strong support that formative assessment prompts that elicit the range and extent of student understanding are a critical component of effective formative assessment. With this knowledge, an instructor can reflect on the degree to which a formative assessment prompt is providing sufficient evidence of student understanding and adjust if necessary. FOI is useful approach for instructors to encourage systematic reflection on enactment of formative assessment and to make sense of the implications of those actions.

For biology education researchers, FOI is a useful approach for designing studies that will lead to even more valid claims about the effectiveness of particular student-centered instructional practices in diverse settings. This is especially important at a time when the biology education research community has moved beyond comparison studies between traditional and active, student-centered instruction (Dolan, 2015). The evidence clearly indicates that active, student-centered learning produces greater learning gains (Prince, 2004; Freeman et al., 2014). As demonstrated here, FOI is an approach for researchers to systematically propose, refine, and measure the effects of the “key ingredients” of various student-centered instructional practices, thereby facilitating comparison between contexts. With respect to formative assessment in particular, this approach facilitates leveraging and/or refining existing quantitative and qualitative tools (Table 3) and developing further measures of the critical components of formative assessment. In doing so, researchers will be better equipped to tease apart reported differences in outcomes associated with various student-centered practices.

CONCLUSIONS

We have applied an FOI perspective to the potentially high-­impact instructional practice of formative assessment as a first step in laying the foundation for more empirical work investigating when, how, and under what conditions formative assessment supports student learning. Consistent with previous FOI work, future work should strive to 1) iteratively refine the critical components identified here, 2) establish valid and reliable measures of each critical component, and 3) design efficacy and effectiveness studies using an FOI approach. Such work will allow the biology education research community to more effectively document how variations in implementation of formative assessment impact learning in modern, student-centered undergraduate biology contexts.

ACKNOWLEDGMENTS

REFERENCES