Concept Inventories as a Complement to Learning Progressions

LEARNING PROGRESSIONS

Learning progressions (LPs) are a model for understanding student thinking (National Research Council [NRC], 2007, p. 8) as well as tools supporting curricular and instructional design (Alonzo, 2011; Black et al., 2011; Duschl et al., 2011; Furtak, 2012; Scott et al., 2019). These progressions represent a structured acquisition of general skills and knowledge (Perkins and Salomon, 1989; Salomon and Perkins, 1989) combined with the mastery of transformative concepts in the discipline (NRC, 2007, p. 8; Duncan and Hmelo‐Silver, 2009; Wilson, 2009; Ross et al., 2010; Cooper et al., 2012; Neumann et al., 2013). Until recently, LPs were encountered mainly at the K–12 level, but the number of LPs published for college levels is increasing (Claesgens et al., 2009; Cooper et al., 2012; Cooper and Klymkowsky, 2013; Sevian and Talanquer, 2014; Romine et al., 2016).

LPs for Biochemistry/Molecular Biology

Different approaches to developing LPs have been taken in different contexts. Because biochemistry is a science that draws on concepts from both chemistry and biology, LPs for biochemistry will require integration of practices and concepts, with an emphasis on “cross-cutting” concepts (Sevian and Talanquer, 2014; Ngai and Sevian, 2018). In the context of biochemistry, students who are struggling with fundamental concepts from chemistry will be ill-equipped to analyze more sophisticated processes such as protein folding and stability, metabolic flux, and ligand binding (Grayson et al., 2001; Sears et al., 2007; Robic, 2010; Ngai and Sevian, 2018). This assertion is backed by the findings of Ebenezer and Fraser (2001) who demonstrated the difficulty of transferring fundamental chemistry concepts to a different field; in their case, engineering.

Learning progressions are particularly useful for organizing complex constructs for which students do not necessarily connect concepts as taught in different courses. One such complex construct is acid–base chemistry, which is broadly covered in general chemistry, organic chemistry, and biochemistry and has applications across the biological sciences. Deep understanding of acid–base chemistry is vital to understanding a multitude of biochemical processes, but students often have misunderstandings and difficulty with prerequisite material. Students’ reasoning about acids and bases has been explored by both quantitative and qualitative methods, at both the high school (Lin and Chiu, 2007) and university levels (McClary and Talanquer, 2011; Bretz and McClary, 2015; Cooper et al., 2016; Romine et al., 2016). These studies have provided insight into students’ explanations of the behavior of acids and bases; how they use the Arrhenius, Brønsted-Lowry, and Lewis models; how they understand acid strength; and how they predict reactions involving these molecules. In some cases, the researchers have suggested new approaches to teaching so as to ameliorate difficulty and misunderstanding.

Concept Inventories for Assessment

Students’ arrival to each step of an LP—whether linearly or not—has been assessed by a variety of tools (NRC, 2007; Bernholt and Parchmann, 2011; Pappa and Tsaparlis, 2011). Assessment tools of growing interest include concept inventories (CIs). CIs are a tool for formative assessment, generally multiple-choice assessment tests that can diagnose difficulty with particular concepts (misconceptions or alternative conceptions) before instruction and can evaluate changes in conceptual understanding after instruction or other interventions (Libarkin, 2008). Although originally used mainly at the introductory level, CIs have increasingly been developed for more advanced-level topics in the molecular life sciences (Smith et al., 2008; Shi et al., 2010; Bretz and Linenberger, 2012; Couch et al., 2015; McFarland et al., 2017). The shortcomings of CIs have been documented when used simply as multiple-choice pre- and posttests (Coppola and Krajcik, 2014), and the nature and order of the questions has been shown to activate different levels of learning (McClary and Talanquer, 2011; Warfa and Odowa, 2015; Undersander et al., 2017). However, they are more likely to probe deep understanding than other types of assessment applied to large groups (Xu et al., 2017) and can be adapted to be more open-ended and, hence, more informative (Dood et al., 2018). Additionally, as noted by Alonzo et al. (2012), use of large-scale assessments (in this case, CIs) can help to incorporate the framework of LPs into existing systems and lead to change in approach or pedagogy more readily than creation of entirely new assessments.

TRANSFER OF CONCEPTS FROM ONE CONTEXT TO ANOTHER

We are particularly interested in how students learn and apply concepts from chemistry to biochemical contexts. This requires transfer of knowledge from course to course (general chemistry to organic chemistry to biochemistry) and from subject area to subject area (biology to chemistry; Wilson et al., 2006; Villafañe et al., 2011b; Haudek et al., 2012; Wolfson et al., 2014; Warfa and Odowa, 2015; Ngai and Sevian, 2018). To understand the curricular structure of the biochemistry sequence as it pertains to acid–base chemistry, we began to develop a hypothetical learning progression (HLP). Such an HLP would span multiple courses and provide a framework to allow for better articulation between courses and instructors. There have been only a few examples of LPs or proto–LPs that map students’ movement through multiple university-level courses and examine the handoff from one instructor to another (Marbach-Ad et al., 2007; Cooper et al., 2012; Cooper and Klymkowsky, 2013; Crandell et al, 2019).

A Note on Terminology

As described earlier, LPs have been defined as “descriptions of the successively more sophisticated ways of thinking about a topic that can follow one another” during students’ learning about a topic over a stretch of time (NRC, 2007, p. 219). It is worth noting, further, that LPs are “empirically grounded and testable hypotheses” (Corcoran et al., 2009, p. 8). Therefore, until an LP has been tested and validated, it should be considered an HLP, and this is the label that we apply to our scheme, although it is based in the literature and was tested with a limited set of students. An example of a fully actualized LP might be that of Johnson and Tymms (2011), who validated the HLP with a large-scale study across institutions and demographic categories and tested the model itself. Here we reserve the term “LP” for the idea of a model and use the term “HLP” for our own construct.

PRELIMINARY CONSTRUCTION OF AN HLP FOR ACID–BASE CHEMISTRY

Acid–base chemistry was the defined construct of interest for our LP, as it encompasses a broad range of cross-cutting concepts. This subject spans the chemistry curriculum—from general chemistry, to organic chemistry, to biochemistry—and is central to many major concepts, including equilibrium, reactivity, molecular structure, and catalysis. All of these topics are central to cell biology, as well, as they are necessary for understanding and teaching about cellular homeostasis, blood chemistry, enzyme catalysis, macromolecular structure, and transport mechanisms. Acid–base chemistry is also relevant to other biology topics such as acidification of lakes and other waterways. Because this concept is quite expansive, it is often taught in segments, and students may have difficulty connecting the underlying relationships from topic to topic or from course to course. This is particularly troublesome in upper-level courses such as biochemistry, because many contexts require the integration of these concepts for deep understanding. One reason for the inconsistency in student preparedness is that there has not been a systematic effort to define the level at which students should have mastery of particular topics at the end of prerequisite chemistry courses.

As described elsewhere (Reed and Wolfson, 2021), we developed the outline of an HLP on acid–base chemistry from general chemistry through biochemistry. We approached the construction of the HLP by consulting empirical studies on student understanding of acid–base chemistry, professional society guidelines, and common biochemistry texts; the latter two sources were included because many factors determine the path of student learning, including the instructional and curricular materials (Stevens et al., 2009). This consolidation of existing knowledge is the initial step in an iterative process of constructing an LP. We also interviewed content experts and instructors.

The initial outline for the HLP in acid–base chemistry was developed by first defining the range of content to include within the area defined by the lower and upper anchors. The lower anchor is what students know “after high school” and was in part based on the Next Generation Science Standards (NRC, 2013). The upper anchor is what students know “after biochemistry” and was in part based on the American Chemical Society guidelines for curricular supplements for biochemistry (ACS, n.d.) and the American Society for Biochemistry and Molecular Biology core concepts and associated learning goals (ASBMB, n.d.). The principles and theories within the concepts of the upper anchor were unpacked for necessary prior knowledge. The expanded concepts were initially ordered by the course in which students were expected to encounter the concepts. To verify and refine the initial order of concepts, literature on student understanding of acid–base chemistry was explored in depth from the literature on high school– and college-level mastery (Banerjee, 1991; Watters and Watters, 2006; Lin and Chiu, 2007; Orgill and Sutherland, 2008; Cartrette and Mayo, 2011; McClary and Talanquer, 2011; Cooper et al., 2016; Pan and Henriques, 2015; Stoyanovich et al., 2015; Romine et al., 2016; Tümay, 2016;); further guidelines from professional societies (ACS, n.d.; ASBMB, n.d.; NRC, 2013); and common biochemistry course texts (Voet and Voet, 2004; Berg et al., 2007; Nelson et al., 2013). Evidence supporting the inclusion of higher-level content was in part retrieved from biochemistry textbooks by searching the table of contents, index, and glossary for key phrases (e.g., acid, base, pH, pK_a) and consolidating knowledge expected of students as it pertains to acid–base chemistry. Steps on the HLP were categorized as occurring after high school, general chemistry, organic chemistry, and biochemistry.

For an example of the progression within a specific topic (Reed and Wolfson, 2021), compare the steps that students might attain in their understanding of pH and pK_a at the end of each course in the chemistry curriculum:

• As students enter their first college (general) chemistry course (after high school chemistry), they may be familiar with pH scale and associate it with acids and bases.

• After general chemistry, they may know the pH scale and recognize that it is a log scale and that each unit represents a factor of 10, and they will know the definition of pK_a and what a larger or smaller pK_a means for acid strength.

• By the end of an organic chemistry course, students should be able to predict general trends of pK_as by structures and estimate pK_as of groups, as well as predicting the direction of an equilibrium given the pK_as of the acids involved.

• When students have completed biochemistry, they should be able to apply concepts of pH and pK_a to a variety of situations, including predicting ionization states from pK_a/pH relationships; predicting enzyme reaction mechanisms that depend on proton transfer; and knowing that microenvironments are relevant to protein structure and function.

Of course, not all students will reach the steps described here at the time they complete a given course, but the HLP describes the progression leading to mastery.

INTRODUCTION OF CIS CONSTRUCTION OF LPS

In developing the HLP, we followed the scheme practiced by other researchers in the field (Anderson, 2008; Stevens et al., 2009; Shea and Duncan, 2013) by going through several cycles of interviews, testing, and analysis of results. However, as described here, we have also added a novel step, matching questions from published concepts inventories to the content in the LP.

Student surveys and interviews are essential to refine the HLP. In many cases, the starting points for student assessment were questions taken from CIs. These were expanded and clarified as a result of interviews, but the CIs themselves gave us some insight into conceptions and misconceptions encountered by instructors and researchers. For example, a set of three true/false/don’t know questions about acid–base equilibria from the Molecular Life Sciences Concept Inventory (Howitt et al., 2008) eventually expanded to a series of interview questions that covered definitions of conjugate acids and bases, buffer calculations, concepts of equilibrium, and ideas about biological steady state.

Existing CIs were mined for questions related to acid–base chemistry. These inventories included those for general chemistry (Mulford and Robinson, 2002), biology (Shi et al., 2010; Couch et al., 2015), and biochemistry/molecular biology (Howitt et al., 2008; Wright and Hamilton, 2008; Villafañe et al., 2011a), as well as test banks from the Journalof Chemical Education (Journal of Chemical Education, n.d.), including Digital Library websites and inventories (Digital Library, n.d.), and papers with questions specific to acid–base chemistry (Calatayud et al., 2007; McClary and Talanquer, 2011; Haudek et al., 2012; McClary and Bretz, 2012; Cooper et al., 2016; Voska and Heikkinen, 2000). (Other test banks, such as the ACS subject tests, ETS major field exam, and ASBMB certification exam were excluded, because these assessments require payment and registration and/or accreditation and, therefore, would not be available to most classroom instructors.) Questions were then mapped to the corresponding content on the HLP. Questions on the inventories that pertained to acid–base chemistry but were not initially represented were used to expand the HLP. Placement of the concept was suggested by surrounding questions on the inventory.

HOW DO CONCEPT INVENTORIES INFORM LPS?

Elements of the HLP and an outline of the full progression have been published elsewhere (Wolfson, 2019; Reed and Wolfson, 2021). Here, we present one section of the HLP (“after organic chemistry” and “after biochemistry” levels) as an example of the alignments between HLP and CIs, and how this analysis can identify gaps in both tools. We propose that such alignment brings an additional element of validation to the development process, particularly regarding upper-level concepts, and can make LPs more accessible to classroom instructors.

Table 1 illustrates selected content (column 1) of the HLP for the two levels “after organic chemistry” and “after biochemistry.” For each topic in the HLP, the literature relevant to its inclusion is cited (footnotes) and a question from a CI or comparable instrument where these questions could be identified is included (column 2). For example, the content “Can predict ionization states from pK_a/pH relationships” is included due to its discussion in multiple sources, including: ACS guideline curricular supplements for biochemistry (ACS, n.d.), Villafañe et al. (2011b), and all textbooks consulted, and can be assessed with a question from Villafañe et al. (2011a). Questions are summarized rather than being quoted in full because of length and so as to avoid separating a CI question from its full context.

Table 1 demonstrates the utility of adding the step of alignments to CIs in validating an HLP. The content of the HLP often corresponds to questions on an existing assessment tool. What we found, however, was that new content for the progression emerged (indicated by an asterisk in column 1) from this analysis, because questions were found for acid–base chemistry concepts that had not initially been included on the progression: prediction of products based on understanding acid–base characteristics and description of open and closed systems. (This is not meant to imply that there is no literature on these topics, merely that we had not included them as steps on the progression until prompted by CI questions.) We also saw that multiple concepts in biochemistry could not be matched with questions (indicated by blank space in column 2), including prediction of function as result of structure due to pH (enzyme reaction mechanism) and understanding of microenvironment as related to pH and pK_a. These gaps indicate the need for more sophisticated CIs or other tools to assess student understanding of acid–base chemistry in complex biological contexts as encountered in undergraduate biochemistry.

LPS AND CONCEPT INVENTORIES: TWO INCOMPLETE FRAMEWORKS THAT CAN STRENGTHEN ONE ANOTHER

Learning progressions and CIs are very different tools that serve very different purposes. LPs are theoretical frameworks most often used by education researchers to develop an understanding of students’ acquisition of thinking skills and specific content. CIs are a means of formative assessment that allow faculty a glimpse into students’ grasp of core ideas at a particular point in their education. The idea of using CIs or other pre- and posttest instruments as a tool for instruction is certainly not new (e.g., see, Marbach-Ad et al., 2010; Villafañe et al., 2011a; Momsen et al., 2013; Xu et al., 2017). However, one of the goals of our research is to make LPs more accessible and useful to college classroom instructors (Reed and Wolfson, 2021). This is a challenging problem that has also been addressed by others (Scott et al., 2019). Connecting the progression to ideas that can be assessed by CIs may be one way to bring LPs to practitioners.

CIs are themselves based on review of the literature and have been tested with large groups of students (D’Avanzo, 2008; Libarkin, 2008). These inventories capture students’ reasoning across a wide range of topics, including some related to acid–base chemistry. By mining these inventories for questions about acid–base chemistry, we have been able to refine the HLP. Questions on the inventories that were not represented in the HLP led us to consider where the concept belonged. On the other hand, it was clear from our examination of existing CIs that there are gaps in available inventories and other assessment tools. For instance, there were very few questions, even in the CIs targeting biochemistry/molecular biology knowledge, that addressed acid–base concepts in biological contexts such as protein structure or microenvironments. This observation speaks to the limitation of CIs for assessing more complex concepts encountered in upper-level undergraduate course work. This gap may be due in part to the difficulty of assessing sophisticated understanding using a multiple-choice test. It may also be because there has been little research until recently on student learning in biochemistry. The situation is shifting with recognition that assessment of conceptual understanding is crucial for all upper-level courses (Bhattacharyya, 2006; Bodner and Weaver, 2008; McClary and Talanquer, 2011), including biochemistry (Cousin, 2006; Haudek et al., 2012).

Figure 1 shows the feedback envisioned between the input of content to steps on an HLP and the verification of these steps. CIs can serve both as evaluation/validation tools and as input in the HLP. And, clearly, the same inputs inform development of CI questions.

Both LPs and CIs are incomplete frameworks for student learning. One of the major criticisms of LPs is that they assume a student’s path through a discipline is linear. This is not always the case, as has been demonstrated, for example, by careful mapping of students’ conceptions about molecular interactions (Johnson, 2013; Sevian and Stains, 2013) and the structure of matter (Talanquer, 2009). In fact, exposure to the language of the discipline and quantitative interpretations may impede student progress through a progression (Sevian and Stains, 2013; Wolfson et al., 2014). Similarly, CIs reveal less information about student reasoning than do interviews or other extended explanations (Cooper, 2015). Despite evidence-based construction of questions and distractors (Garvin-Doxas et al., 2007), CIs can elicit correct answers even when students’ reasoning is faulty (Haudek et al., 2012). However, the use of CIs for pre- and posttesting is a valuable tool for faculty in planning curricula and assessing student progress toward mastery of a concept (Xu et al., 2017). Embedding CIs into LPs may make LPs more accessible to practitioners, who often do not see their purpose for classroom instruction (Reed and Wolfson, 2021).

LIMITATIONS AND CONCLUSIONS

As part of the initial cycle in development of an HLP on acids and bases, we have used CIs to begin evaluating the progression. However, as we have noted several times throughout this Essay, LPs and CIs are very different tools: they have different audiences, different goals, and different levels of granularity. Each is meant to taken as a whole and has been constructed and refined as such, not broken apart into individual pieces or questions, as we have done here. It is unlikely that an entire CI would correspond to a step on an LP, and, in fact, it would be cumbersome to use a full CI to assess students’ arrival at a single step. What we have done is to use individual CI questions as starting points for assessment of students’ travel along a progression. When separated, these items lose the validity of student responses to the full CI. But we find this exercise in alignment to be useful, because it gives both researchers and instructors additional tools for probing students’ learning.

Our choice of acid–base chemistry as the focus of our HLP provided both advantages and limitations. There is a rich literature on student understanding of acid–base chemistry and its applications. Including all contributions to this literature is challenging. On the other hand, the availability of multiple CIs and other tools for assessment of student learning allowed many opportunities to map HLP steps onto CI questions. HLPs for other topics may not have such a wealth of available inventories and questions.

The alignment of CI questions with steps on an HLP allows drawing conclusions about the order of the LP and also identifies gaps in measuring student understanding about acids and bases in the various inventories. This method adds a new tool to development of LPs, and the use of CIs to assess students’ arrival at different points along an LP may encourage classroom instructors to make more use of the LP literature.

ACKNOWLEDGMENTS

REFERENCES