Design and Implementation of a Tool to Assess Students’ Understanding of Metabolic Pathways Dynamics and Regulation

INTRODUCTION

The importance of metabolism in biochemistry and cell biology is clear. The term “metabolism” is referenced in more than 7 million PubMed citations; narrowing the search to include dynamics and regulation yields more than 20,000 citations (www.ncbi.nlm.nih.gov/pubmed). Research over the past century has revealed the complexity and centrality of metabolic regulatory networks in health and disease (Bar-Even et al., 2012; Wegner et al., 2015; Curi et al., 2016). Furthermore, metabolic engineering, which seeks to use cellular pathways to synthesize fuels, chemicals, foods, and pharmaceuticals, is a growing field and relies on deep understanding of metabolic dynamics and regulatory mechanisms (Nielsen and Keasling, 2016). In recent years, advances in computational and experimental techniques have enabled holistic study of metabolic systems that were previously unimaginable (Thiele et al., 2013; Edwards, 2017; Nielsen, 2017).

Curricular Frameworks and Instructional Priorities for Teaching and Assessing Metabolism

Collaborative efforts in undergraduate life sciences education provide guidance as to the concepts and skills that instructors should prioritize in teaching and assessing students’ knowledge of metabolism. Vision and Change identifies “pathways and transformations of energy and matter” as one of the five core concepts for biological literacy (American Association for the Advancement of Science, 2011, p. 13). A second concept identified by Vision and Change, “systems,” also relates to metabolism. This concept highlights the “dynamic interactions among components of a system at multiple functional scales” and “focuses on emergent properties at all levels of organization.” Likewise, an effort supported by the American Society for Biochemistry and Molecular Biology identified “energy/matter transformation” and “homeostasis” as foundational concepts in biochemistry and molecular biology (Tansey et al., 2013, pp. 292–293; American Society for Biochemistry and Molecular Biology, 2019). The description of these concepts underscores how fundamental chemical principles apply within a biological context and that complex control mechanisms have evolved to regulate biological pathways.

Threshold concepts are ideas that are central to learning in a discipline, without which the learner cannot progress (Meyer and Land, 2006). The identification of threshold concepts for a given discipline was conceived as a mechanism to bring disciplinary experts, educators, and students together in a process aimed at revising instruction with the goal of maximizing learning (Entwistle, 2008; Perkins, 2008). Because of their dual focus on disciplinary expertise and teaching, threshold concepts provide a compelling lens through which instructors in higher education can evaluate their instructional practices. A community of biology, chemistry, and biochemistry educators previously identified “biochemical pathway dynamics and regulation” as a threshold concept for biochemistry (Loertscher et al., 2014, p. 524). This threshold concept focuses on the roles that thermodynamics and kinetic regulation play in determining the direction and flux of materials through metabolic pathways. It further emphasizes the emergent properties of metabolism and posits that, once students internalize these concepts, they should be able to make predictions about the effects of perturbations on metabolic dynamics (Table 1). Thus, educators using the threshold concept to guide curricular design can envision the possibilities for student learning when metabolism is considered as a dynamic system governed by chemical principles and shaped by a changing environment. For example, instructors can imagine assessment tools that ask students to interpret the effect of enzyme allosteric modifiers on the rate of product formation, to predict metabolic consequences of changing the environment of an organism, or to propose a strategy to increase production of a pathway product through enzyme mutation.

Assessment of Students’ Understanding of Metabolism

Relatively little is known about students’ understanding of metabolism, especially in the context of undergraduate biochemistry. The ability of introductory-level students to trace matter through cellular respiration and photosynthesis has been well characterized (for examples, see Wilson et al., 2006; Hartley et al., 2011; Parker et al., 2012) as have students’ misconceptions about energy in metabolism, especially those related to ATP (see Kohn et al., 2018, and references therein). Although these misconceptions are situated in the context of metabolic pathways, they focus on students’ understanding of the fundamental principle of conservation of mass and energy. Assessment of students’ understanding of metabolic pathway dynamics and regulation within the context of upper-level undergraduate biochemistry courses is much less common.

Anderson, Grayson, and Crossley present two of the few published studies investigating undergraduate biochemistry students’ difficulties in understanding metabolic dynamics. Using questions that asked students to consider the effects of inhibiting specific steps of the glycolysis pathway, they found that many students failed to understand “the essential nature of all of the reactions (and reaction intermediates) of a metabolic pathway” (Anderson and Grayson, 1994, p. 5). Students believed that reactions, especially those that generate ATP, could continue even if preceding reactions in a pathway had been completely inhibited (Anderson and Grayson, 1994; Grayson et al., 2001). Furthermore, they found that students struggled to apply thermodynamic principles to reactions in sequence and that they had difficulties calculating the energetics of pathways. While these findings reveal important challenges for teaching and learning about metabolism, Degerman and Tibell (2012) found that many metabolic concepts that instructors identify as difficult for students, including regulation of energy-producing steps and interconnection of metabolic networks through shared intermediates, have not been well investigated. Therefore, more work is needed.

Development and Use of an Assessment Instrument for Undergraduate Biochemistry

Assessment tools that shed light on students’ understanding of metabolic systems are much needed to support improved teaching and learning in the molecular life sciences. Here, we describe the community-based development and use of an assessment instrument designed to help instructors evaluate students’ ideas about metabolic pathway dynamics and regulation in undergraduate biochemistry courses. To characterize the potential usefulness of this educational innovation, we undertook the following:

1. Collected and analyzed data generated using the assessment instrument in order to characterize the range of knowledge and skills that students demonstrate when asked to make predictions, propose changes, and justify their reasoning related to an unfamiliar metabolic pathway.

2. Reflected on the instrument development process and on student responses leading to insights that could be used to improve instruction and assessment in undergraduate biochemistry courses.

METHODS

Context and Biochemistry Community

The instrument and rubric were developed and refined by a community of biochemistry, biology and chemistry educators working as part of a National Science Foundation–funded project aimed at improving undergraduate student learning in biochemistry. Goals of the project included identifying threshold concepts in undergraduate biochemistry and developing instructional and assessment tools related to these concepts. Over the course of 5 years, 43 different educators (including authors B.J.H., J.L., T.A.M., V.M., H.T.-T., and S.M.V.) were involved in the development of the instrument, including design of the questions, content validation, rubric development, and/or administration of the instrument in their classrooms. Much of this work took place at multiday faculty development workshops held in the summers of 2014, 2015, and 2016. Community members were drawn from different institution types, including doctoral universities with high/very high research activity (32%), doctoral/professional universities (14%), master’s colleges and universities (26%), baccalaureate colleges (19%), associate’s colleges (7%), and one health professional school. Three of the community members taught at Hispanic-serving institutions, and one taught at a historically black university.

Determining Concepts and Level of Cognitive Processing to Assess

The specific concepts to investigate within the framework of metabolic pathway dynamics and regulation were established by this community of biochemistry educators. The threshold concept is described in Table 1 (adapted from Loertscher et al., 2014). The knowledge statements describe what students should know about metabolic pathway dynamics and regulation, and the subsequent two columns describe what students would be able to do if they understood the concept. Because the threshold concept is complex and encompasses a number of different ideas about metabolic pathways, this community of educators decided to focus the assessment instrument on aspects of the threshold concept that relate to enzymes and enzyme regulation (shown in bold green type in Table 1). This decision was made not because enzymes and enzyme regulation are more important than other aspects of the threshold concept, but because the narrow focus enables a concise instrument suitable for use within a longer in-class exam. Using an iterative and interactive process that occurred both during and after the 2014 workshop, the community sought to operationalize targeted aspects of the threshold concept (shown in bold green type in Table 1) through the development of five learning objectives (Table 2, left column). Using principles of backward design, the community relied on these five objectives to guide subsequent development of assessment items and the scoring rubric.

In addition to targeting specific ideas from the threshold concept, the community wanted to assess students’ thinking about metabolism at higher levels of cognitive processing. This desire arose from a shared perception that assessment questions related to metabolism often focus on memorization or recall of facts about metabolic pathways. Item drafters applied the Blooming Biology Tool (Crowe et al., 2008) to write learning objectives and assessment items at the levels of application, analysis, evaluation, and synthesis when possible. Analysis, evaluation, and synthesis are typically considered higher-order cognitive skills (HOCS), whereas application can act as a bridge from lower-order cognitive skills (LOCS) to HOCS (Crowe et al., 2008). Application questions in biochemistry are often classified as HOCS, because they require conceptual understanding to determine relevant variables to be considered when answering a question (Arneson and Offerdahl, 2018). The learning objectives in Table 2 state that students should interpret figures, predict effects, and create a strategy, all of which map to the application, analysis, and synthesis levels of Bloom’s taxonomy.

Criteria for Instrument Design

As described earlier, the purpose of the instrument is to probe higher-order thinking about the metabolic pathway dynamics and regulation threshold concept. To remove rote recall as a factor, the instrument development community intentionally selected a pathway that would be unfamiliar to most students in an introductory biochemistry course. In addition to the content goals described previously, there were additional design criteria to ensure a practicable tool for both large and small classes. For example, while the inclusion of constructed-response items was deemed necessary for insight into student thinking, a detailed rubric was designed for ease of scoring. The instrument is intended for use within a course-based exam. Educators who use the instrument are free to score it as deemed appropriate for their courses. The scores reported here were determined using the final rubric developed as part of the iterative process described in the following section.

Another key consideration was the visual representation of the pathway itself. Decisions about pathway representations were made over the course of 3 days at a workshop attended by 20 biology, chemistry, and biochemistry educators. Although an effort has been made by the systems biology research community to unify pathway representations (LeNovere et al. 2009), that was not on the minds of faculty at the workshops. The primary goal was to identify a pathway with interconnections and branch points, using a format commonly seen in introductory biology and biochemistry textbooks. Based on their range of experience, this community felt the representation shown in Figure 1 would be accessible for a wide range of instructors and their students and would minimize the cognitive load of the diagram.

Iterative Question and Rubric Development Process

The instrument and scoring rubric were developed using an iterative process over the course of 4 years (Figure 2). The criteria for content and format, described above, were used as the community developed the questions and the rubric. All agreed that the pathway selected reflects a typical metabolic pathway with opportunities to examine complexities that arise in interconnecting pathways in which isoenzymes are present. Working in teams, instrument developers crafted questions to address the targeted learning outcomes (Table 2) at a specified level of cognitive processing using the Blooming Biology Tool. Simultaneously, teams discussed and crafted expected responses and prepared rubrics for the constructed-response questions. They identified important aspects of student responses as separate scoring components of the rubrics.

The instrument developed in year 1 was piloted at seven institutions, including two high/very high research activity doctoral universities, two doctoral/professional universities, two master’s colleges and universities, and one baccalaureate college. The authentic student responses from this range of institutions were used to make further revisions of the instrument and rubric. Feedback from instructors was also solicited and informed numerous cycles of instrument revision. In general, these revisions led to inclusion of less information and simplification of item prompts. Student responses from the pilot study were also used to test and refine the scoring rubric.

Description of Instrument Format and Content

The final instrument consists of a multipart question related to an amino acid biosynthetic pathway in bacteria. This biosynthetic pathway has regulatory characteristics similar to other pathways typically discussed in introductory biochemistry courses, such as glycolysis and the citric acid cycle. While the pathway presented to students during an exam is real, the one shown in Figure 1 is an altered version of the pathway, which preserves the security of the instrument. The instrument includes multiple-choice, fill in the blank, and constructed-response questions divided into four sections. There are a total of 12 question scores, of which six are multiple choice, one is fill in the blank, and five are constructed response along with one confidence question at the end of each section (Table 3). For clarity, the different sections of the instrument will be described in the context of the results.

Student Interviews

Cognitive interviews using a think-aloud protocol were conducted with six students who had completed an introductory biochemistry course 3 to 9 months before the interview. Students were provided the instrument and accompanying diagram and were asked to work through the questions while verbalizing everything they were thinking. Students participating in interviews completed introductory biochemistry at a different institution than students whose written responses were analyzed.

Instructional Context

One of the authors (H.T.-T.) embedded the instrument on final exams in her introductory biochemistry course over several terms. The course covers an introduction to biomolecular structure, enzymes, and metabolism, with 4 of 10 weeks devoted to metabolism. The content questions of the assessment comprised less than 10% of the final exam, and the confidence questions boosted the total to 11.5% of the exam. Although a detailed discussion of instruction is beyond the scope of this study, a brief description of the instructional context, including learning objectives related to metabolism and examples of pathway diagrams used during instruction, is included in Supplemental Material 1.

Exams were scored by the instructor and teaching assistants using criteria that they determined, not using the rubric or other protocols for this study. The questions proved relatively straightforward to score, which was important, given that typical class size was 400–500 students. The course instructor reported being extremely satisfied that the questions appropriately assessed targeted course concepts and that they provided rich insights into students’ thinking. Specifically, the instructor appreciated how the questions were progressively more difficult, first giving students an opportunity to demonstrate their knowledge of the system, then analyze the consequences of changes to the system, next design a new system, and finally evaluate the benefit of complex regulation for living organisms. She felt that the instrument structure allowed all students to display the extent of their knowledge, regardless of ability. Finally, because amino acid biosynthesis is not part of the course, the content was entirely new for the students. Thus, she felt that the assessment truly tested students’ understanding of pathway complexity and not memorized facts.

Data Collection for This Study

The final version of the instrument was administered to students in 14 different sections of the course described above between Fall 2015 and Spring 2018 at a doctoral university with high/very high research activity. Student responses (362) were randomly selected for analysis using a stratified process by term to ensure a representative sample for each term. These responses were then divided so that each rater scored an equal number of responses from the sampled terms. From these, a pool to be scored by both raters was selected so that responses scored in common were also selected from all course sections. De-identified student responses were analyzed using a two-phase strategy, described in the following sections.

Phase 1 Data Analysis: Rubric-Based Scoring

In the first phase of data analysis, two independent raters (authors B.J.H. and T.A.M.) scored student constructed responses using a rubric that had been designed to parse statements within student responses and allow identification of patterns in student responses. The rubric contained criteria to track correctness and completeness of student responses as well as criteria that were intended to track whether student responses contained specific rhetorical moves. For example, use of the word “binding” or words that reflected binding was tracked in two questions (Q4 and Q8/9) with the idea that use of this word could indicate thinking about molecular-level events. All statements that were tracked using the rubric are shown in the right-hand column of Table 2. It is important to note that instructors using the instrument in their courses may decide that students need not include all rubric statements for an answer to be considered fully correct.

Multiple-choice questions were scored as 0 for the incorrect answer and 1 for the correct answer; in a constructed-response question, variable point values were assigned. To capture different ideas expressed by the students, as described earlier, we assigned each independent idea 1 point and tabulated ideas separately. While Q4 is a single free-response question, the data presented here represent the inclusion of certain statements (three in all) in student’s responses; thus, it might appear that Q4 has multiple parts, which it does not. Q8 and Q9 examine related content, and sometimes students provided a response in Q8 that related to Q9. As a result, responses from these two questions were pooled for each student and scored as a single response (R8/9).

Each rater independently scored 146 responses for a total of 292 independently scored responses. The independently scored data were analyzed for this study, and the results are described below. Another set of responses was scored by both raters (70 responses, around 20% of the total) and was used to measure rater consistency in the use of the scoring rubric. The consistency of the scores for constructed-response items was assessed using interrater reliability as measured by Cohen’s kappa (Cohen, 1960). The Cohen’s kappa values were between 0.592 (moderate agreement) and 0.754 (substantial agreement; Cohen, 1960; Landis and Koch, 1977). Therefore, the rubric seems to have been used in a consistent manner, which supports further analysis of the data.

Phase 2 Data Analysis: Analysis of Written Responses

In the second phase of data analysis, four authors (J.L., A.M., V.M., and S.M.V.) undertook an analysis of students’ written responses. Rubric scores for each question were used to provide an initial framework for grouping responses. All possible score patterns for each question were identified using “1” and “0” to indicate whether a given response included (1) or excluded (0) a statement designated by the rubric. For example, a response that included all three statements for Q4 would be assigned to the category “1-1-1,” whereas a response that included statements 4A and 4B, but not 4C, would be assigned to the category “1-1-0.” Next, an iterative process was used to refine categories and the placement of responses into categories. As a first step in the refinement process, cases were identified in which two or more of the initial categories could be collapsed, because the responses were conceptually similar. This iterative process of identifying common characteristics and conceptual themes was continued for responses to all questions. The result at the end of the process was the designation of four categories for each question (Q4, Q5, Q8/9, and Q11). We did not decide ahead of time to arrive at four categories for each question; rather, those groupings arose organically through the analysis of common characteristics. Categories for each question are shown in the left column of Tables 4–7.

Once categories were identified, rubric scores for student responses in each category were tabulated, and patterns associated with each category were identified. Rubric statements corresponding to a given category are shown in the second column from the left in Tables 4–7. Examples of student responses in each category and the number of student responses in each category are also shown in Tables 4–7.

Human Subjects Oversight

This study was determined to be exempt from Institutional Review Board (IRB) review in accordance with federal regulation criteria by the IRB at the University of California, Los Angeles (no. 15-000803) and the IRB at Seattle University.

RESULTS AND ANALYSIS

Interviews Provide Insights into Student Interactions with the Instrument

Interviews provided insights into how students interact with and make sense of the pathway diagram and the questions themselves. Given the large volume of information provided in the opening prompt, the diagram, and prompts to individual questions, it is not surprising that students devoted significant time to organizing and processing information. Students moved back and forth between the opening prompt, question prompts, and the diagram as they used information in the text to decode the pathway. All students reread the opening and individual question prompts at least once, and some reread questions three to four times.

Students actively engaged with the diagram by circling intermediates, drawing lines between intermediates, and crossing out parts of the diagram (indicating relief of inhibition). Yet none sketched pictures of proteins, substrates, or inhibitors to help visualize, for example, where an allosteric inhibitor would bind to enzyme and where the substrate would bind to enzyme. Although question prompts focused on specific sections of the diagram, all students, at some point during the interview, surveyed the entire diagram to evaluate whether upstream or downstream factors could be influencing the part of the diagram they were considering to answer a given question. This suggests an awareness of the fact that parts of the pathway could be influenced by the larger system. Finally, some students made statements indicating that they assumed unidirectional arrows meant that reactions are irreversible. While others did not state this explicitly, none of the students made any statements that would indicate that they thought the reactions in question are reversible.

Students’ Understanding of Pathway Visual Representations (Instrument Section 1)

The first section of questions probes students’ ability to interpret visual representations related to regulation and dynamics in pathways (Q1 and Q2 objectives in Table 2). In this section, the first two questions are multiple choice, and the third question is a confidence question (Table 3). For Q1, students are asked which isoenzyme (V1, V2, V3, or none of them) is affected by increased charlie concentration. They are allowed to select more than one choice. For Q2, students are asked about the effect of charlie on enzyme W (choices are: increases the activity of enzyme W, decreases the activity of enzyme W, has no effect on the activity of enzyme W).

The pathway visual representations are described in the lead-in prompt. As previously described, these questions are intended to determine whether students understand the representations and the lead-in prompt. If students are not able to answer these two questions correctly, it is likely that they would be unable to correctly interpret the pathway in subsequent questions and their responses to those questions would not be meaningful. Analysis of the data showed that 248 (85%) students answered both questions correctly, indicating that they have at least a basic understanding of the visual representations used in depicting some pathways. These 248 students form the basis of the analysis.

Analysis of Students’ Answers to Constructed-Response Questions

Student answers to constructed-response questions (Q4, Q5, Q8, Q9, and Q11) were analyzed in two phases. In phase 1, responses were scored using the rubric to track inclusion or omission of specified ideas or words. Patterns observed for each question are described in the text as percentages of students including a given rubric statement and depicted in bar graphs (Supplementary Figures S1, S2, S3, and S4 in Supplementary Material 2). In phase 2, analysis of students’ responses identified different categories of student responses. The analysis as well as sample student responses are provided in Tables 4–7.

Students’ Thinking about Pathway Regulation (Instrument Section 2)

The second section of questions probes students’ ability to predict the effect of an allosteric modifier on the rate of product formation (Q4 objective in Table 2) and predict the effect of including a downstream product in the growth medium (Q5 objective in Table 2). There are three questions in this section, two constructed-response questions and a confidence question (Table 3).

A Mutation Eliminating Effects of an Allosteric Modifier (Q4).

For Q4, students are told that enzyme X₂ was mutated to eliminate the effects of foxtrot as an allosteric modifier. They are asked to predict the effect of that mutation on the rate of delta production and explain their predictions. The most likely outcome is that delta production would increase, because the pathway diagram shows that foxtrot is an inhibitor of enzyme X₂ activity.

To assess the completeness and correctness of students’ responses, we used the rubric to track three possible statements (shown as R4A, R4B, and R4C in Table 2 and in Supplementary Figure S1). The first statement (R4A) was whether students predicted an increase in rate of delta production. The second statement (R4B) related to including, by name, that foxtrot is an inhibitor of X₂ activity. The third statement (R4C) examined whether students included that the loss of allosteric binding resulted in the increased rate of delta production; the word “binding” needed to be included.

The vast majority of students (81%) articulated that the rate of delta production increases in the mutant cells (R4A). However, the initial scoring of R4B found that only 60% of students stated that foxtrot is an inhibitor of X₂ activity (unpublished data). This result was surprising, as we expected the percentage of students identifying foxtrot as an inhibitor of X₂ activity to be similar to the percentage who identified an increased rate of delta production. Re-examination of student responses revealed that some students used synonyms of “inhibition,” such as “negatively regulated,” and that some students failed to specifically name foxtrot as the inhibitor, but it was clearly implied. Using an expanded view of acceptable R4B responses to account for such variability, the percentage of students who provided a well-reasoned response for R4B rose to 74% (shown in Supplementary Figure S1). Very few students (2%) mentioned the loss of allosteric binding in their response (R4C). The aim of tracking this statement was to collect evidence of molecular-level thinking about the fact that an increase in delta production is related to the loss of allosteric binding. However, the small number of students using “binding” in their responses suggests that this is not a good measure of molecular-level thinking in this case, and therefore most faculty would likely determine statements R4A and R4B as fully correct responses.

Table 4 provides a summary of phase 2 analysis for Q4 and shows sample student responses for categories identified through this analysis. Most students (68%) provided both R4A and R4B statements, which was considered fully correct and well justified, even without including R4C. A significant number of students made the correct prediction about rate but failed to provide a reasoned justification or their wording had flaws (R4A only). Some students made an incorrect prediction about rate, but still recognized foxtrot as an inhibitor (R4B only). Finally, 12% of students produced responses that had none of the statements tracked in the rubric and were considered fully incorrect. Taken together, these results indicate that most students understand the role of allosteric modifiers in pathways and enzyme regulation.

Effect of a Pathway Product on Production of Pathway Intermediates (Q5).

The second constructed-response question (Q5) in section 2 (Table 3) continues to probe students’ understanding of regulation by pathway intermediates. Students were asked to predict what would happen to the overall production of foxtrot in the presence of high concentrations of echo and to offer explanations for their responses. High-quality responses were expected to communicate two major ideas: 1) connected branches of the pathway affect each other through changing concentrations of shared intermediates; and 2) pathways self-regulate to maintain relatively constant concentrations of pathway products. Similar to the previous question, we were interested in students’ ideas and reasoning. There were two statements that were scored separately for Q5 (shown as R5A and R5B in Table 2 and Supplementary Figure S2). The first statement (R5A) indicates that a high level of echo leads to inhibition of enzyme Y and, subsequently, an increase in foxtrot production due to all delta in the pathway being funneled to foxtrot production. The second statement (R5B) relates to the idea that there will be little change in the foxtrot production over time, because foxtrot inhibits the enzyme that catalyzes its own synthesis (enzyme Z), thereby preventing a dramatic increase in foxtrot production. Therefore, this question provides an opportunity for students to consider dynamic relationships within complex systems.

We observed that 68% of students included statement R5A about foxtrot increasing due to inhibition of enzyme Y. However, only 19% of the students included statements consistent with the second idea, that pathways self-regulate (R5B).

Table 5 provides a summary of phase 2 analysis for Q5 and shows sample student responses for categories identified through this analysis. Only 16% of students provided fully correct responses (first row of Table 5) that included both R5A and R5B, indicating they realized inhibition of enzyme Y would increase foxtrot production and then foxtrot would ultimately modulate its own production. In contrast, 52% of students included R5A only, indicating they considered initial effects of inhibiting enzyme Y on the branching pathway but did not consider the ability of foxtrot to modulate its own production. The sample response in the second row of Table 5 is typical of such responses. A small number of students (2%) did not mention the initial effect but made a correct prediction about ultimate foxtrot production (R5B only). Finally, 28% of students produced responses that had neither of the statements tracked in the rubric and were considered fully incorrect. Thus, less than one-fifth of students demonstrated a full grasp of pathway self-regulation, whereas two-thirds of the students demonstrated understanding of branching pathway effects.

Student’s Ability to Design an Experiment Related to Pathway Dynamics and Regulation (Instrument Section 3)

The third section of questions probes students’ ability to create a strategy to maximize production of an intermediate through enzyme mutation (Q7 and Q8/9 objectives in Table 2). There are three conceptual questions and one confidence question in this block. The first question (fill in the blank) asks students which one enzyme would be best to target for mutation (Q7). Mutating enzyme Y to eliminate the inhibitory effect that echo has on its own production was considered the best choice. A high percentage of students, 79% (194 students), chose enzyme Y. For subsequent analyses in section 3, we analyzed only responses for those students who chose enzyme Y for mutation; thus, for the rest of this block, 194 student responses were analyzed.

Students were next asked, through constructed-response questions, to indicate what site they would mutate on the enzyme, how the mutation would affect the enzyme, and why they think the mutation would be effective for overproducing echo. These ideas were probed through responses to Q8 and Q9. We found variability as to whether students wrote specific ideas as responses to Q8 or Q9. Therefore, these two constructed responses were considered together and henceforth are referred to as Q8/9 and R8/9. For Q8/9, there were four statements that were tracked using the rubric (shown as R8/9A, R8/9B, R8/9C, and R8/9D in Table 2 and Supplementary Figure S3). The first statement relates to students choosing the regulatory site for the enzyme mutation (R8/9A). The second and the third statements relate to the effect of the mutation on the enzyme: the second statement is to indicate that the mutation will relieve the inhibition by echo (R8/9B), while the third one states that the mutation would prevent echo from binding (R8/9C). The last statement is related to the effectiveness of the mutation, specifically by stating that this mutation creates unregulated production of echo (R8/9D).

We observed that most students (86%) were able to indicate the site where the enzyme would be mutated (R8/9A) and that, likewise, 86% stated that the mutation will relieve the inhibition by echo (R8/9B). Half of the students discussed binding in their response (R8/9C), but many fewer mentioned unregulated production of echo (R8/9D). Thus, the vast majority of students focused on the regulatory site and knew that a mutation at that site would affect enzyme activity, but many fewer wrote about binding and the overall lack of regulation.

Table 6 provides a summary of phase 2 analysis for Q8/9 and shows sample student responses for categories identified through this analysis. Proposing an appropriate mutation is the first step in properly approaching the experiment; therefore, the first three categories all include the statement R8/9A for choosing the proper mutation site. The first category of student responses includes statement R8/9A, the prediction of the effects on the enzyme (relieve inhibition, R8/9B), and evidence of molecular-level thinking (R8/9C prevent binding). We observed 45% of the student responses fit into the first category. Broken down further, 31% of student responses contained only statements R8/9A, R8/9B, and R8/9C, while 14% of responses also mentioned the unregulated production of echo (R8/9D). All responses in this category are considered fully correct with complete justifications. The second category of responses (R8/9A, R8/9B, and/or R8/9D), given by 32% of students, is primarily composed of responses R8/9A, R8/9B, and there is no mention that the mutation prevents binding. Further only five (2.6%) student responses in this category mention unregulated production (R8/9D). The third category of responses, given by 8% of students, includes only a correct mutation proposal (R8/9A). The last category of responses, given by 14% of students, relates to responses with incorrect thinking and no correct mutation site. For example, some of these students discussed binding effects, but at the incorrect site (see Table 6, last row, for an example). Taken together, these results demonstrate that most students (categories 1 and 2, 78%) have a solid understanding of allosteric enzyme regulation similar to Q4 and Q5, but fewer included statements about binding and/or unregulated production of the pathway product. Thus, many students were able to integrate key concepts to arrive at a good description of how a change, like mutation, can affect a complex system.

Student Responses about Isoenzymes (Instrument Section 4)

The fourth section of questions probes students’ ability to explain the role of isoenzymes in metabolic pathways (Q11 objective in Table 2). There are one constructed-response and one confidence question. The constructed-response question asks students to explain why organisms have multiple isoenzymes in pathways and asks them to use X₁ and X₂ to explain the advantage of having two isoenzymes in Escherichia coli. The rubric was used to track three statements (shown as R11A, R11B, and R11C in Table 2 and Supplementary Figure S4). To address the role of isoenzymes in pathways, statements related to fine-tuning of metabolites under fluctuating conditions (R11A) or different regulation under different conditions (R11B) were identified. Because the wording of students’ responses related to these ideas ranged widely, either statement R11A or R11B was deemed acceptable in explaining why organisms have isoenzymes. Therefore, responses (R11A/B) that discussed “fine-tuning” (R11A) were combined with those that used words like “differentially regulated,” “flexible in regulation,” or “selective regulation” under different conditions (R11B). The phrase “better regulated” was considered too vague and was not tallied. The rubric served as only a rough guide for this question, as students used such a variety of words to convey their thinking. We made it a priority to distinguish the aforementioned responses from those that indicated that isoenzymes were the “backup plan” in the event of mutations. The rubric also tracked students’ use of X₁ and X₂ to explain the advantage of having two isoenzymes in E. coli (R11C).

More than half of the students (62%) were able to make a generalization about the role of isoenzymes (R11A/B) to fine-tune regulation. More students (69%), could use X₁ and X₂ to explain the advantage of having two isoenzymes in E. coli (R11C), but not all included statements R11A or R11B.

Table 7 provides a summary of phase 2 analysis for Q11 and shows sample student responses for categories identified through this analysis. For the first category, 47% of students provided fully correct responses (including the combined R11A/B and R11C), which reflected a well-reasoned understanding of regulation. There was only one student who provided all three individual statements tracked by the rubric (R11A, R11B, and R11C). In the next category, 15% of students made general statements about the role of isoenzymes without specific consideration of X₁ and X₂ (R11A/B only). In the third category, 21% of students limited their responses to using X₁ and X₂ to explain specifically the advantage of isoenzymes in this case instead of making a generalized statement about isoenzymes (R11C only). Finally, in the fourth category, 17% of students did not include any of the possible statements. Some of the students in this last category mentioned that isoenzymes are found in different tissue types, which is true, but not for a unicellular organism like E. coli. Such a response is likely based on recall rather than the pathway presented in the instrument. Thus, more than half of the students demonstrated they understood that isoenzymes provide fine-tuned or selective regulation, while some students were only able to discuss the specifics relating to X₁ and X₂ and did not offer a generalization. Nonetheless this result is encouraging, as isoenzymes figure prominently in pathway regulation.

Student Confidence Data

Student confidence data were collected for each of the four sections of the instrument. In general, students’ confidence followed a normal distribution, with most students reporting confidence in the midrange of the distribution (Supplementary Figure S5 in Supplementary Material 3). This distribution of responses would be expected if the instrument questions function to probe a range of levels of understanding and engagement with the material. Data were analyzed to discover patterns, but no noteworthy trends were identified.

DISCUSSION

To address a lack of assessment resources related to metabolic pathway dynamics and regulation, we designed and implemented an assessment tool for use on exams in undergraduate biochemistry courses. We systematically documented the development and use of the instrument and analyzed students’ responses in order to characterize the range of knowledge and skills that the instrument enables students to demonstrate. The purpose of this analysis was to provide biochemistry instructors who may wish to use the instrument information about the insights into students’ thinking that may be afforded through use of the instrument in their classrooms. We were also able to use insights gained from analysis of these classroom-generated student responses to suggest future directions for biochemistry education research. A detailed discussion of each of these areas is given below.

Development of a Relevant Assessment Instrument for Undergraduate Biochemistry Courses

The instrument described here is the first of its kind, designed by a community of educators to assess students’ understanding of metabolic pathway dynamics and regulation, a threshold concept in biochemistry. The instrument was crafted for convenient use by instructors for a range of courses and institutions. The pathway diagram accompanying the instrument is complex, with seven intermediates, eight enzymes/isoenzymes, and three branches. The instrument targets learning objectives (Table 2) that relate to common elements found across metabolic pathways. These learning objectives, which are likely addressed in most introductory biochemistry courses, represent the starting point to understanding metabolic complexity for students and therefore a starting point for assessment.

Analysis of responses shows that students across a range of abilities are able to demonstrate their knowledge and skills when answering questions on the instrument. While some students met almost all of the expectations defined by the rubric, demonstrating a high level of competency, many of those who provided less-nuanced responses were able to demonstrate a basic understanding of targeted learning objectives that many instructors would consider satisfactory in an introductory metabolism course. Therefore, the instrument is likely to be a useful tool for educators to assess a spectrum of knowledge levels and to differentiate among students with regard to their understanding of pathway dynamics and regulation. Finally, a community of biochemistry educators agreed that the instrument relates to important metabolic concepts, so the fact that students did generally well is reassuring.

Implications for Biochemistry Instruction and Course-Based Assessment

Several attributes of the instrument make it a useful tool to assess students’ understanding and to improve instruction related to metabolic pathway dynamics and regulation. As noted, the instrument captures a range of responses and allows for differentiation among students with different levels of understanding. Furthermore, the structure of the instrument, which maps to learning outcomes (Table 2), enables instructors to identify specific ideas that their students understand well and those to which more attention should be given. Additionally, the rubric, if used by instructors, allows for fine-grained analysis of understandings, misunderstandings, and omissions displayed by students in a given course. Taken together, these attributes facilitate a systematic analysis of student responses, the results of which can inform instructional choices and enable instructors to be responsive to a range of context-specific student learning needs. The following paragraphs provide examples of how we reflected on student response patterns from one institution and planned for improved instruction in the future. Although we expect students’ strengths and difficulties to vary according to institutional context, the process of analyzing and reflecting on students’ responses generated some generalizable insights, which we shared here.

Analysis of student responses revealed that most students at the institution studied grasp the effects of allosteric regulation (Q4, Q5), the importance and role of the regulatory site (Q4, Q5, Q8, Q9), and the role of isoenzymes in pathways (Q11). Furthermore, most students demonstrated basic proficiency in grappling with branched pathways (Q5, Q8/9). Although this study did not investigate the relationship between instructional practices and student performance, these results were gratifying, because many aspects of instruction had been intentionally designed to support learning in these areas. Specifically, instructional time during the metabolism unit focused on pathway logic and regulatory dynamics, including elucidating the order of intermediates in metabolic pathways using worked examples based on experimental data. Furthermore, discussion sections, facilitated by teaching assistants, provided ample time for students to work through relevant pathway problems in small groups (see Supplemental Material 1 for examples). These instructional approaches mirror a study in which introductory biochemistry students’ performance on questions related to inhibition within a linear metabolic pathway was improved through use of metabolic pathway determination problems that analyzed pulse-chase and metabolic inhibitor experiments (Anderson and Grayson, 1994; Grayson et al., 2001). It is possible that something about the nature of these problems is particularly effective in supporting learning, but it is also possible that these problems, used within the framework of an intentionally designed course, function as formative assessments, which are known to support student success (Sambell et al. 2013). Formative assessment relies on thoughtful alignment of desired learning objectives with course assessments and provides students an opportunity to communicate their ideas about a topic and receive actionable feedback to improve performance (Nicol and Macfarlane-Dick, 2006). Vanderlelie and Alexander (2016) present a relevant application of these principles in a metabolic biochemistry course, observing improved student performance on final exam questions after introduction of frequent formative assessments and alignment of formative and summative assessment items with learning objectives. The structure of the instrument described in this study enables instructors to identify gaps in student understanding in their own courses, which could be addressed through use of targeted formative assessments. For biochemistry instructors wishing to enhance formative assessment practices in their own courses, Offerdahl and Arneson (2019) offer a rich review of formative assessment, including a wealth of specific resources that are relevant for learning in the molecular life sciences.

Although many students at the institution studied demonstrated proficiency in the key areas described earlier, the subtle idea of how branched pathways self-regulate over time appears to be more problematic. In Q5, students were asked to predict what would happen to the overall production of foxtrot in the presence of high concentrations of echo. Echo and foxtrot are connected to each other via a common intermediate, delta. Most students (68%) recognized that a high level of echo leads to inhibition of enzyme Y and subsequently an increase in foxtrot production due to all delta in the pathway being funneled to foxtrot production. However, only 20% of students concluded that there will be little change in foxtrot production over time, because foxtrot inhibits the enzyme that catalyzes its own synthesis. Considering the importance of self-regulation in metabolic pathways, these data make clear that our students could benefit from increased exposure to instruction that highlights pathway dynamics. Dynamic models and simulations, unlike static pathway schematics, enable students to make hypotheses and observe how changes, including altering enzyme activity, affect concentrations of pathway intermediates over time. Instructional modules using computational metabolic modeling have been described for use in courses (Rodriguez-Caso et al., 2002; Reyes-Palomares et al., 2009; Helikar et al., 2015; Cell Collective Learn, 2020) and laboratories (Angelani et al., 2018) in the molecular life sciences. It has recently been shown that students’ performance on metabolism questions can be improved through use of metabolic modeling activities (Booth et al., 2021). Our analysis of responses to Q5 provides an example of how lower than expected student performance on a given question can prompt instructors to seek additional resources and re-evaluate their standard instructional practices.

In addition to probing students’ understanding of concepts related to metabolic pathway dynamics and regulation, the instrument also provides instructors a starting point to evaluate students’ ability to interpret visual representations related to metabolic pathways. As instructors, we know that schematic diagrams use symbols to depict parts and connections of complex systems and often omit key details, which can be troublesome for students (Schönborn and Anderson, 2006; Towns et al., 2012; Offerdahl et al., 2017). Although most students at the institution studied correctly interpreted the visual representation of the pathway, 15% were unable to answer Q1 and Q2 correctly. The instrument therefore provides a means for instructors to gauge students’ basic visual literacy related to metabolic pathways in their course. Furthermore, it is possible that even those students who were able to appropriately interpret the visual representations may not deeply understand the underlying concepts (Talanquer, 2011). Interviews revealed that students actively engaged with the schematic diagram by circling intermediates, drawing lines between intermediates, and crossing out parts of the diagram. Yet none sketched pictures of proteins, substrates, or inhibitors, so it is unclear whether students were specifically considering molecular events when interpreting the pathway. Even though the instrument is limited in how much information it provides instructors about students’ conceptual understanding of visual representations, we hope that use of the instrument in their courses will prompt instructors to incorporate instructional practices that help students develop visual literacy skills in biochemistry (Schönborn and Anderson, 2009, 2010).

Finally, consideration of students’ responses as a whole highlights a key challenge in writing effective assessment questions related to metabolic systems. Namely, that the complex information processing, both visual and nonvisual, required to interpret and respond to questions about complex biochemical scenarios poses a significant challenge for novice biochemistry students. Similar to previous reports (see Offerdahl et al., 2017, and references therein), we observed evidence that students struggled during cognitive interviews to organize and apply information provided in the pathway and question prompts. These difficulties may have resulted in the lack of clarity that was observed in some student responses. Interestingly, the instrument questions were not thought to be particularly difficult by the exam development community; however, it is known that experts and novices differ significantly in the cognitive architecture needed to successfully engage in complex learning tasks (Sweller, 1994; Sweller et al., 1998). Therefore, as instructors seek to design and implement assessment questions related to metabolism, they should look to insights arising from the application of cognitive load theory to streamline cognitive load and support students in developing the cognitive architecture required to engage with difficult intellectual tasks (Sweller et al., 1998; Cook 2006; Offerdahl and Arneson, 2019).

Implications for Future Research

There is limited research on students’ understanding of metabolic pathway dynamics and regulation, and therefore deeper investigation is vital. Although the instrument was designed for use in the classroom, analysis conducted as part of this study revealed some areas that could be fruitful for future investigation, which are described below.

Students’ Thinking about Complex Systems.

Metabolism can be characterized as a complex system (Dauer and Dauer, 2016) in which a systems thinking approach could guide both instruction and assessment. Systems thinking is a set of skills used to understand and predict phenomena by considering dynamic interconnections and resulting emergent properties in complex systems (Arnold and Wade, 2015). Systems thinking has several key characteristics, including the ability to identify components and relationships among components of a system, the ability to organize a systems’ components and processes within a framework, the ability to make generalizations, and the ability to think temporally and make predictions (Ben-Zvi Assaraf and Orion, 2005). Students’ difficulty with systems thinking has been documented (Jacobson and Wilensky, 2006), and novices exhibit a markedly different approach to interpreting complex systems compared with experts (Hmelo-Silver and Pfeffer, 2004). Although metabolism is recognized as a natural context in which to develop and assess students’ systems thinking skills (Orgill et al., 2019), limited research has been forthcoming.

In scoring student responses as part of this study, we discovered commonalities among statements not tracked with the rubric, some of which provide insight into students’ thinking about metabolic systems. Of the student responses analyzed, 21% included statements that indicated they were thinking globally about the system as they answered a question about a particular branch of the pathway. For example, some students stated that the metabolite delta would accumulate when X₂ is no longer inhibited, because the enzymes that convert delta to the products foxtrot and echo are each inhibited by their products. This response was not prompted by the question and suggests that some students were thinking about the metabolic system holistically. Other students made comments about enzyme V3 and that inhibition by foxtrot would help modulate the amount of delta produced. These observations suggest that insights into students’ thinking about complex systems could be uncovered through future research that is specifically designed to investigate systems thinking as it relates to metabolic pathways dynamics and regulation. It could be fruitful, for example, to design a complementary assessment instrument, also situated in a metabolic context, which would be designed first and foremost to examine aspects of students’ systems thinking. Design of such an instrument could be guided by the systems thinking hierarchical model, which is based on research on how students engage with learning complex systems (Ben-Zvi Assaraf and Orion, 2005; Orgill et al., 2019; York and Orgill, 2020).

Nature of Students’ Understandings and Misunderstandings.

Analysis of student responses revealed that some students provided incomplete, incoherent, and incorrect responses, while others offered well-justified responses. As teachers, we expect such variability in student responses, but rarely have the time within the context of a course to investigate where students’ difficulties actually lie. It is natural to assume that students whose responses do not meet expectations have fundamental gaps in discipline-specific content knowledge, yet a variety of other explanations are possible (Talanquer, 2011; Coley and Tanner, 2012; Gouvea and Simon, 2018; Offerdahl and Arneson 2019). Discipline-based education researchers have investigated introductory students’ thinking about conservation of mass and energy in metabolic contexts (Wilson et al., 2006; Hartley et al., 2011; Parker et al., 2012; Kohn et al., 2018), but no substantial research has focused on the complexities of teaching and learning metabolic pathway dynamics and regulation in upper-level biochemistry courses. Therefore, contributions in this area would help inform instructional choices going forward. Studies that would be especially relevant include those that illuminate the relationship between visual literacy and learning metabolic pathways as well as those that investigate how instructional and assessment choices related to metabolism may help minimize cognitive load and promote learning.

Conclusions and Future Directions

The assessment tool described herein provides a starting point for evaluating students’ understanding of the metabolic pathway dynamics and regulation threshold concept. As not all aspects of the threshold concepts were examined by the instrument (Table 1), additional course-based assessment tools could be developed in the future. Concepts to explore could include reaction reversibility in pathways, the interplay between kinetics and thermodynamics, and metabolic flux. Assessment of these ideas could be enhanced through contextualization within a systems thinking framework, as described earlier. Furthermore, after development of the instrument described here was complete, the three-dimensional learning assessment protocol (3D-LAP) became available as a tool to aid college instructors in designing assessments that probe three different facets of science learning: scientific practices, crosscutting concepts, and disciplinary core ideas (Laverty et al., 2016). The educators involved in the instrument development process kept those facets of learning in mind, but future assessment tools would benefit from the explicit process outlined by the 3D-LAP. Finally, the success of this assessment tool is due in large part due to the collaborative work of a community of biochemistry educators who devised and refined all aspects of the instrument and rubric so that it would be useful for instructors across a range of courses and institutions. Although engagement with a community of experts is considered an essential step in designing assessment instruments for research (Reeves and Marbach-Ad, 2016), such a community effort is rarely undertaken for development of course-based assessments. Based on our experience, we hope that a collaborative approach to course-based assessment will become much more prevalent in biochemistry and beyond.

ACKNOWLEDGMENTS

REFERENCES