Understanding Randomness and its Impact on Student Learning: Lessons Learned from Building the Biology Concept Inventory (BCI)

INTRODUCTION

Efforts to improve teaching and learning in science, technology, engineering, and mathematics (STEM) disciplines through the adoption of constructivist approaches have become more widely implemented with varying degrees of success (see Hake, 1998). The driving force behind these reforms remains somewhat obscure, and their ultimate longevity and impact remains to be ascertained; in part they may be attributable to professional pride—once instruments like the Force Concept Inventory (FCI; see below) became available, and their results taken to heart, it became apparent that even the highest quality students at elite institutions, taught by universally admired instructors, often failed to robustly understand the conceptual foundations of key topics. Adding to this “internal” impetus one might argue that the wider acceptance of a business/performance metaphor for higher education (e.g., Solomon and Solomon, 1993), and calls for increased federal oversight and assessment of education efficacy in the light of rapidly escalating costs (see Commission for Higher Education, 2006), may also be involved. This has provoked a number of responses ranging from outright rejection of mandated assessment to calls for self-assessment (see National Association of State Universities and Land Grant Colleges, 2006; Mehta, 2006). Rarely, a professional society such as the American Chemical Society has taken a lead in establishing curriculum standards and standardized outcomes assessments, but this is not the case in most STEM fields.

One response of at least a sector of the STEM education community has been increased efforts on replacing the more traditional emphasis on rote-level learning with the goal of enhancing students' conceptual-level understanding (Brainard, 2007). In the physics education research community, assessment instruments designed, researched, and validated as measures of student understanding at the conceptual level, often known as concept inventories, have played an enormous role in catalyzing the development, implementation, and increased adoption of constructivist teaching approaches. The FCI (Hestenes et al., 1992; Hestenes and Halloun, 1995), the most widely used instrument, was designed to measure students' conceptual understanding of Newton's laws of motion (a staple of most first-semester physics courses) and, as a result, has also been the most influential. More recently, the FCI has been joined by the Force and Motion Concept Evaluation (FMCE; Thornton and Sokoloff, 1998), which covers similar ideas, and the Brief Electricity and Magnetism Assessment (BEMA; Ding et al., 2006), which deals with concepts common to many second-semester physics courses.

In the biological sciences there has been an increased awareness of the need for similar assessments as a means to drive improved teaching and enhanced students' understanding of key ideas (Klymkowsky et al., 2003; Garvin-Doxas et al., 2007a). This has been associated with faculty development in both the general sense as well as in particular subdisciplines, for example the Ecological Society of America supports Teaching Issues and Experiments in Ecology (http://tiee.ecoed.net/), an online journal of “ecological educational methods,” while the American Society for Cell Biology sponsors the journal CBE—Life Sciences Education (www.lifescied.org). The BCI is an attempt at developing a class of instruments that can probe at the conceptual level the wide range of biology subjects covered in introductory undergraduate (and many high school) courses.

BCI DEVELOPMENT

The development of the BCI faced the challenge of a relative dearth of research on student understanding of foundational concepts in the biological sciences. Moreover, much of what research has been done has focused on K–12 students, rather than on undergraduates. Some instruments have been described; there exist instruments focused on diffusion and osmosis (Odom and Barrow, 1995), natural selection (Anderson et al., 2002), and there are ongoing efforts to build concept inventories covering other areas.

Our approach has been to map student understanding using a series of open-ended questions on a wide range of topics in the biological sciences. These student responses are typically 100 to 200 words long and are captured and analyzed using our Ed's Tools Web software system (Klymkowsky and Garvin-Doxas, 2008; Garvin-Doxas et al., 2007b). Responses are examined using content analysis, a technique designed to identify patterns in text-based data that range from the use of vocabulary words to recurring phrases and the meaning of the words (Holsti, 1969; Stemler, 2001). Whereas content analytical techniques sometimes use a priori categories, for the BCI project we began with categories based on recurring patterns found in students' discourse. Consistent with this type of qualitative content analytical technique, the first coding round used very detailed categories whereas each succeeding round used increasingly broader category/pattern “names” or labels to subsume similar and related early phrases, meaning(s), and ideas found in the students' responses. In this way, we were able to identify not only how students talk about their understanding of biology, but those areas where they hold common misconceptions and others where students appear to have a consistent and valid conceptual understanding. The process as a whole is iterative, and the analysis of student essays is followed by thematic and structured think-aloud interviews with students, further essay questions, and analysis. The process was repeated until an instrument was developed that looks like a multiple-choice “test” but with distracters that capture commonly held student misconceptions, based on the essay and interview data.

As most instructors quickly realize, students often fail to take the time to understand what an open-ended question is asking. Rather, they see key words or topics they recognize, and respond by listing everything they know about them. This type of behavior, where students provide the sort of “rote” responses they believe their STEM teachers desire and expect, is based on their assumption that answers in STEM fields are unambiguous, that is, there is only a single correct response to any given question and anything else is simply wrong. In the context of the BCI Project, we characterize this behavior as students restricting themselves to the “rhetoric” of science. We mean this in the Aristotelian sense, that is, where they know and believe that they understand their audience—the teacher or grader—and seek to provide an answer that they believe will appeal to that audience. They seek to use the appropriate words to persuade their audience that they have provided a good response to their question (Aristotle, 1991). Although it is frustrating to read essays where students fail to respond to the actual question, it has been critical to the development of the BCI in two ways:

1. Student essays provide us with the natural language students use to explain or talk about things that they know (or think they know) about the biological sciences. Because students are often locked into the rhetoric of science, they tend to interpret any biology vocabulary as a “clue” to the correct response. We find that the only way we can get students to reveal their true conceptual-level understanding is to use their natural language rather than the “technical language” they learn in their courses. Students' natural language (majors and nonmajors alike) is surprisingly sophisticated and reflects the fact that they are exposed to concepts and vocabulary in the biological sciences throughout their earlier K–12 educational experiences.

2. The essays provide the topics and context for development of thematic interview questions that are used to explore in greater detail the most commonly held misconceptions, and serve as pointers for the subsequent rounds of essay question–based analyses. Analysis of student responses to several, superficially unrelated questions, can suggest difficulties with a particular conceptual area, but the data may not be focused enough to indicate the precise nature of their misconception. It does, however, point out the direction that future research questions and interviews should take.

ASPECTS OF STUDENT UNDERSTANDING: NATURAL SELECTION AND EVOLUTION

Evolutionary change in biological systems is based on three processes: (1) the appearance of genetic variation through mutation and the capture of genetic information through either (2) selection (natural and sexual) or (3) random processes, such as genetic drift, genetic bottlenecks, and founder effects. Genetic drift is a completely random process, whose importance appears to be poorly understood, even among working scientists (see Lynch, 2007). In this context mutations are, at least as a first order approximation, random, and the accumulation of neutral or near-neutral mutations (Kimura's “Neutral Mutation” hypothesis) serves as the basis of most phylogenic analyses. From a broader perspective an understanding of what “random” means is therefore important in a variety of areas of the biological sciences, including molecular and cellular biology, and extends throughout other STEM disciplines (e.g., radioactive decay, molecular collisions, etc.). Although it is common for educators to encourage students to make connections across disciplines, it is often a challenge because students rarely see how what they are learning in physics or chemistry relates to their understanding in the biological sciences, and vice versa.

Student responses to three different essay questions during our first round of data collection were collected and analyzed through our online Ed's Tools system, a java-based system designed to create a database of metatagged student language (described in more detail in Klymkowsky and Garvin-Doxas, 2008 and Garvin-Doxas et al., 2008b). These responses indicated that most students experience some sort of challenge when it comes to conceptual-level understanding of evolutionary processes. None of the three questions used focused directly on natural selection, but rather examined their understanding of evolutionary processes in a general way. We note that most introductory-level courses do not focus on natural selection, and many do not explicitly cover evolutionary processes at all. Unfortunately, the precise nature of students' difficulties were unclear in terms of their responses to the essay questions. What could be concluded was that students often write about evolutionary processes in contradictory ways and that there were several patterns among their responses. This indicated that we needed to conduct thematic interviews on the subject to discover whether or not this was an area that needed to be explored further in the context of introductory courses in the biological sciences (often the only courses future K–12 science teachers take). Through these thematic think-aloud interviews we sought to discover more about the meaning of the language students used to discuss their understanding of evolutionary processes. In addition, we included an open-ended question that asked students to explain more about natural selection, how it works, and its relationship to evolution. A total of 28 thematic student interviews were conducted on this and related topics.

The interviewer (KGD) is a social scientist, rather than a biologist, a point she made explicit to all interviewees. To reinforce the impression that she was not an “expert” in the biological sciences, and so unlikely to be judgmental, she often asked clarifying questions about simple biological terms and processes; and, as interviews progressed, her questions were placed in the context of ideas she had read or heard about from someone else. This strategy leads to increased student comfort during the interview process and ensures, as much as possible, that students feel free to talk, expand, and discuss.

Questions in thematic interviews are not structured and vary from interview to interview, in response to the relationship and communication patterns that develop between the interviewer and student. In general the questions used took a format similar to: “I was reading something in an essay the other day and it made me wonder more about natural selection. I'm confused about how that relates to evolution [or bottlenecks; or genetic drift; etc]. Would you explain a little more about that to me?” The point of this question format is to provide a context for the student to speak openly during the interviews without worrying about being judged, or whether they are or are not correct. Listening to a student explaining ideas in a relaxed, face-to-face setting is one of the most effective means for coming to an understanding of what is “inside” students' heads, what they really mean when they select a particular distracter, how they interpret questions, etc.

Analysis of the essay response and interview data focused on determining whether or not a particular topic was an area that should be addressed in the BCI. In the specific area of evolutionary processes, it also provided hints on how to best discover (on a broader scale than interviews allow) students' fundamental, introductory-level misconceptions. Our first-round essay and interview data confirmed that most students confounded natural selection and evolution; they were able to define each, but unable to explain the relationship between them in terms of differences—they assume that random events lead to natural selection and that natural selection is equal to evolution. Some were able to define bottlenecks and genetic drift correctly, but many could only say, “we covered that [in high school], but I don't really know what they are.” I think they're … “; those who were more familiar with the terminology could still not accurately explain the relationship between natural selection and phenomena such as bottlenecks and genetic drift. In other words, they clearly believed that these random phenomena were related to evolution because they cause natural selection and natural selection equals evolution—they see all evolution as a product of natural selection alone. If a direct, fact-based question was posed, students could usually respond correctly. If a different type of question, one asking students to combine ideas or consider relationships, was posed, even more advanced students experienced a great deal of difficulty—particularly with regard to evolution and natural selection.

This process enabled us to better understand students' conceptual beliefs, but we quickly realized that we required additional data. We used our first-round observations to design new questions focused on this conceptual area and we attempted to explore student understanding of the idea of randomness. Analysis of their responses to the questions, combined with structured, think-aloud interviews convinced us to pursue this topic further. We therefore designed and administered additional essay questions.

Student Examples

Random events and evolution (Note, the responses have been corrected for spelling but not grammar).

PIECES OF STUDENT UNDERSTANDING: DIFFUSION

Concurrent with this phase of the iterative data-collection process, we realized that our students were experiencing related conceptual difficulties with diffusion. The presence of such problems has been discussed previously by Odom and colleagues (Odom, 1995; Odom and Barrow, 1995). In addition to classroom observations, we developed supplemental questions for our development cycle. One of these questions dealt explicitly with diffusion: What is diffusion and why does it occur?

Student Examples – Diffusion

The last response was one of the very few responses that acknowledged the role of random molecular motion. The majority (> 95% of approximately 100) of responses are typified by the other examples, where diffusion is viewed as directional movement that takes place only when some kind of gradient exists. There is no apparent appreciation displayed that random processes can give rise to emergent behavior, such as net directional movement of molecules. During interviews, both structured and unstructured, this picture was reinforced. When asked directly a question about diffusion, students could tell us that it was random—a rote response. However, when they were asked questions about a particular process that involved diffusion, they were unable to tell us that the underlying process was random. So, when given an example, the idea of random disappeared from their understanding of diffusion.

Essay responses to this question were consistent with some of the ways in which students characterized mutations (particularly during interviews), in the sense that one thing they tend to fail to mention about mutations is that they occur all the time, and randomly. In both cases, while students were busy explaining about, and listing characteristics of, diffusion and mutation, they consistently failed to include that either held any random component. It was not so much a matter of what they did say, but about what they consistently left out.

STUDENT UNDERSTANDING AND EVOLUTIONARY PROCESSES

This pattern remained consistent during follow-up think-aloud questions, so we added open-ended questions about random. We asked things like, “Is this a random process?” and “What sorts of things are random [in the context of molecular interactions]?” We continued to probe and began asking more advanced student interviewees why it was that they did not consider biological processes to have a random component.

Eventually, we came to three conclusions:

1. The teaching of biology focuses to such a high degree on the notion that biological systems are extremely efficient and represent essentially perfect adaptations (perhaps an echo of William Paley's argument from design). In part this may reflect many teachers' emphasis on the power of adaptation versus the less “design-centric effects” of other evolutionary processes. At the same time, the notion of efficiency is so foreign to student understanding of random processes that they simply “could not go there” with their thinking. They believed that the idea of random is fundamentally inconsistent with the efficiency of biological systems. Even students with college physics, who could describe diffusion correctly in that context, could not make the transfer to biological systems. That randomness is a difficult concept is a point made in the context of “real world” by Nassim Taleb in his book Fooled by Randomness (2005).

2. Our students live in a highly socialized world where evolution is portrayed in popular literature and movies exclusively in terms of “survival of the fittest.” This same exclusively adaptionist perspective is explicitly or implicitly assumed by a number of practicing molecular biologists, particularly those not familiar with the lessons of population genetics (see below). With students, the problem lies with the conflation of fitness with strength. The rhetoric we use in society focuses on the idea that “only the strongest survive.” In addition, students' talk about evolution focuses on “adapt or disappear,” with no room for underlying random processes, i.e., evolution must be a directional movement driven by natural selection, just as diffusion is directional movement driven by concentration gradients. Such a view implies meaning to traits that may be attributable to nonadaptive processes. There are, in fact, data from studies by population geneticists that genetic drift and related random events play a key role in molecular evolution, creating and disrupting regulatory sites, etc. (see Lynch and Conery, 2003; Lynch, 2006; Yi, 2006; Brockhurst, 2007; Lynch, 2007). This implies that learning in the biological sciences must overcome the prevalent “anti-random” bias, so that students come to appreciate that in biological systems survival means that the species has managed to reproduce successfully, and that over time differential survival leads to evolutionary change. Whereas students may respond appropriately to questions on this point when placed in “rote” situations, when asked questions at the conceptual level, they tend to revert to social rhetoric about what fitness means.

3. In his review of morphogen gradients, Lander (2007) makes the important point that not only students, but many working molecular biologists do not understand how random movements associated with diffusion can act to establish signaling systems, often confusing a macroscopic “ballistic” view of movement with a microscopic “diffusive” one. Taken together with the misunderstanding of genetic drift and its role in gene and genome evolution (see above), this is further evidence that directly addressing issues of emergent behavior arising from random processes is of importance in understanding molecular biology and evolutionary dynamics.

RESPONSE TO TEACHING

Through the use of the BCI it is possible to examine the effects of instruction on student views of random processes. Here we consider responses to two questions (25 and 30) that relate to this topic. In response to question 25 (Imagine that you are an ADP molecule inside a bacterial cell. Which best describes how you would manage to “find” an ATP synthase so that you could become an ATP molecule?) We see little improvement in recognition of the correct response, even after two or three “conventional,” in Hake's terminology (1998), i.e., lecture-based biology courses (Figure 1). In contrast, the response to question 30 (How is genetic drift like molecular diffusion?) does show some modest improvement over time, although the majority of students entering the fourth course in the molecular, cellular, and developmental biology (MCDB) sequence still answer it incorrectly. In preliminary studies, we have seen improvement in responses to both questions in a course that relies heavily on interactive engagement, i.e., Biofundamentals (Klymkowsky, 2007). One part of the Biofundamentals curricula involves the use of a virtual laboratory that deals directly with diffusive movements. As part of this lab (freely accessible on the web: http://virtuallaboratory.net/Biofundamentals/labs/WaterDiffusionMembranes/InWater.html) students are asked to explore the nature of molecular motions, consider efficiency of diffusion as a driver of molecular motion as a function of distance, and begin to examine how these motions can be described mathematically. Similarly, exercises from a Web-based java applet (http://darwin.eeb.uconn.edu/simulations/jdk1.0/drift.html) can be used to drive students to consider allele loss or fixation as a function of population size, a particularly fruitful topic when considering evolutionary events in small population, such as likely to be associated with speciation events. Although these results are preliminary and need to be supported by further study in terms of their reproducibility and generality across various curricula, they do suggest that conceptual improvements in this area are possible through directed activities—similar conclusions have been reported by Meir et al. (2005).

CONCLUSIONS

In the process of researching student misconceptions in biology for the construction of the BCI, we discovered that a class of student difficulties impacting genetics, molecular, and evolutionary biology appear to arise from a fundamental misconception about random processes. Students carry the underlying belief that random processes are inefficient whereas biological systems are extremely efficient, and are therefore loath to ascribe macroscopic biological phenomena to random underlying processes. They seek alternative rational explanations, the dominant one being the existence of drivers. Although they correctly name random processes when asked about them in isolation, when asked to explain complex behavior they always resort to drivers, in the absence of which the complex behavior stops. Thus evolution is directly driven by natural selection alone, and diffusion is directly driven by density gradients alone (and both stop when the driver is not present). The concept of an underlying random process that is taking place all the time giving rise to emergent behavior is almost totally absent from their explanations.

REFERENCES