How Do You Like Your Science, Wet or Dry? How Two Lab Experiences Influence Student Understanding of Science Concepts and Perceptions of Authentic Scientific Practice

INTRODUCTION

The contributions of authentic research experiences to students’ recognition and comprehension of a wide range of science practices have been widely established in the science education community (Driver et al., 1996) and have been called out in A Framework for K–12 Science Education (hereafter referred to as the Framework; National Research Council, 2012), which underlies the Next Generation Science Standards (NGSS; NGSS Lead States, 2013). The science practices highlighted in the NGSS include, among others, planning and carrying out investigations, analyzing and interpreting data, constructing explanations, and engaging in argumentation from evidence.

Closely allied with the practices of science woven throughout the NGSS is the importance of providing students a means of understanding science as both a way of knowing and a body of knowledge—an understanding of the nature of science (NOS). “Engaging in the practices of science helps students understand how scientific knowledge develops” (National Research Council, 2012, p. 42). The NOS has been defined and discussed extensively in the science education research literature and generally includes the following aspects: 1) science is both empirical and inferential; 2) scientific knowledge is reliable but tentative; 3) scientific investigation requires creativity; 4) scientific knowledge is partly subjective (theory-laden) because of the beliefs, prior knowledge, training, and currently accepted theories of scientists; 5) the laws of science are different from scientific theories; 6) science reflects society and culture; and 7) scientists use many methods and practices to develop scientific knowledge (for descriptions of the NOS, see Lederman and Lederman, 2004; Abd-El-Khalick, 2012).

The science education research community is divided on the usefulness of instruction in the NOS at the K–12 level, in part due to lack of a unified view of the NOS among scientists (Wong and Hodson, 2009). In addition, some argue that the NOS described above does not comprise a useful framework to guide K–12 students as they develop their understanding of science concepts and practices (Elby and Hammer, 2001). In the research described in this paper, we did not set out to teach the NOS, although several questions asked during the discussion groups probed student understanding of aspects of the NOS as well as the practices of science as defined in the Framework and the NGSS. Many of the responses made by students during the focus groups reflected a growing understanding of the NOS, especially after completing both research experiences. For this reason we will include discussion of the NOS throughout this paper and point out examples of student comments related to their understanding of the NOS.

As discussed by Wong and Hodson (2009, p. 110), “the goal of improving NOS understanding is often prejudiced by stereotyped images of science and scientists consciously or unconsciously built into school curricula and perpetuated by science textbooks.” In contrast to the controlled experiment typically taught in precollege classrooms, Windschitl and colleagues point out,

Thus, as educators strive to expand students’ images of science, it is critical to immerse the students in authentic research experiences that more closely resemble the practices conducted by scientists, while identifying and expanding upon student perceptions of how science is carried out.

In many biology courses, teachers have relied on molecular wet lab experiments to provide context and hands-on experiences for high school students. Though effective, these wet lab experiments often require expensive laboratory reagents and equipment and a substantial time commitment from teachers, limiting access to these experiences. Increasingly, scientists carry out “dry lab” research using publicly accessible computer tools and databases, and these resources have potential for immersing secondary-level science students in authentic research practices as an alternative or addition to traditional molecular biology. This approach has been used successfully in a few innovative biology units at the high school, undergraduate, and graduate levels (Campbell, 2003; Rice et al., 2004; Kumar, 2005; Shaffer et al., 2010; Kovarik et al., 2013). However, the use of authentic online tools and databases in high school science courses does not necessarily support students in developing an expanded understanding of inquiry or the NOS, as described in a study by Waight and Abd-El-Khalick (2011). For students to optimize their ability to conduct meaningful research using scientific tools and databases, they need to understand related scientific concepts; the context, purpose and history of the tools and/or database; and the process and limitations for their use.

The gold standard for engaging students in authentic research has been through multiweek internships in science labs, which have proven to be a powerful tool for promoting student science, technology, engineering, and mathematics (STEM) interest and learning (Seymour et al., 2004; Hunter et al., 2007; Lopatto, 2007, 2010; Sadler and McKinney, 2010). However, relatively few such research opportunities exist for high school students. As an alternative approach, we have developed two linked science education programs that enabled students to participate in authentic research as part of their biology courses under the direction of their teacher. These two multiyear projects involved students in contributing to an ongoing epidemiological study of smoking behavior developed specifically for classroom research. This approach allowed hundreds of students to participate in research each year through a classroom experience that lasted approximately 2 weeks.

Through the foundational project in which the smoking behavior study was designed and implemented, we worked with high school teachers, scientists, biostatisticians, and bioethicists to design a research investigation of smoking behavior and develop a curriculum that enabled high school students to contribute as scientists to aspects of the research (Munn et al., 2010). The smoking investigation used a case control study design to compare ∼150 case subjects who were adult regular smokers and ∼150 control subjects who were adult nonsmokers who had tried smoking but never became regular smokers. The investigation included the use of a questionnaire to collect information about a wide range of environmental factors to which research subjects may have been exposed when they first tried smoking and the collection of a small blood sample used for genotyping three candidate gene loci that were selected based on association with smoking in the literature.

At the start of the foundational project, high school students contributed to the study design by writing questions related to environmental factors that they thought might influence becoming a smoker, and their questions were integrated into the research questionnaire. Later, after the study design had been approved by the University of Washington (UW) Institutional Review Board (IRB) and subject recruitment had begun, students from subsequent classes contributed by genotyping subject DNA through an in-class wet lab. Aspects of the smoking study that were not appropriate for students to conduct, including subject recruitment, consenting, blood draws, questionnaire collection, and DNA isolation, were conducted by professionals at the UW. The questionnaire and genotyping data were entered into the queriable smoking behavior database. The completed smoking behavior database became the centerpiece for a new classroom-based research program called Exploring Databases (Munn et al., 2013) in which students could propose and test their own research questions using the genetic and environmental data for ∼300 research subjects contained in the database.

In the study described in this paper, we examined how two different research experiences influence students’ perceptions and understanding of the practices of scientists, how scientific research is conducted, and related scientific concepts. Students carried out 1) the wet lab used to genotype human DNA at three loci associated with smoking behavior, as conducted in the foundational project; and 2) a dry lab in which they used the smoking behavior database to test their own hypotheses about factors that affect smoking behavior. Classrooms were assigned to two possible sequences to complete the lessons and activities so the influence of each research activity could be evaluated. Through their participation with the project, students learned about genetics and neuroscience, developed skills in formatting and testing hypotheses and applying scientific reasoning, and used information and communication technology. The two research experiences do not follow the experimental design traditionally taught in high school textbooks, thus helping to broaden student understanding of how research is conducted, what kinds of research questions scientists ask, and the multiplicity of approaches used by scientists to pursue the research questions that interest them.

This study addresses the following questions:

• To what extent do students learn science concepts as a result of the two research experiences?

• Does this curriculum influence student attitudes toward ­science?

• Do students perceive wet lab genotyping and dry lab database experiences to be “authentic science” to the same extent?

METHODS

Overview of Study

The Exploring Databases curriculum used for this study is described in Table 1 and is provided in the Supplemental Material. In lesson 1, students were exposed through videotaped interviews to the many methods used by scientists from different fields to conduct their research (as opposed to a single “scientific method”). Lessons 2–5 guided students in developing their own theoretical and disciplinary knowledge related to smoking behavior and case control studies through examination of smoker profiles and lessons on genetic influences on smoking behavior, the case control study design, and relevant statistical analyses. Through instructions in lesson 6, students used the database to test their own research questions. Two lessons from the curriculum of the foundational project were used for the genotyping experiment, activity D3, in which students modeled genotyping using a paper and scissors activity, and activity D4, which provided the genotyping protocol (Munn et al., 2010; provided in the Supplemental Material).

The curriculum was implemented by teachers in western Washington during the 2010–2011 school year. Implementation from start to finish took 2–3 weeks of classroom time. All teachers completed the lessons as described herein, although they each made some modifications to customize for the learning needs of their students.

The overall structure of the study is shown in Figure 1. All classrooms involved in this study completed the six lessons of the Exploring Databases curriculum, including the database research and the genotyping experiment. Classrooms were assigned one of two different orders to present the curriculum so we could measure the impact of each type of research experience independently and after students had completed both. In sequence 1, referred to as GTDB (genotyping → database), teachers presented lessons 1–4 of the database curriculum, modeled and performed the genotyping experiment, and then followed with lessons 5 and 6. In sequence 2, called DBGT (database → genotyping), the six lessons of the curriculum were presented sequentially, followed by the genotyping modeling activity and experiment. Thus, all students experienced the same lessons and research experiences, but in a different order. Students completed pre and post surveys at the beginning of the curriculum and after completing both lab experiences. The survey included 10 science content questions to probe student understanding of science concepts and the Scientific Attitude Inventory II (SAI II; Moore and Foy, 1997; see Table 3 later in this article) to measure their attitudes toward science. A subset of randomly selected students participated in a focus group after each of the two research experiences to probe students’ understanding of the practices of science as it pertained to the two research experiences.

Wet Lab Research Experience.

Students worked in teams to genotype DNA from research subjects from the original epidemiology study (two to three class periods to complete activities D3 and D4, as described in Table 1). They analyzed polymorphic sites in the three candidate genes for smoking behavior already included in the smoking behavior database, and their data verified the previous data for those subjects. The three genetic loci had been selected by the project design team based on their association with aspects of smoking behavior published in previous scientific publications (Gelernter et al., 1998; Duan et al., 2003; Yu et al., 2006). Teachers guided their students in using the protocol developed in the foundational project (Munn et al., 2010; provided in the Supplemental Material) and equipment and reagents provided by this program. Students followed the reverse dot-blot protocol (analogous to microarray technology), first binding PCR-amplified DNA from a single subject to DNA probes bound to a nylon membrane and then staining the membrane so they could visualize the positions where subject DNA had bound. After conducting the experimental process, student groups interpreted their own results and compared their findings with those of other groups to determine the most accurate genotype for each subject. Their teachers submitted their data to program staff for final verification and entry into the database.

Database Research Experience.

Working in teams, students developed a research question and related hypothesis about how specific genetic or environmental factors might influence becoming a persistent smoker (two to three class periods to complete lessons 5 and 6, as described in Table 1). Through the database, students were able to query the responses of case and control subjects to many of the 86 questions included in the research questionnaire, as well as subjects’ genotypes for the three genetic loci included in the study. Students selected up to four items in the database that were related to their research question, with each item being a question from the research questionnaire or one of the three genetic loci. Typically, case control studies use a statistic called an odds ratio to compare the odds that case subjects were exposed to a particular factor with the odds of control subjects being exposed. For each odds ratio, a 95% confidence interval is also calculated to determine statistical significance. Guided by directions embedded in the database and curriculum, students set up an odds-ratio calculation for each item of interest by defining the parameters for each query, used the database software to calculate the odds ratio and confidence interval, and then applied the criteria typically used by epidemiologists to determine whether the association was likely to be causal (Bradford Hill, 1965). Student teams presented their results and analyses through a research poster or PowerPoint presentation to their classmates, who provided feedback and critiques.

Participants

IRB Approval for Research Study.

Before beginning this research, we applied for and received IRB approval for research conducted with teachers and students through the UW Human Subjects Division (#38651).

Teacher Participants.

Teachers consenting to participate in the study were assigned to one of the two orders of curriculum implementation, such that similar student numbers, grade levels, and courses were represented in the two groups. At the end of the study, the classrooms of six teachers (out of 11 who ­initially signed up for the study) had completed both research experiences, had valid data for pre and post surveys, and allowed us to collect focus group data after each research ­experience.

Table 2 shows the teachers (numbered to maintain anonymity) and courses for the two study conditions and the number of students by grade level in each course. Teachers received training on the genotyping process during a weeklong teacher ­professional development (PD) workshop of the foundational project or during a 1-day genotyping workshop offered at the start of this study. The six teachers also participated in a 3- or 4-day PD workshop focused on implementation of the Exploring Databases curriculum. During the workshop, teachers experienced the lessons as their students would, discussed strategies for implementing each lesson, and participated in informal discussions with invited speakers on topics such as “Learning and Equity,” “The Use of Databases in Genome Research,” and “Research on Complex Traits: Ethical and Scientific Issues.” They also participated in a 1-day review workshop on the database research before the start of this study.

Student Participants Providing Pre–Post Survey Data.

At the end of data collection, the majority of students (87%), including all the 10th graders, were in the GTDB condition, as shown in Table 2. Because the distribution was so uneven, we did not attempt to make comparisons of student performance pre to post based on sequence of delivery. Instead, data from students in both sequences were combined for analysis of overall changes pre to post on content questions and SAI II items. To be eligible for inclusion in the survey analysis, subjects were required to have both pre and post SAI and content assessment data and to have a score for each SAI subtest and an answer to all 10 content assessment items. The number of students in each class that met the above criteria is given in parentheses in Table 2.

Student Participants Providing Focus Group Data.

The six teachers randomly selected five to eight students from each class to participate in the focus group. Focus groups were held midcurriculum after completion of the first research experience and postcurriculum after completion of the second research experience.

A total of 86 mid- and 85 postcurriculum focus group worksheets were collected from students participating in focus groups, including students in grades 9, 10, 11, and 12. Seventy students participated in both the mid- and postcurriculum focus groups and provided worksheets for each. Sixteen students participated only in the mid focus group but not the post focus group, while 15 students participated only in the post but not the mid focus group. Thus, focus group data came from a total of 101 unique students (70 + 16 + 15).

Data Collection for Pre–Post Survey

Before the curriculum implementation began, students completed a pre survey (see the Supplemental Material) that included 10 scientific content items and a vetted scientific attitude survey. The same survey was administered after implementation of the second lab experience.

Pre–Post Survey Scientific Content Items.

The content portion of the survey consisted of 10 items developed by the curriculum designer and evaluator. Of these, four were multiple choice and the remaining were open-ended. To simplify scoring, a value of 1 was assigned to each correct response, with no partial points given. The open-ended items were scored by two external content experts who were trained by the lead evaluator in the use of the scoring rubric. After 10 tests were scored as a group and discussed, a sample of 32 tests were pulled and scored independently by the scorers. Scores for each item were compared across all 32 tests. For each item a “1” was indicated if both scorers agreed and a “0” if they disagreed. The total agreement score for each test was computed by summing for each item and then dividing by the number of items. Agreements ranged from 50% to 100% across all tests. The percent agreement (82.7%) was calculated by averaging the percent agreement across all tests. The Cronbach’s alpha for reliability for the pre and post content items were 0.701 and 0.793, respectively. Content items had a high degree of face validity, because they were drawn from the curriculum.

Pre–Post Survey Scientific Attitude Inventory.

We included the SAI II (Moore and Foy, 1997; see Table 3) in the pre and post survey to test whether there was any change in students’ attitudes toward science as a result of the curriculum. The SAI II was integrated into a post survey administered to teachers during the 2011 Exploring Databases summer workshop and was also administered informally to 17 UW scientists (unpublished data). SAI scores were computed for each of the 6 SAI subtests and converted to a percentage for each item.

The SAI II is based on the SAI (Moore and Sutman, 1970) but was revised to include fewer attitude statements and simpler, gender-neutral language. The SAI II includes the same 12 position statements as the original inventory. These were divided into six subscales, each with one positive attitude toward science and a matching statement expressing the opposing view. For each position statement, there are either three or five attitude statements (40 total). Responses are given using a five-point Likert scale ranging from “strongly agree” to “strongly disagree.” Attitude statements that best supported each position statement were determined by a panel of 10 expert judges for the original SAI, and this scoring was maintained for the revised inventory.

The first five subscales of the SAI II encompass many aspects of the NOS, including the tentative and empirical nature of scientific knowledge, the theory-laden aspect of conducting scientific research, the need for creativity/imagination in generating scientific ideas, and the reflection of society and culture in science. The last column of Table 3 cross-references the NOS items described in the Introduction to each of the first five subscales. The final subscale is focused on students’ personal reflections on whether they would want to be a scientist. This instrument was selected for this study because it probed a wide range of students’ attitudes about the NOS as well as their personal interest in being scientists, was easily comprehended and completed by students, and was easy to score. In previewing the 40 questions included in the survey, we anticipated that we might see a change in student attitudes as a result of the enacted curriculum and research experiences.

As shown in Table 3, position statements marked with an “A” are meant to be positive, while those with a “B” are negative (position statements quoted from Moore and Foy, 1997). Item choices associated with the positive subscales are scored 5–1 while item choices associated with negative items are scored 1–5 for the five possible responses ranging from “strongly agree” to strongly disagree.” Total scores for each subscale were calculated by adding total items of both positive and negative items. Results were analyzed at both the subscale and individual item levels to identify changes from pre to post administration. Moore and Foy (1997) calculated a reliability statistic (Cronbach’s alpha = 0.805) using data from 557 subjects in the development of the SAI II, and our reliability calculations for alpha (pre: 0.792; post: 0.826) yielded similar results.

Data Collection for Focus Groups

A focus group protocol was developed to probe more deeply into the impact of the genotyping and database experiences on student perceptions about science. As described earlier, focus groups of five to eight students were held twice during the curriculum implementation after each research experience.

During each visit, the researcher observed the classroom and then held a focus group either after class, at lunch, or after school. Each focus group consisted of two parts: 1) a focus group worksheet and 2) a facilitated discussion using a set of guiding questions.

Focus Group Worksheet.

Students completed the worksheet individually at the start of the focus group to collect quantitative data and to focus student attention on the discussion issues (provided in the Supplemental Material). Components of the worksheet are summarized below:

• The first item asked students to list words, concepts, and images of a “typical” scientist at work. Using this mental image as a point of reference, students then placed a rating on a seven-point semantic differential scale to indicate how the lab that they just completed was similar (or not) to their ideal image of what scientists do.

• Students were then presented with 13 task descriptions typically associated with scientific inquiry and were asked which of these tasks they had performed in the lab activity just completed.

Facilitated Focus Group Discussion.

After students completed the focus group worksheet, the discussion was started by the researcher. The responses to the worksheet items were discussed in detail and analyzed for common themes and unique perspectives. As time permitted the researcher used several discussion questions (provided in the Supplemental Material) to elicit more detail from the students about how each type of lab influenced their thinking about the scientific enterprise.

RESULTS

Results for Pre–Post Survey

Content Items (Items 14–24 of the Survey).

Figure 2 shows the percentage of students scoring correctly on each of 10 content items. The questions corresponding to each item are provided in the Supplemental Material. Overall, increased percentages of students from pre to post were able to correctly answer each content item, as shown in Figure 2. The increase was significant (p < 0.05) for all questions except 15 (purpose of controls) and 21 (explanation for their hypothesis in previous question). Item 24 (Explain your answer using math) had the lowest number of correct responses on both the pre- and posttests compared with the other questions, although 6.5 times as many students answered correctly after the intervention than before.

Scientific Attitude Inventory.

A high score on the SAI II reflects a positive attitude toward science, as defined by the panel of 10 expert judges who developed the scoring guide for the original SAI (Moore and Sutman, 1970). As shown in Figure 3, from pre to post there were no significant differences in the paired means for any of the six SAI subscales for students (p > 0.05). Teachers scored higher than students on all six of the subscales. Teachers and students scored lowest and nearly the same on SAI 4, which has items dealing with the purpose of science: knowledge building (positive subscale) versus technology development for practical applications that serve mankind (negative subscale).

Results for Focus Groups

Focus Group Worksheet Analysis

Which Research Experience Did Students Rate as Being More Like “Real” Science?

Regardless of the order in which they had completed the research experiences, students rated the genotyping activity higher in terms of being more like “real” science than the database activity (means = 5.3 and 4.94, respectively, on a seven-point scale). This result (t test for independent samples) is significant at the p < 0.05 level (p = 0.026). While the total effect size is relatively small, it nonetheless supports the general stereotyped view of science held by many students that science is more about manipulation of equipment and tools to generate data than about analysis of data to generate ideas (see Focus Group Discussion Analysis below).

As part of the worksheet, students were given a list of tasks associated with scientific research and asked to indicate whether or not they had the opportunity to engage in each task for both the database and genotyping activities. The results are presented in Table 4. Students reported that, on average, they engaged in 9.5 of the 13 scientific tasks presented on the worksheet during the database activity and 8.3 tasks during genotyping. This difference is significant at the p < 0.01 level (p = 0.001; t test for independent samples). The magnitude of the difference is small (∼1 additional task indicated for the database activity) but illustrates that students recognize the scientific tasks that they engaged in during the database activity, even though they generally felt that genotyping was more like real science.

As shown in Table 4, a significantly higher percentage of students (p < 0.05 level) associated the following inquiry tasks with the database activity compared with the genotyping activity:

• Came up with a research question to answer

• Came up with at least one hypothesis to test

• Designed a way to test your hypothesis

• Analyzed data using qualitative methods such as describing results using words

• Analyzed data using quantitative methods such as numbers and statistics

• Came up with a conclusion based on your data analysis

• Came up with an answer to your question

• Justified your conclusions to others

One task, “Used the same kind of ‘tools’ and ‘equipment’ that scientists use,” was rated higher for the genotyping activity by a significantly larger percentage of students than the database activity. The focus group discussion analysis confirmed that students are more likely to perceive the tools and equipment of scientists to consist of items such as lab coats, beakers, and test tubes than computers and databases.

How Did Students Rate Their Understanding and Interest in the Two Research Experiences?

While there were no significant differences between how students rated their understanding of each lab type (nearly 90% indicated that they understood each totally or somewhat), a higher percentage of students indicated that they “totally” understood the database lab (34.7%) as compared with genotyping (26%). Similarly, a larger proportion of students indicated that they understood genotyping not very much or not at all (12.5%) as compared with the database activity (8.9%). In general, there was more understanding of what students were doing in the database lab than in the genotyping lab, although the percentages are high for both types of labs. A nearly equal proportion of students indicated that they were interested in both types of labs. A quarter reported that they were totally interested, and more than half were somewhat interested in the labs. About a fifth rated their interest as not at all or not very much. The type of lab does not appear to have much influence on the interest level of students toward the lab.

Focus Group Discussion Analysis.

Each focus group discussion was recorded and transcribed. The researchers read each transcript and identified common and unique responses to each question. Responses were then compiled and organized by focus group occurrence (mid or end of curriculum) and by curriculum sequence condition (GTDB or DBGT). The results for three questions are discussed below.

Did Completing Both Research Experiences Influence Student Images of Scientists and What They Do?

A summary of GTDB and DBGT responses based on both the worksheets and discussions are found in Tables 5 and 6, respectively.

Students in the GTDB condition were asked to describe their mental images of a scientist on two occasions: just after the genotyping lab and just after they had completed the database activity. Responses fell into six general categories: setting, materials, physical description, cognitive profile, science processes, and topics investigated. After the genotyping lab but before the database activity (Table 5, column 1), GTDB students generated a stereotypical view of scientists in action: in a lab wearing a white coat and goggles, using pipettes, and mixing chemicals in test tubes. The scientists they described generally looked like Einstein and were smart, curious, thoughtful, and concerned about the future. The activities students attributed to scientists included creating hypotheses, conducting experiments, dissecting animals, recording data, and making new discoveries.

During the second focus group, just after the database activity (Table 5, column 2), GTDB students held onto the first focus group visions of the materials, setting, and physical descriptions of scientists. However, some students reported a broader understanding of science, reflecting that scientists not only discover new knowledge but examine and test existing knowledge; scientists explore a variety of factors and variables to generate hypotheses; and scientists are learners. A question about what scientists consider as they do their work was introduced and discussed.

Students in the DBGT group were also asked what their mental images of a scientist were on two occasions: just after the database lab and just after they had completed the genotyping activity. The first focus group, described in Table 6, column 1, yielded responses similar to those of the GTDB group. After the second focus group, summarized in Table 6, column 2, DBGT students added that computers were used more than they had previously thought, and one group added that, through the use of computers and the database, they were able to do more high-level research than might otherwise have been possible. Another group discussed the variety of methods used by ­scientists to answer similar questions. They also introduced technological science and psychology for the first time in the discussions.

When asked what influenced their images of scientists, student responses typically reflected their own experiences rather than the immediate influence of either of the two research experiences they conducted for this study. Common responses included their parents or other adults they know, the school lab, and depictions of scientists in movies and on television, including Bill Nye the Science Guy.

Regardless of the order of completion, students provided more insightful descriptions of scientists after they had completed both research experiences compared with only one. They also demonstrated a deeper appreciation for the roles of computers and databases in scientific research after both experiences, independent of the order in which they completed them.

How Did Students Perceive the Similarities and Differences between the Genotyping Lab and What “Real Scientists” Do?

This question was asked of both groups right after they had finished the genotyping activity. Students gave a semantic differential rating on the focus group worksheet and then were asked to explain why they gave that rating. Table 7 presents a summary of responses. These responses came from both first and second focus groups, depending on when students had just completed the genotyping activity. No students rated or explained how genotyping was not like their ideas of what scientists do. Students equated the activities of science with doing something “hands-on.” Genotyping was similar, because they followed a process (like scientists do), took on the role of a scientist, participated in a “real” study, and explored an open-ended question. Scientists would actually do this lab, although on a larger scale. Reasons why genotyping was not as similar included the reduced time students had to solve the problem, that scientists might deal with different subject matter, that genotyping was a recipe lab (not open-ended or driven by student questions), and that it was only a small window into what scientists do. Overall, students indicated that the genotyping lab was mostly similar to real science.

How Did Students Perceive the Similarities and Differences between the Database Lab and What “Real Scientists” Do?

Again, this question was asked just after the database activity, and their responses are summarized in Table 8. Many students saw the database activity as just computer work, not hands-on, as group work (not individual, in the way they imagined scientists would work), and not consistent with their stereotypical views of what scientists look like or do. The responses for why the database activity was somewhat similar to what scientists do reflected a view that it was similar in some respects (had to follow a procedure, used the same equipment) but it was not “stressful” enough, not a complete experiment, and did not allow students to obtain and work with their own data. Surprisingly, students held these views even though they had seen videos in which scientists discussed the highly collaborative and multifaceted aspects of their research, including using computers and databases containing data collected by others. The student comments that the database research was “just computer work” and not “stressful” enough may reflect the design of the database query, in which students set up the odds-ratio calculation by defining exposures for cases and controls, but the database software calculated the odds ratio and 95% confidence interval. This design was intended to focus students on the analysis of their queries rather than the potentially confusing mathematical aspects, but it may have had the unintended consequence of oversimplifying the process to the extent that students did not equate what they were doing as being real science. Responses in column 3 of Table 8, “Why DB Is Completely Similar,” were provided entirely by students in the GTDB group after they had completed both experiences. Many of these students were able to provide reasons for why the database activity was completely similar to what scientists do. Students suggested that it was real science, but a scaled-down piece of it with fewer variables and not as complex. It was completely similar because it involved trying to prove a hypothesis using research and statistics, and it consisted of looking for effects of interactions on human characteristics, like psychology research.

What Were Student Views on the NOS after Completing Both Research Experiences?

During each focus group discussion, students were asked several questions that probed their understanding of the NOS as time allowed. Their responses are summarized in Table 9. Overall, students indicated that scientific knowledge is reliable but tentative and partly subjective, dependent on the background of the scientists, and that scientific investigation requires creativity. In response to questions about the purpose of science and how scientists achieve that purpose, most discussions focused on using the scientific method or testing to answer questions we do not know, making discoveries, or proving something. Most groups supported the view that the purpose of science is to help society or find cures. Only one student disputed this view.

What Common Themes Emerged during Focus Group Discussions?

The following ideas were discussed during several focus groups:

• Students valued both types of research because the answer was not known and they were contributing to a real research study.

• Many students equated genotyping with “doing science,” while the database portion was “doing research” similar to what they might do in an English or history class.

• Students in several groups reflected on the role of databases in science, both in formulating research questions as part of developing a project and as part of data analysis. One student commented on the fact that data analysis is the part of science that is often not depicted in the news or popular media.

• Some students considered the database research to be “just an observational study.” However, other students discussed the necessity of conducting an observational study instead of a randomized control study because of the ethical implications of assigning human subjects to a condition such as smoking that is known to be harmful to human health.

• Students had different reactions to the database research. In one focus group, there was a consensus reaction that the causes of smoking were already well understood, so the ­students were not discovering anything new. These students were generally not interested in the topic of smoking. In contrast, students in other focus groups indicated genuine interest in and surprise about their database analyses, whether their results showed strong association or were “right in the middle.” One student remarked that “all results tell a story.”

• Students discussed their frustration with several aspects of the database, including the small sample size, lack of diversity, and the fact that they could not contribute their own questions to the questionnaire. One group expressed a desire to combine questions in their query (i.e., to be able to calculate the odds of having two variables, X and Y). This feature was deliberately not included in the data analysis, because there would be insufficient power for two questions to be combined given the sample size.

• Students expressed some frustration with the genotyping experiment, including lack of understanding of what was happening at each step, the high level of precision needed, not being able to design their own protocol (although several students pointed out that they did not have the background to do so), and not knowing more about the research subject whose DNA they were analyzing, such as whether they were a case or a control in the study.

• In several classes, students indicated that they felt rushed when doing each of the research experiences and wished they had more time.

DISCUSSION

This study examined how two kinds of authentic research experiences, a wet lab genotyping experience and a dry lab database experience, influence students’ perceptions and understanding of scientific research. This discussion addresses our initial questions.

To What Extent Do Students Learn Science Concepts as a Result of This Curriculum?

Students showed significant learning gains (p < 0.05) on most of the 10 questions in the content survey (except items 15 and 21) after they had completed the curriculum. Almost 70% of students scored correctly on item 15 before and after completing the curriculum, with no significant increase. This item asked about the role of controls in scientific research, a topic most high school students would have studied throughout their ­science courses, explaining the high score in the pretest. The small gain made pre to post on item 21 was not statistically significant at the p < 0.05 level. In this question, students were asked to explain the hypotheses they had created for the previous question, essentially to provide explanations for the claims they had made. In our own experience, students often have difficulty articulating how data support their claims, which may explain the small gain for this question.

The lowest percentage of correct responses occurred for item 24, in which students were asked to use math to explain their interpretations of some study data (and were expected to use an odds ratio in their responses). It is not surprising that very few students scored correctly on the pretest, because the odds ratio is not normally taught in high school. However, only 20% of students were able to provide the correct response after the curriculum. This may reflect the fact that students were not required to calculate the odds ratios for their own queries—this was done by the database software to avoid calculation errors—although they did calculate an odds ratio in an earlier lesson. For this curriculum, the emphasis was on interpretation of the odds ratio (and the 95% confidence interval) rather than setting up the calculations. If making these calculations is an important part of a teacher’s instructional design, then more emphasis would need to be placed on this aspect.

Nearly 90% of all students reported that they partially or totally understood the content of the curriculum as it pertained to both the database and genotyping content. This suggests that students were confident in their learning gains, which is consistent with the results of the post survey.

Does This Curriculum Influence Student Attitudes Toward Science?

There was no significant difference in student attitudes toward science before and after the curriculum as measured by their responses to the SAI II included in the pre and post surveys. Even on the pretest, students had relatively high scores for each SAI item (the lowest mean score was 60% and the highest was greater than 80%). Within the context of this 2-week intervention, there may have been insufficient time and specific instruction to change students’ broad perceptions of science. On all items, teachers scored higher than students, and the scientists who were informally surveyed had scores very similar to the teachers.

Both teachers and students scored lower on SAI 4 than other position statements. Disaggregation of this statement into ­positive and negative components showed that students and teachers (as well as scientists) tended to agree with the positive statement, “Science is an idea-generating activity. It is devoted to providing explanations of natural phenomena. Its value lies in its theoretical aspects.” However, they also tended to agree with the negative statement, “Science is a technology-developing activity. It is devoted to serving mankind. Its value lies in its practical uses,” which resulted in a lowering of the overall score for this statement. The fact that students, teachers, and scientists all tended to agree with the negative statement may reflect a shift in the way our society views the role of science compared with when this test was originally designed in 1970. For example, we now group science with technology, engineering, and mathematics in the common educational acronym, STEM. Furthermore, there has been a change in how science is conducted in academic settings, which is promoted by federal funding agencies, with more emphasis placed on solving problems related to technical and medical issues. Student responses during focus groups also supported a frequently expressed view that the purpose of science is to help society (see Table 9).

In contrast to the results of the SAI II, analysis of the focus group discussions indicated that students developed more nuanced views of the nature of scientific research after completing both of the lab activities, demonstrated by their reflections on the theory-laden aspects of scientific research and the multifaceted approach that scientists use (column 2 of both Tables 5 and 6). Student responses to specific questions related to the NOS (Table 9) demonstrate a general grasp of the features of the NOS among students who participated in the focus groups. Their level of understanding may in part be attributed to participation in this study as well as other experiences through the science class they were taking. Many aspects of the NOS were integrated into student activities conducted through lessons 1–4 of the Exploring Databases curriculum used in this study, although they were not explicitly presented in the context of the NOS. As discussed by others, students might have better recognized the NOS elements they were conducting had they been versed explicitly in what constitutes the NOS (Lederman and Lederman, 2004; Sadler et al., 2010).

Do Students Perceive Wet Lab (Genotyping) and Dry Lab (Database Experiences) to be “Authentic Science” to the Same Extent?

While the students rated both research experiences to be like real science, they gave a higher rating to the genotyping experiment than to the database research experience. Furthermore, students were more likely to indicate that genotyping “used the same kinds of ‘tools’ and ‘equipment’ that scientists use.” The perception that the wet lab experience is more like real science may reflect how science is most commonly portrayed in the public media, which typically highlights the more visually appealing data-collection aspects of science. In several focus groups, students distinguished between doing experiments and analyzing data, which they equated with the kind of literature research they would do for an English or history project. However, students associated more scientific inquiry tasks with the database experience than the genotyping activity (Table 4). They also reported a better understanding of what they were doing in the database lab, which may reflect the more structured directions for completing the database research than the genotyping protocol.

In the parlance of Chinn and Malhotra (2002), the genotyping experiment most closely resembles a “simple experiment” in which students are provided the research question, follow a given protocol with variables and control group specified for them, and are told what to measure. The authentic aspect of this experiment is that they do not know what the outcome is for the sample they are analyzing, so they must draw their conclusions based on their own results and then verify their findings by comparing their data with those of other student groups in their class who are analyzing the same samples. In addition, the genotyping data they produced are included in a bona fide research database as part of an ongoing research study. In contrast to genotyping, the database research is most closely aligned to “authentic inquiry” as described by Chinn and ­Malhotra (2002), because students generate their own research questions within the context of what can be asked using the smoking behavior database and select their variables by defining parameters for “exposed” and “not exposed” when calculating their odds ratios. They also present their research to their fellow students and defend or alter their claims in response to critiques from their colleagues.

Focus group results suggest that the participation in both types of scientific inquiry broadens student understanding of the tools, processes, and approaches used in doing science. While students tend to hold fast to their stereotypical visions of scientists and what they do, many students were able to articulate an expanded perspective that science is accomplished in many ways and that a case control study, for example, is another legitimate method of generating scientific knowledge. Moreover, the content of the database—real data about smokers and nonsmokers along with their environmental and genetic exposures—made the database inquiry activity interesting, with the feel of an authentic science activity. The same perspective was shared about the genotyping activity—students were excited to be working with genetic data from real people. Thus, after both research activities, students identified aspects of the NOS embedded within their experiences, including the multifaceted approaches to doing research in a particular field and the significance of doing research in an area that has personal and/or social significance, in this case smoking behavior.

Limitations of the Study

While results of the curriculum implementation are encouraging, there are cautions that should be acknowledged. In any human research project, the assigning of teachers and students to condition groups is challenging. Some teachers were unable to implement the curriculum as initially planned or to provide times for complete data collection. The length of time that teachers spent on the curriculum was determined by the demands and opportunities that faced each teacher. The final distribution and numbers of students in each condition group were not as balanced as we had intended because of incomplete participation from some classrooms. In addition, while we tried to match at the classroom level, it is possible that some results were influenced by unaccounted variables. We addressed some of these issues by collecting a variety of data types in order to triangulate our findings.

CONCLUSIONS

In spite of their own recognition that the database lab incorporates more scientific tasks than the wet lab, some students persisted in maintaining a stereotypical view of science based on the use of tools and equipment to collect data as in the wet lab. This has been reported by others (Driver et al., 1996). However, after completing both lab experiences, students expressed more sophisticated views of the nature of scientific research, including the theory-laden aspects of science and the multiplicity of approaches used by scientists to conduct their research. This outcome highlights the importance of immersing students in many kinds of research experiences through their precollege education so they can develop a richer understanding of how scientists conduct their research compared with the very procedural scientific method traditionally taught in science textbooks (Windschitl et al., 2007).

Through the two science education programs described in this paper, we developed research experiences that could be integrated into a wide variety of high school courses, including introductory biology, so that students at all levels could be involved. The two research experiences included in this study each present advantages and disadvantages for integration into a science class. The genotyping experience enabled students to use some of the classical tools and techniques of molecular biology to generate authentic data for the ongoing research study. For this application, it was critical that students follow a standardized protocol to ensure data quality, and the experience did not have the flexibility for students to pose research questions or design their experiments. However, the opportunity to use those tools and techniques and contribute data to the study was clearly an experience students valued. In contrast, the database experience allowed students to pose and test their own research questions, albeit in the context of a larger research study that they had not designed using data that they had not collected, and again, they placed a high value on the experience. In both cases, students could take ownership of the research they conducted because of their unique contributions to the study, an important aspect of developing interest in science, as discussed by others (Polman and Pea, 2001; Hanauer and Dolan, 2014).

The proliferation of publicly accessible scientific databases and data-analysis tools presents exciting opportunities to engage students in authentic scientific research using pre-existing data, similar to what many research scientists do (Hug and McNeill, 2008). As teachers strive to engage their students in meaningful activities that reflect scientific practice as recommended in the NGSS (NGSS Lead States 2013), the use of databases is a viable option. To increase students’ recognition of database research as a scientific practice, we argue that the science education community will need to address the following questions:

• How can we foster students’ understanding that manipulation and analysis of data collected by others is a valid research activity conducted by practicing scientists?

• How can we help teachers develop the necessary conceptual understanding to guide students in recognizing the validity of different approaches in science?

We have attempted to address these issues through a curriculum that exposes students to the many practices of scientists and immerses them in activities that foster engagement in science practices, including developing their own theoretical and conceptual framework for conducting research on smoking behavior and designing research questions that they can test using the smoking behavior database. To broaden their teachers’ understanding of science practices, teacher professional development workshops provided opportunities for teachers to engage in discussions of scientific research with scientists and science educators. The understanding of science practices in the context of discipline-specific content and cross-cutting concepts by all students is an important goal of the NGSS that should guide future curriculum development and classroom practice. On the basis of our results, we recommend that teachers and students have opportunities to engage in science practices that reflect the unique approaches taken in different fields and to participate in deliberate discussions of those approaches so they develop a more nuanced view of the practices and nature of science.

ACCESSING MATERIALS

The current Exploring Databases teacher professional development website, curriculum, PowerPoint presentation, and the smoking behavior database can be downloaded from https://gsoutreach.gs.washington.edu/instructional-materials/exploring-databases.

The curriculum from the foundational project, Investigating the Effects of Genes and Environment on Smoking Behavior, from which the genotyping protocol is taken, as well as a related ethics unit and all PowerPoint presentations, can be downloaded from https://gsoutreach.gs.washington.edu/instructional-materials/investigating-smoking-behavior.

ACKNOWLEDGMENTS

REFERENCES