ASCB logo LSE Logo

The C.R.E.A.T.E. Approach to Primary Literature Shifts Undergraduates’ Self-Assessed Ability to Read and Analyze Journal Articles, Attitudes about Science, and Epistemological Beliefs

    Published Online:https://doi.org/10.1187/cbe.11-03-0027

    Abstract

    The C.R.E.A.T.E. (Consider, Read, Elucidate hypotheses, Analyze and interpret data, Think of the next Experiment) method uses intensive analysis of primary literature in the undergraduate classroom to demystify and humanize science. We have reported previously that the method improves students’ critical thinking and content integration abilities, while at the same time enhancing their self-reported understanding of “who does science, and why.” We report here the results of an assessment that addressed C.R.E.A.T.E. students’ attitudes about the nature of science, beliefs about learning, and confidence in their ability to read, analyze, and explain research articles. Using a Likert-style survey administered pre- and postcourse, we found significant changes in students’ confidence in their ability to read and analyze primary literature, self-assessed understanding of the nature of science, and epistemological beliefs (e.g., their sense of whether knowledge is certain and scientific talent innate). Thus, within a single semester, the inexpensive C.R.E.A.T.E. method can shift not just students’ analytical abilities and understanding of scientists as people, but can also positively affect students’ confidence with analysis of primary literature, their insight into the processes of science, and their beliefs about learning.

    INTRODUCTION

    As scientific information continues to accumulate at a rapid pace, there is a growing sense among science educators that long-established practices need to be reconsidered. Numerous 21st-century science reform documents (American Association for Higher Education, 2000; U.S. Department of Education, 2000; National Research Council [NRC], 2003; Malcom et al., 2005; Alberts, 2005; NRC 2007, 2009; American Association for the Advancement of Science [AAAS], 2010) suggest focusing less on content coverage and more on approaches that reveal science to be an ongoing creative process. Ideally, such a change would help to stem the long-standing attrition of bright students from science majors and, by extension, science research careers (Seymour and Hewitt, 1997; Cech and Kennedy, 2005; DePass and Chubin, 2009). At the same time, student attitudes toward learning (student epistemologies; Schommer 1990, 1993), student self-efficacy (confidence in ability to work effectively in a particular context; Lawson et al., 2007), and student attitudes about science (Osborne, 2003) have been demonstrated to be important factors affecting students’ success in the science classroom. Thus, both changes in how science is taught and consideration of factors influencing students’ ability to learn deserve focus in science education reform efforts.

    A variety of new approaches that employ alternatives to lectures, including hands-on classroom activities and small-group work (Klionsky, 1998; Handelsman et al., 2004; Allen and Tanner, 2005; Knight and Wood, 2005), highlighting controversy to stimulate student engagement (Seethaler, 2005; Campion et al., 2009), student participation in ongoing grant-funded research projects (Hanauer et al., 2006; Call et al., 2007; Lopatto et al., 2008; Clark et al., 2009), case study approaches (Herreid 1994a; Chaplin, 2009), use of the popular press (Strauss, 2005; Hoskins, 2010), and analysis of primary literature (Herreid, 1994b; Janick-Buckner, 1997; Lynd-Balta, 2006; Kozeracki et al., 2006; Hoskins et al., 2007; Hoskins, 2008; Schinske et al., 2008; Yarden, 2009), shift classroom focus from a teacher-centered situation in which students are largely passive, to a student-centered classroom (Freeman et al., 2007; Klymkowsky, 2007; Armbruster et al., 2009; Hoskins and Stevens, 2009) more supportive of cognitive activities associated with learning (Bloom and Krathwohl, 1956; Chickering and Gamson, 1987; Zull, 2002).

    We have focused on primary literature as a portal into the scientific research process through the C.R.E.A.T.E. (Consider, Read, Elucidate hypotheses, Analyze and interpret data, Think of the next Experiment) method. C.R.E.A.T.E. uses intensive critical analysis of a series of papers generated sequentially from one lab, coupled with email interviews of paper authors via a survey of student-generated questions, to demystify and humanize science. The approach is an iterative method, whereby individual steps of the process (Figure 1 and Table 1) provide students an organized approach to individual journal articles that are “dissected” using a series of novel or adapted pedagogical tools in preparation for intensive class discussion. Papers from a single lab are read in series (Figure 1), allowing students to follow the arc of a research project as it actually progressed. Students are not provided with the full series of papers in advance, nor with the titles, authors, abstracts, or discussions of the papers under consideration. While students could Google the missing information, doing so is ultimately more a hindrance, blunting creative thought, than a help. We encourage students to instead treat the course as a process of discovery. By working with a suite of novel or adapted pedagogical tools to prepare for class, students are empowered to participate actively in the lab-meeting atmosphere of the class sessions, where figures and tables are examined individually, and the logic of the overall study is examined. Previous work has documented precourse versus postcourse shifts in C.R.E.A.T.E. students’ critical thinking and content integration abilities, as well as changes in self-assessed attitudes about science and scientists, as determined by postcourse interviews and the Student Assessed Learning Gains instrument (Hoskins et al., 2007). We report here the results of a survey designed to examine additional aspects of students’ attitudes, beliefs, and self-assessed abilities, comparing responses pre- and post-C.R.E.A.T.E. course.

    Figure 1.

    Figure 1. Overview of the C.R.E.A.T.E process. Papers 1–4 form a “module”—a series published by the same lab group as they followed a particular question in sequential studies. See Table 1 for details of each step of the C.R.E.A.T.E analysis. Variations on this approach, for example, discussing papers from different lab groups with conflicting data, or using the method in shorter-term analysis of newspaper/Internet reports of science studies, are also effective (see Hoskins, 2008, 2010).

    *Defining/discussing grant panel criteria is done during this iteration only.

    Table 1. Overview of C.R.E.A.T.E. steps and associated activities, many of which are carried out by students in preparation for classa

    C.R.E.A.T.E. stepStudent activities
    ConsiderConcept map paper introduction, note topics for review, define new issue(s) to be addressed, begin defining relevant variables and determining their relationships.
    ReadDefine unfamiliar words, annotate figures, create visual depictions (sketch “cartoons”) of the individual substudies that underlie each figure or table. Transform data presented in tables into a different format (graph or chart).
    Elucidate hypothesesFor each figure, define the hypothesis being tested or question being addressed by the work that generated the data illustrated. Rewrite the title of each figure in your own words.
    Analyze and interpret the dataUsing the hypotheses, questions, cartoons, diagrams, and charts and/or graphs, determine what the data mean. Fill in a data analysis template for each figure to track the logic of each experiment and prepare for class discussion. After all figures and tables have been analyzed, create a concept map for the paper, using each illustration as a map node to reveal the logic of the study design.
    Think of the next ExperimentConsider: “If I had carried out the studies described in this paper, how would I follow up?” Design two distinct studies, and cartoon one on a transparency for in-class discussion (see Student grant panels, below).
    Additional C.R.E.A.T.E. classroom activities
    Student grant panelsStudents work in small groups first to define criteria panels “should” use in allocating funding. After these are discussed by the whole class, students view all of the student-designed experiments, then return to small groups to evaluate the proposed studies, with the goal of reaching consensus on the one that most merits funding.
    Email surveys of authors of papersThroughout the semester, students are encouraged to jot down questions that arise regarding “the research life” or the researchers themselves. Late in the semester, 10–12 of the questions are compiled into a single survey and emailed to each paper author. Responses from authors (60–75% response rate) reveal novel behind-the-scenes insights.

    aModified from Hoskins (2010, Table 1); see Hoskins et al. (2007) for additional details on each step and the overall process.

    Students’ beliefs about learning and knowledge affect their ability to learn and their application of metacognitive strategies, including their integration of prior knowledge with the task at hand, studying for understanding rather than superficial recall, and assessing what they do and do not comprehend. Such approaches can significantly facilitate students’ understanding of science (Hartman, 2002; Schraw et al., 2006; Pulmones, 2010). Students’ attitudes about the nature of knowledge, for example, whether knowledge can change over time, and their attitudes about intelligence, for example, whether it is innate and fixed or malleable, comprise a set of epistemological beliefs that affect learning and understanding (Schommer, 1990). Students’ epistemological understandings are typically less sophisticated than those of their professors (Hogan and Maglienti, 2001), and these views can affect study approaches, as well as the extent to which students persist in challenging tasks (Schommer, 1993, 1994; Hofer, 2004). Reasoning ability has also been linked to epistemological beliefs, with students whose epistemologies are more sophisticated showing enhanced skills (Zeineddin and Abd-El-Khalick, 2010).

    As the constructivist C.R.E.A.T.E. method uses a number of activities and pedagogical tools (Table 1) designed to increase both student engagement and metacognition, we hypothesized that students’ attitudes toward primary literature, the practice of science, and the nature of learning might change during the semester. We developed a questionnaire aimed at assessing students’ self-assessed views about science, scientists, the research process, and aspects of learning, and administered it on the first and last days of the 14-wk semester. Analysis of student responses indicates that C.R.E.A.T.E. students shifted significantly in their understanding of numerous aspects of scientific research, their approach to reading scientific literature, their confidence in their ability to understand science, and, perhaps most interesting, their epistemological beliefs. Because naïve epistemological beliefs can affect students’ study approaches, learning gains, and ability to interpret complex scientific information (Schommer, 1990; Kardash and Scholes, 1996; Pulmones, 2010), shifting student epistemology can be a first step toward developing attitudes toward knowledge and learning more supportive of student success. In this regard, it is notable the changes reported here were achieved during a single semester in a course that essentially costs nothing to implement and does not involve a hands-on laboratory component.

    METHODS

    Participants in the study were students in an upper-level elective at the City College of New York (CCNY). The class met twice weekly for a total of 140 min (2005; three credit hours) or 200 min (2006–2009; four credit hours) per week. Class size averaged 27 students (range: 19–32). Seven iterations of the course are included in this study. Most students were junior or senior biology majors who had completed the course prerequisites at CCNY: a year of introductory biology, and one semester each of genetics and cell/molecular biology. A few students (<10% of each class) were participants in the CCNY post-baccalaureate program. These students had earned degrees in other fields and returned to college to complete premed requirements. In the seven classes represented, 65% of students were female and 61% were African American, Hispanic, or Native American, all groups currently underrepresented at all levels of academic science (National Science Foundation [NSF], 2002, 2008; Atwell, 2004).

    Presurvey/Postsurvey of Student Attitudes and Self-Rated Abilities

    On the first and last days of the semester, students filled out an anonymous Likert-style survey aimed at elucidating their degree of agreement/disagreement with a series of statements. They also answered several open-ended questions on the survey. The survey statements focused on attitudes and beliefs about issues the C.R.E.A.T.E. approach was designed to address, including students’ self-rated ability to understand and analyze primary literature; whether primary literature had influenced their understanding of science; students’ understanding of the scientific research process; and students’ self-rated science reading ability, confidence in their ability to “think like a scientist,” understanding of “scientists as people,” and sense of whether research science was an appealing career choice. We designed the survey based on our experiences teaching from primary literature in previous classes, focusing on a variety of issues we had determined to be problematic for previous students. Open-ended questions requiring written answers focused on students’ understanding of the activities undertaken by research scientists were set aside for later analysis. This survey was administered in each C.R.E.A.T.E. class (two sections per year in 2005 and 2008 and one section per year in 2006, 2007, and 2009). The 2005 cohort of students was included in a previous analysis of the effects of the C.R.E.A.T.E. class on student critical thinking, content integration, and attitudes toward science/scientists (Hoskins et al., 2007).

    Surveys were anonymous and coded with numbers known only to the students themselves, to allow alignment of data for within-subject statistical analysis. The survey included 52 statements to which students reacted by marking “I strongly agree,” “I agree,” “I’m not sure,” “I disagree,” or “I strongly disagree” on their survey sheets. Some sample statements were phrased positively (e.g., “I could make a simple diagram that provided an overview of an entire experiment”) and others negatively (e.g., “I do not have a good sense of what motivates people to go into research”).

    Three additional propositions were aimed at eliciting students’ overall sense of their ability to read/analyze journal articles, their understanding of the nature of science, and the extent to which primary literature had helped them to understand the nature of science (A, B, C). Question C had four possible responses ranging from “no influence” to “major influence,” and all other questions had five possible responses, phrased in parallel to the question posed (e.g., for question B, on understanding of the nature of science, possible answers ranged from “no understanding” to “understand it very well”). Postcourse, all surveys for which both “pre” and “post” copies were available were scored on a five-point scale, with “strongly agree” = 5 and “strongly disagree” = 1. The additional questions were scored in a parallel way, with scores for C ranging from 1–4 rather than 1–5.

    RESULTS

    The C.R.E.A.T.E. Survey

    The C.R.E.A.T.E. survey included a collection of seven summary items (Table 2) and 52 specific skill and attitude items deemed relevant to the course goals. Following the accrual of 140 cases for the summary items and 155 cases for the skill and attitude items, the data were used to improve the usefulness of the survey. Initial inspection of the 52 skill and attitude items revealed some items were repetitious and others were unrelated to other items (low communalities) or to summary items. These items were set aside, leaving 38 candidates for continued analysis. Of the 38 items, 13 were similar to items used in research on epistemological attitude (Schommer, 1990) and attitude toward science. These are described below (Tables 5, 6, and related text). The epistemological items were a sampling of items developed by Schommer (1990) as part of a much broader investigation of epistemological beliefs. The items here were drawn from across the factors derived from Schommer's survey, and so include items representing the belief that knowledge is certain (e.g., that different scientists will come to similar conclusions) and that ability is innate (e.g., that scientists were born with a special talent), as well as items assessing attitude toward science (e.g., science is a creative endeavor). Because Schommer's previous work showed that these epistemological beliefs do not constitute a single scale, we did not include them in exploratory factor analysis, but analyzed them separately. The remaining 25 items were analyzed by means of a principal component analysis (PCA) to explore underlying factors that might aid in the reduction of the 25 variables to a more manageable set. The PCA was performed on 25 skill and attitude items, with a varimax rotation to aid with interpretation. The resulting analysis yielded eight factors accounting for 64% of the variance in the data; however, some variables were “split” across factors, resulting in two factors that were uninterpretable. These were set aside. The remaining six factors, with their member items and factor loadings, are shown in Table 3. The first factor, which we name Decoding Primary Literature, includes items that refer to scientific language and scientific literature, and indicates the respondent's feelings about reading primary scientific literature. The second factor, Interpreting Data, includes items that have to do with data presented in tables and graphs, as well as with data transformations. The third factor, Active Reading, includes items about diagrams, displays, and method. Visualization, the fourth factor, includes visualizing the method of a study and interpreting graphs. The fifth factor is named Thinking Like a Scientist and includes items about explaining a scientific paper and thinking of experiments. The final factor is called Research in Context and includes items about animal models and controls in experiments.

    Table 2. Seven summary items used on the C.R.E.A.T.E. survey

    On a scale of 1–5, rate your confidence in your ability to read and analyze science journal articles.a
    On a scale of 1–5, rate your understanding of “the way scientific research is done” or “the scientific research process.”b
    When was the last time that you read an article from the primary scientific literature (e.g., a journal article)?
    How many articles from the primary scientific literature (e.g., journal articles) have you read?
    How much influence have journal articles had on your understanding of science?
    Outline the path from a scientist's initial thoughts to a completed research study in a published journal article. Please be as detailed and complete as you can.
    Journal articles are (choose the single best answer) a) hard to read and not worth the effort, b) hard to read but worth the effort, c) easy to read but not worth reading, or d) easy to read and worth reading.

    a For this item, 1 = zero confidence, 2 = slightly confident, 3 = confident, 4 = quite confident, and 5 = extremely confident.

    b For this item, 1 = I don't understand it at all, 2 = I have a slight understanding, 3 = I have some understanding, 4 = I understand it well, and 5 = I understand it very well.

    Table 3. Items from the C.R.E.A.T.E. survey arranged according to a PCA with varimax rotationa

    FactorItemFactor loadingCronbach's alpha
    1The scientific literature is difficult to understand (R).0.776
    Decoding Primary LiteratureWhen I see scientific journal articles, it looks like a foreign language to me (R).0.593
    I am not intimidated by the scientific language in journal articles.0.558
    I am confident in my ability to critically review scientific literature.0.5000.71
    I am comfortable defending my ideas about experiments.0.328
    2 Interpreting DataIt is easy for me to transform data, like converting numbers from a table to percents.0.796
    If I see data in a table, it is easy for me to understand what it means.0.680
    If I am shown data (graphs, tables, charts), I am confident that I can figure out what it means.0.6220.72
    It is easy for me to relate the results of a single experiment to the big picture.0.352
    3 Active ReadingI could make a simple diagram that provides an overview of an entire experiment.0.763
    If I am assigned to read a scientific paper, I typically look at the methods section to understand how the data were collected.0.584
    I do not know how to design a good experiment (R).0.5220.63
    The way that you display your data can affect whether or not people believe it.0.345
    4 VisualizationWhen I read scientific information, I usually look carefully at the associated figures and tables.0.694
    When I read scientific material it is easy for me to visualize the experiments that were done.0.6490.75
    If I look at data presented in a paper, I can visualize the method that produced the data.0.592
    When I read a paper, I have a clear sense of what physically went on in a lab to produce the results and information I am reading.0.584
    5 Thinking Like a ScientistAfter I read a scientific paper, I don't think I could explain it to somebody else (R).0.735
    I am confident I could read a scientific paper and explain it to another person.0.655
    I enjoy thinking of additional experiments when I read scientific papers.0.3940.59
    I accept the information about science presented in newspaper articles without challenging it (R).0.231
    6 Research in ContextExperiments in “model organisms” like the fruit fly have led to important advances in understanding human biology.0.774
    Progress in curing diseases has been made as a result of experiments on lower organisms like worms and flies.0.5970.35
    I understand why experiments have controls.0.540

    a Items followed by an (R) are reverse-scored. Cronbach's alpha, an index of inter-item consistency, is also shown.

    Pretest–Posttest Differences

    Raw scores for the items in each factor were summed, resulting in six pretest scores and six posttest scores for each student respondent. A paired-difference t test was performed on each of the six factors. The results are shown in Table 4. Each pretest–posttest difference is highly significant in the expected direction of posttest gains. The magnitude of the change, estimated as standard deviations in the final column of Table 4, is medium to large (Cohen, 1988). This magnitude of change, as well as the stringent level for significance testing, argues against the presence of a type I error (spurious significant differences).

    Table 4. The results of paired-difference t tests for raw data totals for each of the six factors in Table 3

    FactorPretest mean (SD)Posttest mean (SD)Statistical significanceMean difference/SD of the differencea
    115.5 (3.6)19.2 (2.9)p < 0.0010.93
    213.6 (2.5)16.4 (2.1)p < 0.0011.00
    313.6 (2.2)16.2 (2.4)p < 0.0010.84
    413.2 (2.5)15.8 (2.3)p < 0.0010.96
    513.5 (2.3)16.2 (2.1)p < 0.0010.97
    612.6 (1.7)14.0 (1.3)p < 0.0010.74

    aEstimate of the magnitude of the effect.

    Epistemological Beliefs

    Table 5 shows 13 items related either to Schommer's constructs of certain knowledge and innate ability or to a general attitude toward science. It would be expected that students with an insightful attitude toward science would believe that knowledge is not certain and unchangeable; that scientific ability is not fixed and innate, and that science is a creative and collaborative endeavor. To explore the change in these variables from pre- to posttest, the items under the certain knowledge heading were summed (Cronbach's alpha = 0.66 for this scale). Similarly, the two items for innate knowledge were summed. The attitude items were examined individually. The result of paired-difference t-tests for pre- versus postcourse data are shown in Table 6. There were significant positive gains on all the variables. The possibility of the presence of a type I error is reduced by the stringent level for significance and the moderate effect sizes.

    Table 5. Items from the C.R.E.A.T.E. survey that measure epistemological beliefsa

    Knowledge is certain.
    If two different groups of scientists study the same question, they will come to similar conclusions. (R)
    The data from a scientific experiment can only be interpreted in one way. (R)
    Because scientific papers have been critically reviewed before being published, it is unlikely that there will be flaws in scientific papers. (R)
    Because all scientific papers are reviewed by other scientists before they are published, the information in the papers must be true. (R)
    Sometimes published papers must be reinterpreted when new data emerge years later.
    Results that do not fit into the established theory are probably wrong. (R)
    Ability is innate.
    I think professionals carrying out scientific research were probably straight-A students as undergrads. (R)
    You must have a special talent in order to do scientific research. (R)
    Attitude toward science.
    Science is a creative endeavor.
    I have a good sense of what research scientists are like as people.
    I do not have a good sense of what motivates people to go into research. (R)
    Scientists usually know what the outcome of their experiments will be. (R)
    Collaboration is an important aspect of scientific experimentation.

    a Items followed by an (R) were reverse-scored for analysis.

    Table 6. The results of paired-difference t tests for items (certain knowledge, innate ability, and attitude toward science) in Table 5

    ItemPretest mean (SD)Posttest mean (SD)Statistical significanceMean difference/SD of the differencea
    Certain knowledge19.7 (2.2)20.7 (2.7)p < 0.0010.40
    Innate ability7.5 (1.7)8.1 (1.5)p < 0.0010.36
    Creativity4.1 (0.85)4.4 (0.73)p < 0.0010.30
    Sense of scientists3.1 (0.93)3.8 (0.77)p < 0.0010.70
    Sense of motives3.6 (0.95)4.0 (1.0)p < 0.0010.31
    Known outcomes4.0 (0.82)4.3 (0.81)p < 0.0010.30
    Collaboration4.4 (0.73)4.6 (.66)p < 0.0060.22

    aEstimate of the magnitude of the effect.

    Relationships to Summary Item of Reading Confidence

    In an effort to synthesize the findings, we chose to ask whether any of the components of the C.R.E.A.T.E. survey identified so far relate to the student participant's overall view of his or her ability to read science articles. For this purpose, we set the data for the first summary item (rate your confidence in your ability to read and analyze science journal articles) as a dependent variable for multiple linear regression. Specifically, the postcourse confidence data were analyzed using a multiple regression analysis, with predictors including precourse responses to the same item (pretest reading confidence), the sums of the raw data scores indicated by the factor groupings in Table 3 (postcourse data), and the epistemological items in Table 5 (postcourse data). A stepwise multiple regression analysis indicated that a model with three significant predictors, the scores relating to Decoding Primary Literature, the first factor in Table 3 (standardized coefficient 0.33; p < 0.01); the scores relating to Thinking Like a Scientist, the fifth factor in Table 3 (standardized coefficient 0.27; p < 0.01); and the pretest reading confidence level (standardized coefficient 0.17; p < 0.02), significantly predicted posttest reading confidence (r = 0.58; df = 3, 126; p < 0.05).

    DISCUSSION

    Experiences That Shift Students’ Attitudes about Science and Their Own Scientific Abilities

    The one-semester C.R.E.A.T.E. course enhanced students’ confidence in their ability to read, understand, and explain science, as well as to understand how research is carried out (Tables 3 and 4). Students changed significantly on summary variables that assessed self-rated ability to design experiments, visualize methods based on data, visualize lab activities based on the written account in the journal article, manipulate data, relate results of individual experiments to “the big picture,” critically review data, read science with appropriate skepticism, and explain results to others. These topic areas touch on a wide range of activities undertaken by working researchers.

    Our survey findings are consistent with a model that predicts that students’ overall confidence in their ability to read and analyze journal articles is based largely on their sense that they can: 1) Decode Primary Literature (Table 3, factor 1) and 2) Think Like a Scientist (Table 3, factor 5). The gains in students’ confidence in these abilities are likely related to the design of the C.R.E.A.T.E. course. Class discussion and homework assignments challenge students to work through the experiments and interpret the findings of published studies as if they had done the work themselves. During the semester, students carry out numerous activities typical of working scientists. As such, the approach may promote more epistemological engagement than does a standard class, thus engendering changes in students’ attitudes and beliefs about science.

    Overall, the approaches typical of the C.R.E.A.T.E. classroom are likely to encourage students to employ metacognitive strategies that help them interpret complex information (Hartman, 2002; Nordell, 2009). Gaining deeper understanding of challenging material may in turn help students overcome widely held misconceptions about research science, for example, that experiments are done to demonstrate concepts already known, and therefore research is not a creative activity, and that researchers are simply fact-gatherers (Sandoval, 2003). Students’ repeated opportunities to design and evaluate experiments and models throughout the C.R.E.A.T.E. semester (Figure 1 and Table 1) may contribute to the shifts we noted in their views about the creativity of science (Tables 5 and 6). Developing their own questions for paper authors (e.g., “Do you have to be a straight-A student to become a researcher?” “What would be your ‘dream discovery’?”) encourages students to think beyond the data of the papers and consider the overall process of becoming and being a scientist.

    Currently, undergraduate research experiences (UREs) are considered to be one of the most important mechanisms for stimulating students’ interest in science careers. The effects of UREs have been investigated using surveys of students’ experiences (Rauckhorst et al., 2001; Lopatto, 2004a, b, 2007, 2009; Russell, 2006; Russell et al., 2007) and extended interviews (Hunter et al., 2007). These studies have reported high student enthusiasm for UREs, as well as major benefits for multiple aspects of students’ understanding of science, their hands-on research skills, and their attitudes toward research careers. In a longitudinal ethnographic study of the effects of UREs on students at four liberal arts institutions, participants noted in interviews that UREs increased their research confidence and sense of “feeling like a scientist” (Hunter et al., 2007). Students who participate in science UREs are likely to already be interested in scientific research, and UREs clearly reinforce the aspirations of these students. In this respect, C.R.E.A.T.E. may reach a broader group of students, including many who have not previously considered careers in science.

    The concept of “nature of science” includes, for many science educators, the ideas that scientists build understanding on observations of nature, that explanations and understanding can change over time, and that creativity comes into play throughout the research process (Lederman, 1992; Karakas, 2009). While there is general agreement that students need to understand the nature and processes of science, or “science as a way of knowing” (AAAS, 1993, p. 2; see also AAAS 1989, 2010; NRC 2009), it is less clear how to accomplish this goal. Teaching approaches focused on inquiry have been suggested as a way to build student understanding of the nature of science (Aulls and Shore, 2008; Shore et al., 2008) and enhance learning (Quitadamo et al., 2008). “Inquiry” alone, however, may not be sufficient to shift students’ concepts of the nature of science (Lederman et al., 1998; Sandoval, 2003; Schwartz et al., 2004) or to encourage students to use metacognitive approaches when studying science (Butler et al., 2008). Our study finds that, although the C.R.E.A.T.E. course does not include a laboratory component or independent research projects, students nevertheless report substantial changes in attitudes and beliefs about science during the semester. Being challenged to devise their own research questions, analyze and interpret data, design experiments, and carry out peer review of studies devised by other students may stimulate C.R.E.A.T.E. participants to examine their personal beliefs about science. In addition, student interview data suggest the email survey of authors plays a role in shifting students’ understanding of “who does science, and why?” (see Tables 1 and S1 in Hoskins et al., 2007). In this context, it is notable that students participating in a novel Deconstructing Scientific Research course, which focuses on intensive analysis of an individual research seminar and also lacks a hands-on component, showed large gains in multiple categories addressed by the Survey of Undergraduate Research Experiences instrument (Lopatto, 2004a, 2007, 2009), including the ability to “understand how knowledge is constructed” (Clark et al., 2009).

    Epistemologies, Learning, and the Nature of Science

    Our survey also addressed aspects of students’ epistemological beliefs. Schommer and colleagues (Schommer, 1990; Schommer et al., 1992) have identified epistemological beliefs that moderate learning in a variety of intellectual domains. For example, the beliefs that knowledge is certain, that authority should be trusted, that learning is quick and simple, and that intellectual talent is innate can interfere with striving to learn. The C.R.E.A.T.E. program, by uncovering the process of scientific thinking and by providing contact between students and professionals, may influence these epistemological beliefs in a beneficial way. We found substantial changes during the semester in students’ views of “scientists”; moderate shifts in students’ sense of whether knowledge is certain and ability is innate, the creativity of science, or understanding of motives that drive scientists; and a small shift in students’ views of science as a collaborative activity (Tables 5 and 6).

    Undergraduates’ epistemological beliefs shift during the college years from a sense that knowledge is certain, typical of freshmen, to a more nuanced view of the relative nature of knowledge, held by seniors (Perry, 1970), and a longitudinal study suggests such views continue to change postcollege (Baxter Magolda, 2004). Epistemological beliefs change slowly during the college years (Perry, 1970), and only a minority of students achieve mature epistemological understanding by senior year (Baxter Magolda, 1992). For both high school (Schommer, 1993) and college (Schommer, 1990, 1993; Hofer, 2000, 2004) students, the sophistication of their epistemological beliefs correlates with their reading comprehension and academic performance, with naïve beliefs linked to lesser achievement. Epistemological beliefs also affect student metacognition (Hartman, 2002; Schommer-Aikins, 2002; Hofer, 2004), ability to integrate information (Schommer, 1993), and persistence when confronted with a challenging task (Dweck and Leggett, 1988). The interrelationships among personal epistemologies, metacognition, and learning are complex, but there is general agreement that naïve epistemologies may interfere with learning (Hofer, 2004; Pulmones, 2010). Overall, students with naïve epistemologies employ fewer of the metacognitive strategies (e.g., setting goals, monitoring progress, self-questioning, and connecting new information to broader concepts) that support self-directed learning (Zimmerman, 1990; Hartman, 2002; Pieschel et al., 2008; Strømsø et al., 2008).

    The C.R.E.A.T.E. method's combination of epistemically challenging approaches applied in an authentic context may underlie the changes we saw in students’ epistemological beliefs. Several aspects of the C.R.E.A.T.E. approach present students with novel cognitive challenges and associated epistemic load. C.R.E.A.T.E. students employ visualization when sketching cartoons that fill the gap between the methods section and the charts, graphs, blots, and/or photomicrographs presented. Integrating verbal information with visual information promotes integration of different modalities. Such integrative thinking, reinforced by C.R.E.A.T.E.'s repeated use of concept maps, both as a tool for review and a way to organize papers’ central themes, can facilitate learning (Novak, 1991; Van Meter and Garner, 2005; Schwartz and Heiser, 2006). Class discussion often focuses on a point of controversy, which can both increase student engagement (Bell and Linn, 2002) and stimulate students to “do the real intellectual ‘work’ of synthesizing ideas across subdomains” (Seethaler, 2005, p. 273). C.R.E.A.T.E.'s narrow focus on a few papers may encourage students to work toward deep rather than superficial understanding (Schwartz et al., 2009) as they engage in cognitively stimulating activities corresponding to upper levels of Bloom's taxonomy (e.g., analysis, synthesis, evaluation: levels 4–6; Bloom and Krathwohl, 1956). Extended discussions involving scientific argumentation are rare in lecture-dominated classrooms (Osborne, 2010), but can be of substantial benefit, especially when students feel free to develop their understanding through discussion and to speculate aloud as they do so (Sawyer, 2006). C.R.E.A.T.E. “grant panel” activities encourage student reflection on the research process beyond the details of individual journal articles.

    UREs might also be expected to have a strong effect on students’ epistemological beliefs. This has been seen in some cases (Rauckhorst et al., 2001; Lopatto, 2004b; both studies include both science and nonscience URE participants), but in other studies epistemological beliefs appeared not to shift significantly during the URE, either as reported by student participants or their faculty mentors (Hunter et al., 2007). A recent meta-analysis of independent research experiences in science (Sadler et al., 2010) suggests that supplementing research experiences with specific additional activities, such as keeping a journal of reflections on the research experience (Rauckhorst et al., 2001, college students; Schwartz et al., 2004, high school teachers) or interacting with peers also involved in research apprenticeships (Grindstaff and Richmond, 2008; high school students), can expand gains made in UREs and enhance understanding of the nature of science. These researchers further note that developing a deeper understanding of the nature of science will probably require instructional approaches that ensure undergraduates’ participation in developing hypotheses and analyzing data, both considered “epistemically demanding practices” (Sadler and McKinney, 2010, p. 48). Other investigators have suggested that epistemological beliefs are more likely to change if students are trained to think critically in a context that encourages metacognition and includes controversy (Valanides and Angeli, 2005). In studies of high school students (Bell et al., 2003) and undergraduates (Ryder et al., 1999), changes in scientific thinking were seen in students who participated in projects that demanded substantial epistemic engagement. Conversely, classrooms lacking in authentic scientific inquiry activities can reinforce naïve epistemological beliefs, for example, that scientific logic is simple and conclusions certain (Chinn and Malhotra, 2002). Students’ ability to carry out research projects may be constrained by such beliefs (Ryder and Leach, 1999).

    We consider it likely that the shifts we see in epistemological beliefs of C.R.E.A.T.E. students are attributable to students’ experiences in the C.R.E.A.T.E. course. We did not measure presemester/postsemester epistemological beliefs in an independent control group of students who did not take the course, as no such control group was available. We are not, however, aware of any studies that show that increased sophistication of epistemological beliefs results from mere maturation or passage of time during a single semester. The experiences of science students in UREs may provide some insight into the malleability of students’ epistemological beliefs. Science students’ UREs would be expected to support or enhance any shifts in their epistemological beliefs that occurred “maturationally” during an academic semester. Thus, the finding that epistemological beliefs tend to remain stable in science URE participants interviewed repeatedly over several years (Hunter et al., 2007), suggests that the epistemological beliefs of undergraduate science students do not shift rapidly. We feel it is likely that the postsemester versus presemester changes we document were brought about by students’ experiences in the semester-long C.R.E.A.T.E. course.

    Although the C.R.E.A.T.E. approach is not unique in focusing on primary literature, it is unusual in its combination of intensive analysis of a series of related publications with an email survey of their authors, and its concentration in the classroom on discussion and analysis aimed at simultaneously decoding the figures and tables, modeling the research process, and humanizing the scientists behind the papers. The C.R.E.A.T.E. teaching method encourages students to engage in conversations, debates, and creative thinking, which involve cognitive challenges that can help develop understanding of complex material (Driver et al., 2000; Marbach-Ad and Sokolove, 2000; Bell and Linn, 2002; Seethaler, 2005; Campion et al., 2009) and at the same time encourage creative approaches to such material (DeHaan, 2009). Overall, our findings indicate that the C.R.E.A.T.E. method increases students’ confidence in their ability to read and understand primary literature, improves their self-assessed understanding of the nature and processes of science, and encourages their development of more sophisticated epistemological beliefs. We suggest that complementing existing curricula with inexpensive C.R.E.A.T.E.-style courses could be an effective way to help students develop deeper insight into the nature and practices of science. Finally, students who recognize early in their college years that science is creative and open-ended might be more likely to take advantage of the UREs that can stimulate and reinforce interest in science research careers.

    ACKNOWLEDGMENTS

    Many thanks to CCNY students for participating in the C.R.E.A.T.E. course and associated assessments. The C.R.E.A.T.E. project is approved by CUNY IRB, protocol H-0633. We thank the NSF for support. S.G.H. also thanks the Helen Riaboff Whiteley Center at the Friday Harbor Laboratories, Friday Harbor, Washington, for academic support during a recent sabbatical, and Ross Nehm for discussion of survey design. This material is based upon work supported by the NSF CCLI/TUES program under Grants 0311117, 0618536, and 1021443. Any opinions, findings, and conclusions or recommendations expressed in this material are those of the authors and do not necessarily reflect the views of the NSF.

    REFERENCES

    • Alberts B (2005). A wakeup call for science faculty. Cell 123, 739-741. MedlineGoogle Scholar
    • Allen D, Tanner K (2005). Infusing active learning into the large-enrollment biology class: seven strategies, from the simple to complex. Cell Biol Educ 4, 262-268. LinkGoogle Scholar
    • American Association for Higher Education (2000). Targeting Curricular Change: Reform in Undergraduate Education in Science, Math, Engineering, and Technology, Washington, DC: American Association for Higher Education. Google Scholar
    • American Association for the Advancement of Science (AAAS) (1989). Science for All Americans: A Project 2061 Report on Literacy Goals in Science, Mathematics, and Technology, Washington, DC: AAAS, www.project2061.org/publications/bsl/online/index.php?chapter=1 (accessed 23 July 2011). Google Scholar
    • AAAS (1993). Benchmarks for Science Literacy, New York: Oxford University Press. Google Scholar
    • AAAS (2010). Vision and Change—A Call to Action. http://visionandchange.org/files/2010/03/VC_report.pdf (accessed 23 July 2011). Google Scholar
    • Armbruster P, Patel M, Johnson E, Weiss M (2009). Active learning and student-centered pedagogy improve student attitudes and performance in introductory biology. CBE Life Sci Educ 8, 203-213. LinkGoogle Scholar
    • Atwell R (2004). The long road ahead: barriers to minority participation persist. In: Reflections on 20 Years of Minorities in Higher Education and the ACE Annual Status Report, Washington, DC: American Council on Education, Center for Advancement of Racial and Ethnic Equity, www.acenet.edu/Content/NavigationMenu/ProgramsServices/CAREE/2004_reflections_msr.pdf (accessed 23 July 2011). Google Scholar
    • Aulls M, Shore B (2008). Inquiry in Education, vol. I: The Conceptual Foundations for Research as a Curricular Imperative, New York: Lawrence Erlbaum. Google Scholar
    • Baxter Magolda MB (1992). Knowing and Reasoning in College: Gender-Related Patterns in Students’ Intellectual Development, San Francisco, CA: Jossey-Bass. Google Scholar
    • Baxter Magolda MB (2004). Evolution of a constructivist conceptualization of epistemological reflection. Educ Psychol 39, 31-42. Google Scholar
    • Bell P, Blair L, Crawford B, Lederman N (2003). Just do it? Impact of a science apprenticeship program on high school students’ understandings of the nature of science and scientific inquiry. J Res Sci Teach 40, 487-509. Google Scholar
    • Bell P, Linn M (2002, Ed. B. HoferP. Pintrich, Beliefs about science: how does science instruction contribute? In: Personal Epistemology: The Psychology of Beliefs about Knowledge and Knowing, Mahwah, NJ: Lawrence Erlbaum, 321-346. Google Scholar
    • Bloom BS, Krathwohl DR (1956). Taxonomy of Educational Objectives: The Classification of Educational Goals, by a Committee of College and University Examiners, Handbook 1: Cognitive Domain, New York: Longman. Google Scholar
    • Butler D, Pollock C, Nomme K, Nakonechny J (2008, Ed. B. ShoreM. AullsM. Delcourt, Promoting authentic inquiry in the sciences: challenges faced in redefining university students’ scientific epistemology In: Inquiry in Education, vol. II., New York: Routledge, 301-324. Google Scholar
    • Call GB, et al. (2007). Genomewide clonal analysis of lethal mutations in the Drosophila melanogaster eye: comparison of the X chromosome and autosomes. Genetics 177, 689-97. MedlineGoogle Scholar
    • Campion N, Martins D, Wilhelm A (2009). Contradictions and predictions: two sources of uncertainty that raise the cognitive interest of readers. Discourse Process 46, 341-368. Google Scholar
    • Cech T, Kennedy D (2005). Doing more for Kate. Science 310, 1741. MedlineGoogle Scholar
    • Chaplin S (2009). Assessment of the impact of case studies on student learning gains in an introductory biology course. J Coll Sci Teach 39, 72-79. Google Scholar
    • Chickering AW, Gamson Z (1987). Seven principles for good practice. AAHE Bull 39, 3-7. Google Scholar
    • Chinn C, Malhotra B (2002). Epistemologically authentic inquiry in schools: a theoretical framework for evaluating inquiry tasks. Sci Educ 86, 175-218. Google Scholar
    • Clark IE, Romero-Calderón R, Olson JM, Jaworski L, Lopatto D, Banerjee U (2009). “Deconstructing” scientific research: a practical and scalable pedagogical tool to provide evidence-based science instruction. PLoS Biol 7, e1000264. MedlineGoogle Scholar
    • Cohen J (1988). Statistical power analysis for the behavioral sciences, 2nd ed Hillsdale, NJ: Lawrence Erlbaum. Google Scholar
    • DeHaan R (2009). Teaching creativity and inventive problem solving in science. CBE Life Sci Educ 8, 172-181. LinkGoogle Scholar
    • DePass A, Chubin D (2009). Understanding Interventions That Encourage Minorities to Pursue Research Careers: Summary of a Workshop, Washington, DC: National Academies Press. Google Scholar
    • Driver R, Newton P, Osborne J (2000). Establishing the norms of scientific argumentation in classrooms. Sci Educ 83, 287-312. Google Scholar
    • Dweck CS, Leggett EL (1988). A social-cognitive approach to motivation and personality. Psychol Rev 95, 256-273. Google Scholar
    • Freeman S, O’Connor E, Parks JW, Cunningham M, Hurley D, Haak D, Dirks C, Wenderoth MP (2007). Prescribed active learning increases performance in introductory biology. CBE Life Sci Educ 6, 132-139. LinkGoogle Scholar
    • Grindstaff K, Richmond G (2008). Learners’ perceptions of the role of peers in a research experience: implications for the apprenticeship process, scientific inquiry, and collaborative work. J Res Sci Teach 45, 251-271. Google Scholar
    • Hanauer D, Jacobs-Sera D, Pedulla M, Cresawn S, Hendrix R, Hatfull G (2006). Teaching scientific inquiry. Science 314, 1880-1881. MedlineGoogle Scholar
    • Handelsman J, et al. (2004). Scientific teaching. Science 304, 521-522. MedlineGoogle Scholar
    • Hartman H (2002, Ed. H. Hartman, Developing students’ metacognitive knowledge and skills In: Metacognition in Learning and Instruction: Theory, Research and Practice, Dordrecht, The Netherlands: Kluwer Academic, 33-68. Google Scholar
    • Herreid CF (1994a). Case studies in science—a novel method of science education. J Coll Sci Teach 23, 221-229. Google Scholar
    • Herreid CF (1994b). Journal articles as case studies: The New England Journal of Medicine on breast cancer. J Coll Sci Teach 23, 349-355. Google Scholar
    • Hofer BK (2000). Dimensionality and disciplinary differences in personal epistemology. Contemp Educ Psychol 25, 378-405. MedlineGoogle Scholar
    • Hofer BK (2004). Exploring the dimensions of personal epistemology in differing classroom contexts: student interpretations during the first year of college. Contemp Educ Psychol 29, 129-163. Google Scholar
    • Hogan K, Maglienti M (2001). Comparing the epistemological underpinnings of students’ and scientists’ reasoning about conclusions. J Res Sci Teach 38, 663-687. Google Scholar
    • Hoskins S (2008). Using a paradigm shift to teach neurobiology and the nature of science—a C.R.E.A.T.E.-based approach. J Undergrad Neurosci Educ 6, A40-A52. MedlineGoogle Scholar
    • Hoskins S (2010). “But if it's in the newspaper, doesn't that mean it's true?” Developing critical reading and analysis skills by evaluating newspaper science with C.R.E.A.T.E. Amer Biol Teach 72, 415-420. Google Scholar
    • Hoskins SG, Stevens LM (2009). Learning our L.I.M.I.T.S.: less is more in teaching science. Adv Physiol Educ 33, 17-20. MedlineGoogle Scholar
    • Hoskins SG, Stevens LM, Nehm R (2007). Selective use of primary literature transforms the classroom into a virtual laboratory. Genetics 176, 1381-1389. MedlineGoogle Scholar
    • Hunter A, Laursen S, Seymour E (2007). Becoming a scientist: the role of undergraduate research in students’ cognitive, personal, and professional development. Sci Educ 91, 36-74. Google Scholar
    • Janick-Buckner D (1997). Getting undergraduates to critically read and discuss primary literature. J Coll Sci Teach 27, 29-32. Google Scholar
    • Karakas M (2009). Cases of science professors’ use of nature of science. J Sci Educ Technol 18, 101-119. Google Scholar
    • Kardash C, Scholes R (1996). Effects of preexisting beliefs, epistemological beliefs, and need for cognition on interpretation of controversial issues. J Educ Psychol 88, 266-271. Google Scholar
    • Klionsky DJ (1998). Application of a cooperative learning approach to introductory biology. J Coll Sci Teach 27, 334-338. Google Scholar
    • Klymkowsky MW (2007). Teaching without a textbook: strategies to focus learning on fundamental concepts and scientific process. CBE Life Sci Educ 6, 190-193. LinkGoogle Scholar
    • Knight J, Wood W (2005). Teaching more by lecturing less. Cell Biol Educ 4, 298-310. LinkGoogle Scholar
    • Kozeracki CA, Carey MF, Colicelli J, Levis-Fitzgerald M, Grossel M (2006). An intensive primary-literature-based teaching program directly benefits undergraduate science majors and facilitates their transition to doctoral programs. Cell Biol Educ 5, 340-347. AbstractGoogle Scholar
    • Lawson A, Banks D, Logvin M (2007). Self-efficacy, reasoning ability, and achievement in college biology. J Res Sci Teach 44, 706-724. Google Scholar
    • Lederman N (1992). Students’ and teachers’ conceptions of the nature of science. J Res Sci Teach 29, 331-359. Google Scholar
    • Lederman N, Wade P, Bell R (1998). Assessing the nature of science: what is the nature of our assessments?. Sci & Educ 7, 595-615. Google Scholar
    • Lopatto D (2004a). Survey of Undergraduate Research Experiences (SURE): first findings. Cell Biol Educ 3, 270-277. LinkGoogle Scholar
    • Lopatto D (2004b). What undergraduate research can tell us about research on learning. http://web.grinnell.edu/science/ROLE/Presentation_2004_CUR_annual_meeting_WI.pdf (accessed 23 July 2011). Google Scholar
    • Lopatto D (2007). Undergraduate research experiences support science career decisions and active learning. CBE Life Sci Educ 6, 297-306. LinkGoogle Scholar
    • Lopatto D, et al. (2008). Undergraduate research: genomics education partnership. Science 322, 684-685. MedlineGoogle Scholar
    • Lopatto D (2009). Science in Solution: The Impact of Undergraduate Research on Student Learning. Council on Undergraduate Research and the Research Corporation for Science Advancement. www.rescorp.org/gdresources/downloads/Science_in_Solution_Lopatto.pdf (accessed 23 July 2011). Google Scholar
    • Lynd-Balta E (2006). Using literature and innovative assessments to ignite interest and cultivate critical skills in an undergraduate neuroscience course. CBE Life Sci Educ 5, 167-174. LinkGoogle Scholar
    • Malcom SM, Abdallah J, Chubin DE, Grogan K (2005). A System of Solutions: Every School, Every Student, Washington, DC: AAAS. Google Scholar
    • Marbach-Ad G, Sokolove P (2000). Can undergraduate biology students learn to ask higher level questions?. J Res Sci Teach 37, 854-870. Google Scholar
    • National Research Council (NRC) (2003). Bio2010: Transforming Undergraduate Education for Future Research Biologists, Washington, DC: National Academies Press. Google Scholar
    • NRC (2007). Rising Above the Gathering Storm: Energizing and Employing America for a Brighter Economic Future, Washington, DC: National Academies Press. Google Scholar
    • NRC (2009). A New Biology for the 21st Century, Washington, DC: National Academies Press. Google Scholar
    • National Science Foundation (NSF) (2002). Science and engineering degrees, by race/ethnicity of recipients: 1991–2002. NSF 02–329. www.nsf.gov/statistics/nsf02329/front.htm (accessed 23 July 2011). Google Scholar
    • NSF (2008). Broadening participation at the National Science Foundation: a framework for action. www.nsf.gov/od/broadeningparticipation/nsf_frameworkforaction_execsummary_0808.pdf (accessed 23 July 2011). Google Scholar
    • Nordell S (2009). Learning how to learn: a model for teaching students learning strategies. Bioscene: J Coll Biol Teach 35, 35-44. Google Scholar
    • Novak J (1991). Clarify with concept maps: a tool for students and teachers alike. Sci Teach 58, 45-49. Google Scholar
    • Osborne J (2003). Attitudes toward science: a review of the literature and its implications. Int J Sci Educ 25, 1049-1079. Google Scholar
    • Osborne J (2010). Arguing to learn in science: the role of collaborative, critical discourse. Science 328, 463-466. MedlineGoogle Scholar
    • Perry W (1970). Forms of Intellectual and Ethical Development in the College Years: A Scheme, New York: Holt, Rinehart and Winston. Google Scholar
    • Pieschel S, Stahl E, Bromme R (2008). Epistemological beliefs and self-regulated learning with hypertext. Metacog Learn 3, 17-37. Google Scholar
    • Pulmones R (2010). Linking students’ epistemological beliefs with their metacognition in a chemistry classroom. Asia-Pacific Educ Res 19, 143-159. Google Scholar
    • Quitadamo IJ, Faiola CL, Johnson J, Kurtz MJ (2008). Community-based inquiry improves critical thinking in general education biology. CBE Life Sci Educ 7, 327-337. LinkGoogle Scholar
    • Rauckhorst WH, Czaja JA, Baxter Magolda M (2001). Measuring the impact of the undergraduate research experience on student intellectual development. Paper presented at Project Kaleidoscope Summer Institute, Snowbird UT, July 15–28, 2001. Google Scholar
    • Russell S (2006). Evaluation of NSF support for undergraduate research opportunities. www.sri.com/policy/csted/reports/university/documents/URO%20FollowupSurveyRpt.pdf (accessed 27 September 2010). Google Scholar
    • Russell S, Hancock M, McCullough J (2007). Benefits of undergraduate research experiences. Science 316, 548-549. MedlineGoogle Scholar
    • Ryder J, Leach J (1999). University science students’ experiences of investigative project work and their images of science. Int J Sci Educ 21, 945-956. Google Scholar
    • Ryder J, Leach J, Driver R (1999). Undergraduate science students’ images of science. J Res Sci Teach 36, 201-219. Google Scholar
    • Sadler T, Burgin S, McKinney L, Ponjuan L (2010). Learning science through research apprenticeships: a critical review of the literature. J Res Sci Teach 47, 235-256. Google Scholar
    • Sadler TD, McKinney L (2010). Scientific research for undergraduate students: a review of the literature. J Coll Sci Teach 39, 43-49. Google Scholar
    • Sandoval WA (2003). The inquiry paradox: why doing science doesn't necessarily change ideas about science. In: Proceedings of the Sixth International Computer-Based Learning in Science Conference, July 5–10, 2003, Nicosia, Cyprus, 825–834. http://cblis.utc.sk/cblis-cd-old/2003/Nicosia.htm (accessed 23 July 2011). Google Scholar
    • Sawyer R (2006, Ed. R. K. Sawyer, Introduction: the new science of learning In: The Cambridge Handbook of the Learning Sciences, New York: Cambridge University Press, 1-16. Google Scholar
    • Schinske JN, Clayman K, Busch A, Tanner K (2008). Teaching the anatomy of a scientific journal article. Science Teach 75, 49-56. Google Scholar
    • Schommer M (1990). Effects of beliefs about the nature of knowledge on comprehension. J Educ Psychol 82, 498-504. Google Scholar
    • Schommer M (1993). Epistemological development and academic performance among secondary students. J Educ Psychol 85, 406-411. Google Scholar
    • Schommer M (1994, Ed. R. GarnerP. Alexander, An emerging conceptualization of epistemological beliefs and their role in learning In: Beliefs about Text and Instruction with Text, Hillsdale, NJ: Lawrence Erlbaum, 25-40. Google Scholar
    • Schommer M, Crouse A, Rhodes N (1992). Epistemological beliefs and mathematical text comprehension: believing it is simple does not make it so. J Educ Psychol 84, 435-443. Google Scholar
    • Schommer-Aikins M (2002, Ed. BK HoferPR Pintrich, An evolving theoretical framework for an epistemological belief system In: Personal Epistemology: The Psychology of Beliefs about Knowledge and Knowing, Mahwah, NJ: Lawrence Erlbaum, 103-118. Google Scholar
    • Schraw G, Crippen K, Hartley K (2006). Promoting self-regulation in science education: metacognition as part of a broader perspective on learning. Res Sci Educ 36, 111-139. Google Scholar
    • Schwartz D, Heiser J (2006, Ed. R Sawyer, Spatial representations and imagery in learning In: The Cambridge Handbook of the Learning Sciences, New York: Cambridge University Press, 283. Google Scholar
    • Schwartz M, Sadler P, Sonnert G, Tai R (2009). Depth vs breadth: how content coverage in high school science courses relates to later success in college coursework. Sci Educ 93, 798-826. Google Scholar
    • Schwartz RS, Lederman NG, Crawford BA (2004). Developing views of nature of science in an authentic context: an explicit approach to bridging the gap between nature of science and scientific inquiry. Sci Educ 88, 610-645. Google Scholar
    • Seethaler S (2005). Helping students make links through science controversy. Amer Biol Teach 67, 265-274. Google Scholar
    • Seymour E, Hewitt N (1997). Talking about Leaving: Why Undergraduates Leave the Sciences, Boulder, CO: Westview Press. Google Scholar
    • Shore B, Aulls M, Delcourt M (2008). Inquiry in Education, vol. II: Overcoming Barriers to Successful Implementation, New York: Lawrence Erlbaum. Google Scholar
    • Strauss BS (2005). PubMed, The New York Times and The Chicago Tribune as tools for teaching genetics. Genetics 171, 1449-1454. MedlineGoogle Scholar
    • Strømsø HI, Bråten I, Samuelstuen MS (2008). Dimensions of topic-specific epistemological beliefs as predictors of multiple text understanding. Learn Instruct 18, 513-527. Google Scholar
    • U.S. Department of Education (USDOE) (2000). Before It's Too Late: A Report to the Nation from The National Commission on Mathematics and Science Teaching for the 21st Century Washington, DC USDOE. Google Scholar
    • Valanides N, Angeli C (2005). Effects of instruction on changes in epistemological beliefs. Contemp Educ Psychol 30, 314-330. Google Scholar
    • Van Meter P, Garner J (2005). The promise and practice of learner-generated drawing: literature review and synthesis. Educ Psychol Rev 17, 285-325. Google Scholar
    • Yarden A (2009). Reading scientific texts: adapting primary literature for promoting scientific literacy. Res Sci Educ 39, 307-311. Google Scholar
    • Zeineddin A, Abd-El-Khalick F (2010). Scientific reasoning and epistemological commitments: coordination of theory and evidence among college science students. J Res Sci Teach 47, 1064-1093. Google Scholar
    • Zimmerman B (1990). Self-regulated learning and academic achievement—an overview. Educ Psychol 24, 3-17. Google Scholar
    • Zull J (2002). The Art of Changing the Brain, Sterling, VA: Stylus. Google Scholar