The CREATE Strategy for Intensive Analysis of Primary Literature Can Be Used Effectively by Newly Trained Faculty to Produce Multiple Gains in Diverse Students

INTRODUCTION

The explosion of information in the biological sciences since the mid-20th century necessitates new approaches in the classroom (National Research Council, 2003, 2009; American Association for the Advancement of Science [AAAS], 2011). Recent studies suggest that undergraduates taught in traditional ways fail to achieve mastery of key concepts essential to working scientists (Shi et al., 2011; Trujillo et al., 2012). Textbooks may underemphasize the scientific process in figures (Duncan et al., 2011), thereby contributing little to students’ understanding of how data should be represented or interpreted (Rybarczyk, 2011) and making it difficult for students to recognize that the material in their textbooks was largely derived from experimental or observational studies. Traditional teaching methods may not promote development of critical-reading, analytical, or thinking skills routinely employed by working scientists, especially when such teaching focuses on more basic skills (Momsen et al., 2010). Traditional multiple-choice testing may even impair students’ cognitive development (Stanger-Hall, 2012).

Active-learning approaches that increase student engagement enhance learning in some situations (Knight and Wood, 2005; Freeman et al., 2007) but not in others (Andrews et al., 2011), with instructor training likely to be an important component of successful outcomes. Thus, although some new styles of teaching may be of significant value to students, it may be challenging for inexperienced faculty members to carry them out effectively. Even the highly motivated faculty members who attend professional development workshops can have difficulty applying the lessons learned (Silverthorn et al., 2006; Ebert-May et al., 2011; Henderson et al., 2011).

These issues raise questions about the best ways to achieve pedagogical reform. In particular, are workshops an effective way to train faculty in the use of nontraditional teaching approaches? In addition, if a technique is effective when used by its developers, often science education specialists, will it also be effective when practiced by workshop-trained faculty members who do not have science education expertise? Finally, if a new teaching strategy produces cognitive and attitudinal gains in a particular cohort of students, will it also be effective when used by different faculty members with substantially different student cohorts? We have addressed these questions in relation to the CREATE (Consider Read, Elucidate the hypotheses, Analyze and interpret the data, and Think of the next Experiment) strategy, a teaching/learning approach that uses intensive analysis of primary literature to demystify and humanize science (Hoskins et al., 2007).

CREATE was developed to address the shortcomings of traditional science teaching described above (Hoskins et al., 2007) and utilizes primary literature, rather than textbooks, as its focus. CREATE guides students in a deep analysis of scientific papers that challenges them to interpret the data as if it were their own; students must understand both the design and the broader context of a given study. These students are likely to be motivated to think more deeply than when reading traditional textbooks, which typically summarize results or present simplified outlines of experiments in the absence of data (Duncan et al., 2011). In addition, the recent Discipline-based Education Research (DBER) report states that DBER “clearly shows that research-based instructional strategies are more effective than traditional lecture in improving conceptual knowledge and attitudes about learning” (Singer et al., 2012, p. 3).

CREATE was originally designed for upper-level students (Hoskins et al., 2007) and has since been adapted for first-year students as well (Gottesman and Hoskins, 2013). In the original version of CREATE, students in an upper-level elective that was focused on “analysis of primary literature” read a series of papers that had been published in sequence from a single laboratory. The students learned to use a variety of new and adapted pedagogical tools that included concept mapping, cartooning (sketching “what went on in lab” to generate the data in each figure), annotation of figures (rewriting caption and narrative information directly onto figure panels), and “translation” (paraphrasing complex sentences from the narrative). At the conclusion of data analysis for each paper, and before the next paper in the series was revealed, students designed their own follow-up experiments and vetted these in grant panel exercises designed to mimic activities of bona fide panels. Late in the semester, students generated a set of questions for the authors of the papers, addressing both personal and professional motivations and experiences. These were compiled into a single survey that was emailed to each author. The diversity of candid responses, from principal investigators (PIs), postdoctoral fellows, graduate students, and other collaborators, illuminated multiple aspects of “the research life.” These unique insights into the people behind the papers complemented students’ deep understanding of the papers and of the study design, affecting students’ views of “who” can become a scientist (see student postcourse interview data in Tables 1 and S1 of Hoskins et al., 2007).

In previous studies at City College of New York (CCNY), we found that upper-level students in CREATE courses made gains in critical thinking, content integration, and self-assessed learning (Hoskins et al., 2007), and also experienced positive shifts in science attitudes, self-rated abilities, and epistemological beliefs about science (Hoskins et al., 2011). In a pilot first-year Introduction to Scientific Thinking CREATE cornerstone elective, CCNY freshmen also made significant gains in critical thinking and experimental design ability, exhibited positive shifts in science attitudes and self-rated abilities, and showed maturation of some epistemological beliefs about science (Gottesman and Hoskins, 2013). Thus, for both upper-level and first-year students at CCNY, CREATE courses promote cognitive as well as affective gains.

To investigate the applicability of CREATE approaches at other institutions and in other topic areas, we trained faculty members from multiple campuses in the CREATE approach in a series of intensive workshops. We then followed a subset of workshop participants as they applied CREATE methods at their home campuses (Figure 1). With the aid of an outside evaluator (OE), we tested three hypotheses: 1) that our workshops would be an effective way to convey CREATE pedagogies and change participants’ views about science education and their classroom approaches; 2) that workshop-trained faculty from diverse colleges and universities would teach CREATE effectively in their first attempt; and 3) that students on diverse campuses, in courses taught by CREATE workshop–trained faculty, would make cognitive and/or affective gains paralleling those seen previously at a minority-serving institution (MSI).

Our data indicate that the workshop series changed aspects of participants’ thinking about science teaching, that the subset of seven workshop-trained faculty participants who subsequently taught courses as part of our study enthusiastically and effectively implemented CREATE, and that students at every implementing campus demonstrated significant shifts in cognitive and affective categories similar to those examined earlier at CCNY. Gains varied on the different campuses, but it was notable that even short-term, partial-semester CREATE implementations produced some important changes. CREATE faculty development workshops are thus an effective way to convey CREATE teaching strategies, as faculty members trained in such workshops can successfully teach CREATE courses the first time they attempt to do so. Importantly, CREATE students from a wide range of campuses (public, private, liberal arts, research intensive) made gains in critical thinking and concurrently underwent positive shifts in self-assessed attitudes, abilities, and epistemological beliefs related to science. We conclude that our workshop model is an effective way to train faculty to teach CREATE and that the CREATE method produces cognitive and affective gains in diverse student cohorts.

METHODS

Structure of Workshops

We recruited 16 faculty workshop participants through presentations at both national and regional meetings of professional societies (including the American Society for Cell Biology, National Association of Biology Teachers, Society for Neuroscience, Society for Developmental Biology, Northeastern Nerve Net, Mid-Atlantic Regional Society for Developmental Biology, and SENCER [www.sencer.net]). Faculty participants met at the City University of New York (CUNY) Graduate Center in midtown Manhattan once per month from August to December 2007 for workshops aimed at presenting both theoretical and practical aspects of the CREATE strategy. Each workshop ran from 10 am to 4 pm and was focused on particular tools of the CREATE pedagogical tool kit (Hoskins and Stevens, 2009). As part of the training, workshop participants played the parts of students as a workshop leader (S.G.H.) taught using CREATE strategies. Some participants “student taught” by using CREATE methods, with their fellow participants acting as students. Faculty participants learned how to select and prepare appropriate primary scientific literature and related materials, to teach using CREATE approaches, and to assess students in CREATE courses. Some of the workshops also featured visits from an experienced CREATE teacher who learned the strategy from our publications and discussions with S.G.H. and from a panel of seven students who were alumni of the 2006 CCNY upper-level CREATE class. The latter panelists answered participants’ questions about the experience of being a student in a CREATE class. Between workshop sessions, participants completed assignments that are typically given to CREATE students, using CREATE tools such as concept mapping, cartooning, figure annotation, and designing future experiments. Other assignments focused on selecting appropriate primary literature modules and designing classroom activities suitable for students working in groups.

Recruitment of CREATE Implementers

At the final workshop in December 2007, seven participants committed to teaching CREATE courses in the 2008 Spring or Fall semester (Table 1). Faculty participants who implemented CREATE did so by using CREATE strategies in courses they were already scheduled to teach on their home campuses. The PIs worked with the implementers to plan their integration of CREATE into a single existing course and to obtain institutional review board (IRB) approval for the education research project. Seven CREATE implementers (CIs) carried out implementations on five separate campuses. Two campuses had two CIs each, in different courses. Of the seven courses used for the implementations, four were in biology, two were in psychology, and one was in biochemistry. Each CI chose papers appropriate to his or her own student cohort and course (Table 1); thus the implementations were individual case studies rather than exact duplications of the CREATE courses and modules of papers taught at CCNY.

Design of Implementations

Students self-selected into the courses on each campus; no effort was made to recruit students for a research study. Per IRB requirements, CIs invited their student cohorts to participate in a research study involving “CREATE, a new way to teach biology (or chemistry, or psychology)”; thus all students were aware that research on a novel teaching/learning strategy was taking place in their classrooms. Students were also informed that two course sessions would be observed and that on her second visit the OE would invite those who wished to participate to take an anonymous survey regarding their reaction to their course. All cohorts learned the tools of CREATE, but not all faculty participants took time in class to describe the CREATE project specifically. CIs administered anonymous pre- and postcourse surveys to those who chose to take part and forwarded these to the PIs for scoring. Each CI also coordinated with the OE to arrange for her to observe their CREATE class twice during the semester at an appropriate interval (observations 4 wk apart for full-semester implementations; 2–4 wk apart for shorter implementations). The PIs communicated informally (phone or email) with CIs during the semester and held a midsemester meeting with CIs at the CUNY Graduate Center to share experiences and troubleshoot any issues that arose. At the conclusion of their work, implementers received a stipend in recognition of their efforts. One hundred and eight students participated in the study in the seven implementations. Totals for particular surveys may vary slightly due to individual student absences on days that different surveys were administered. The majors of the participating students were psychology (40%), biology (32%), premed (12%), biochemistry (11%), and undeclared (5%). Of the participants 63% were female, and 30% described themselves as “ethnic minority” (Hurley, 2009).

Evaluation of Workshops (Figure 1, Hypothesis 1)

PIs’ Evaluation.

Workshop participants’ reactions to the workshops were assessed by the responses of faculty participants to 1) Likert-style participant surveys prepared by the PIs and administered before (August) and after (December) the series of monthly workshops (Table 2), and 2) surveys regarding individual workshops. Workshop participants completed the participant surveys using code numbers that allowed the PIs to pair the pre- and postworkshop surveys while preserving anonymity. The participant surveys asked workshop participants to respond to 18 statements about pedagogical practice and teaching/learning issues. Phrasing of statements varied so that the “optimal” postcourse response was not always the one indicating stronger agreement. For example, following the workshops we would expect less agreement with the statement “Science classes are different from courses in other subjects in that students must learn the content before they can think about it analytically.” The seven statements that were presented identically on the pre- and postsurveys were scored on a 5-point scale (I strongly disagree; I disagree; I’m not sure; I agree; I strongly agree), and pre/post workshop differences were calculated using a paired t test (Excel). For 11 statements, participants were asked (preworkshop) to respond regarding the extent to which they already engaged in particular practices characteristic of the CREATE classroom (e.g., having students work in small groups, spending class time closely analyzing data, teaching concept mapping, making efforts to humanize science), whereas the postworkshop survey asked participants to indicate the likelihood that they would implement such practices in the future. For these 11 statements, we calculated the overall percentage of respondents who agreed or strongly agreed.

For evaluation of each individual workshop, we adapted the Student Assessment of Their Learning Gains (SALG) anonymous evaluation tool provided free to the college-level teaching community (www.salgsite.org). At the conclusion of each individual workshop session, the participants completed a SALG survey that posed questions directed at that month's workshop activities.

OE's Evaluation.

Independent evaluation of the workshop series was provided by OE Marlene Hurley, Ph.D. (in science education). Dr. Hurley attended each workshop, tracked numerous aspects of interactions of workshop participants and workshop faculty using observation protocols designed for this purpose, observed individual workshop participants as they “student taught” the rest of the group, and observed question/answer sessions between the workshop participants and visiting CREATE students or faculty. During each workshop session, the OE tracked the extent to which individual aspects of the CREATE process were introduced, discussed, modeled, and/or practiced. After each workshop, the OE provided immediate formative feedback to the PIs in a half-hour discussion of the workshop day. At the conclusion of the workshop series, the OE provided written summative feedback. This evaluation and the OE's observations and evaluations of the individual CREATE implementations (outlined below) were compiled by the OE in a 127-page final report and provided to the PIs (Hurley, 2009)

OE’s Evaluation of Teaching by CIs (Figure 1, Hypothesis 2)

For evaluation of CREATE faculty implementers on individual campuses, the OE modified two existing observation protocols to adapt them specifically for the CREATE classroom. One, the Weiss Observation Protocol for Science Programs, was originally devised for evaluation of National Science Foundation (NSF)-funded projects (Hurley, 2009; see also Weiss et al., 1998). The OE adapted this instrument so that it could be used during each observation session to gather evidence on four focus areas and key indicators related to each area (Supplemental Material). The focus areas were: design of session (12 key indicators), instruction of session (18 key indicators), science content (seven key indicators), and the nature of science (three key indicators). Each focus area was tracked during the observation with reference to these measures. For example, design of session included “The design of the session appropriately balanced attention to multiple goals within the CREATE structure” and “Adequate time and structure were provided for reflection.” During the teaching session, performance in each area was tracked on a numeric scale, where 1 = “not at all”; 2, 3, and 4 were unlabeled; and 5 = “to a great extent.” The scale also included 6 (don't know) and 7 (not applicable). Each focus area was summarized in a synthesis rating based on a scale of 1–5, where 1 = “Design of the session was not at all reflective of reflective of best practices for CREATE”; 2, 3, 4, were unlabeled; and 5 = “Design of the session was extremely reflective of best practices for CREATE.” This synthesis rating in turn was supported by additional written comments. For each of the four focus areas, the observer also rated faculty members’ teaching on a scale of 1 to 5 (1 = not successful, 2 = slightly successful, 3 = moderately successful, 4 = highly successful, 5 = extremely successful).

The OE also used a Flanders Observation Protocol (Flanders, 1963) that was adapted to broaden its applicability in college teaching situations with diverse formats. This was used to collect quantitative data on interactions between CREATE implementers and their students; these data were used by the OE to determine the extent to which implementers ran student-centered CREATE classrooms.

Owing to scheduling constraints, not every observation was made by the primary OE, Dr. Hurley. Some observations were made by her colleague, Dr. Fernando Padró. To ensure consistency, the two evaluators pilot-tested the observation protocol instruments during initial observations in Spring 2008 to establish interrater reliability (Hurley, 2009). To supplement the data gleaned from the instruments described above, the OEs took descriptive notes during observations. Late in the semester, the primary OE interviewed each faculty member to elicit overall reactions to teaching with the CREATE strategy for the first time as well as his or her sense of potential student gains.

Evaluation of Potential Cognitive and Affective Gains by Students in CREATE Implementations (Figure 1, Hypothesis 3)

The PIs coordinated with the CIs to administer: 1) a critical-thinking test (CTT) adapted from the Field-tested Learning Assessment Guide (www.flaguide.org), which was administered pre- and post-CREATE implementation; 2) the Student Attitude Survey (SAS), a Likert-style survey examining students’ self-assessed abilities, attitudes, and beliefs, also administered pre- and post implementation, with scores analyzed by Wilcoxon signed-rank test (http://vassarstats.net/wilcoxon.html); and 3) a postcourse online SALG survey. All surveys were anonymous; on the CTT and SAS surveys, students used secret code numbers that allowed their pre- and postcourse responses to be paired for statistical analysis. No coding was used with the SALG, which was administered postcourse only.

CTT.

As one goal of our study was to determine whether the critical-thinking gains seen at CCNY would be replicated on campuses with different student cohorts, the CIs administered a CTT that included four questions that also appeared on the CTT used in our original study (see Figure 2, Q 1–4, of Hoskins et al., 2007). Questions A, C, and D were from the “General Science/Conceptual Diagnostic Test/Fault Finding and Fixing/Interpreting and Misinterpreting Data” section of the Field-tested Learning Assessment Guide website and were presented in an identical form on the pre- and postimplementation CTT. Question B was designed by the PIs to focus on biological data analysis and, as in our previous study, was presented in isomorphic form (identical form with different contexts and data pre- and postimplementation).

Each question challenges students to interpret charts or graphs, find trends in data, and determine whether the conclusions stated in the narrative accompanying each question follow logically from the data shown. The question asks students whether they do or do not agree with the conclusion stated and prompts them to explain their thinking with reference to the relevant chart, graph, or table, for example, “Do you agree or disagree? Explain why, using data to support your answer.” Students wrote short responses to each question. We used a rubric developed in our previous study (Hoskins et al., 2007) to score the following measures: Whether a student agreed, disagreed, or took no stance on the conclusion stated in the question; the number of logical justifications presented as part of the student's explanation; and the number of illogical justifications presented. Total numbers of logical and illogical justifications in each response were compiled separately. Responses that appropriately disagreed with an illogically stated conclusion received an additional “logical” point, while those that inappropriately agreed with an illogical conclusion received an additional point in the “illogical” category. Students who took no stance on the “agree/disagree” prompt had no points added to their totals. Initially, 20 CTT responses were scored independently by a research assistant and one of the authors (S.G.H.). Scores were compared and discrepancies reconciled. An additional 15 tests were then scored by each scorer, with 87% of scores identical. The research assistant then scored all of the surveys. Not all students were present for both the pre- and the postimplementation test, and we scored responses only for which we had both assessments. The CTT scores were analyzed by paired t test (Excel).

SAS.

In the previous CCNY studies, we documented a positive effect of the CREATE approach on students’ confidence in their ability to think like scientists and to read/analyze primary literature, as well as on their attitudes about their own abilities, the nature of science, and the nature of knowledge, or epistemological beliefs (Hoskins et al., 2011). To determine whether similar changes occurred in the student cohorts in the implementations, CIs administered a SAS to their students pre- and postimplementation. The 26-statement SAS is derived from the 37-statement Survey of Student Attitudes, Abilities, and Beliefs (SAAB), the development of which has been described (Hoskins et al., 2011; Gottesman and Hoskins, 2013). Factor analysis of SAAB statements was used to derive broad categories addressed by that survey. Statements in the SAS are identical to those in the SAAB, but the SAS contains fewer statements in some categories. Students responded to SAS statements on a 5-part Likert scale to which we assigned the following point values: 1 = “strongly disagree,” 2 = “disagree,” 3 = “I’m not sure,” 4 = “agree,” 5 = “strongly agree.” We calculated post- versus preimplementation differences using the Wilcoxon signed-rank test, a nonparametric statistical test considered appropriate for Likert-style data (Lovelace and Brickman, 2013).

SALG.

The SALG surveys were constructed to assess reaction to multiple aspects of the CREATE implementations, including students’ sense of whether particular tools enhanced their learning, whether the course structure enhanced their learning, and whether their views of scientists and the research process changed during the CREATE semester. This survey was given postcourse only. Spring 2008 implementations (CI1–CI5) used the original version of the site, which was redesigned and altered slightly in Summer 2008 by the survey developers. As the original SALG version was no longer available, the Fall 2008 implementations, CI6 and CI7, used the new version, potentially affecting survey validity. We designed surveys to be as parallel as possible and aligned questions for scoring.

Most questions were phrased “How much did [a particular tool or course aspect] help your learning?” and answered with a number ranging from 1 to 5 (no help; little help; some help; much help; great help). We analyzed the data by pooling responses to the set of statements within individual categories (e.g., grouping the multiple individual statements that addressed categories including “the class overall”; “class activities”; or “your understanding of course content”). Means and SDs were calculated in Excel. The SALG also allows construction of open-ended questions to which students can respond in writing. Here, we asked questions about students’ confidence in their scientific thinking ability and their postcourse views of scientists.

RESULTS

I. Test of hypothesis 1: Our workshops would be an effective way to convey CREATE pedagogies and change participants’ views about science education and their classroom approaches.

PIs’ Evaluation—Participant Survey

Table 2 summarizes data from the 18-statement participant survey addressing workshop participants’ teaching approaches and attitudes about science pedagogy pre- and postworkshop. With regard to statements examining participants’ preworkshop practices compared with what they planned to do postworkshop, we saw changes in 10 out of 11 survey statements. The largest (more than 20% different postworkshop) changes were in response to statements regarding the use of small groups in lecture classes, on assessing students’ precourse knowledge, on referring to the published literature on science education, and on using concept mapping. The response to the latter increased to almost 90% from less than 50% preworkshop. Statements showing smaller changes tended to have a strong response even preworkshop, indicating that many of the highly motivated workshop participants were already engaging in some activities that encourage student engagement, although it was clear from discussions in the first workshop that all faculty participants used lecture as their most typical classroom approach. We used the “pre” survey to gain insight into the extent to which participants were also using some of the approaches typical of the CREATE classroom.

With regard to the statements presented in identical form pre- and postworkshop, we saw significant change with moderate effect size (ES) on two statements: “The time required to learn new styles of teaching is prohibitive” and “I find it difficult to understand the literature on science education.” In each case, faculty participants agreed significantly less with the statement postworkshop. An additional statement, “Students need to have completed introductory course work in science before they can read and understand primary scientific literature” showed significant change (less agreement postworkshop) with small ES.

PIs’ Evaluation—SALG Survey

In the SALG surveys given at the conclusion of each workshop session, participants ranked multiple aspects of each workshop on a scale of 1–5, with 5 = “great help”; 4 = “much help”; 3 = “moderate help”; 2 = “some help”; and 1 = “little help.” Overall, workshop rankings averaged 4, “much help,” for each of the five workshops (data not shown). In written comments, participants expressed particular enthusiasm for the November workshop (#4), which included a discussion with the panel of CCNY CREATE students.

OE's Evaluation—Observation of Workshops

The OE's assessment of the effectiveness of workshop sessions focused on the question “Was CREATE successfully introduced, modeled, and disseminated in the faculty development workshops in order to adequately prepare college faculty for methods of science teaching that are beyond the traditional lecture and lab?” (Hurley, 2009, p. 3). These data complemented those collected by the PIs in the surveys described above. As part of her evaluation, the OE tracked the extent to which individual aspects of CREATE were introduced, taught, and reinforced in each workshop, the degree to which the PIs modeled student-centered teaching, how much “student” (faculty participant) participation ensued, the nature of classroom discussions, and the degree to which faculty participants seemed to be learning the CREATE strategy. Based on naturalistic notes taken during the workshops, including specific tracking of the extent to which key aspects of CREATE (constructivism, active engagement, inquiry, concept mapping, cartooning and annotating, “design the next experiment,” grant panels, and email contact with science researchers) were included in each session (data not shown), the OE concluded: “In response to Evaluation Question 1 [above], and in light of all the evidence presented and considered, CREATE was successfully introduced, modeled, and disseminated in the faculty development workshops in order to adequately prepare college faculty for methods of science teaching that are beyond the traditional lecture and lab” (Hurley, 2009, p. 81).

II. Test of hypothesis 2: Workshop-trained faculty from diverse colleges and universities would teach CREATE effectively in their first attempt.

OE's Evaluation of Implementer Teaching

The conclusions of the primary OE regarding CI effectiveness relied on data from multiple sources, including 1) student activities during the two observations of each implementing classroom, 2) faculty classroom performance as rated on the observation protocols, 3) interviews with the CIs, 4) naturalistic notes taken during the observation periods, and 5) her overview of student survey data, which included a statement asking students to rate their teacher's performance on a 1–5, low→great scale (data not shown). Two CIs were rated “extremely successful,” one “highly to extremely successful,” two “highly successful,” one “moderately to highly successful,” and one “moderately successful” by the OE. There was no obvious relationship between the success of each implementation and the CI's years of teaching experience, whether the campus was public or private, or whether the overall implementation was full semester or partial semester.

The response of the CIs to using the CREATE approach was uniformly positive. In their 2008 interviews with the OE (Table 3), all CIs stated that they would use the teaching strategy again and would recommend it to colleagues. Further, multiple CIs remarked that they would recommend CREATE for courses beyond the ones they had taught, including graduate courses, courses for non–science majors, and courses in psychology, history, and social science. Implementers noted their enjoyment of the group activities, their sense that students were building metacognitive awareness, their students’ growing ability to discuss and argue productively with peers, the students’ increased confidence in their reading/analysis ability, and overall high student engagement in the classroom.

III. Test of hypothesis 3: Students on diverse campuses, in courses taught by CREATE workshop–trained faculty, would make cognitive and/or affective gains paralleling those seen previously at an MSI.

Assessments of Implementers’ Students

CTT.

For many of the individual implementations, when pre- and postimplementation scores were compared, significant increases and decreases were seen in the number of logical and illogical justifications, respectively (Table 4, A–D). To diminish the variability arising from differences between the effectiveness of individual CIs, we pooled the data for all the implementations and found that, for all four questions, the students exhibited statistically significant increases in their logical statements. These are consistent with an increase in the ability of the students to think critically and analytically about data, that is, in their ability to “think like a scientist.”

We also examined the CTT data for differences between full-semester and partial-semester implementations and between public and private campuses by pooling the relevant categories. In partial-semester implementations, faculty members typically presented material in a traditional format in the course's early weeks and then shifted to CREATE-style teaching later in the semester. For the pooled full-semester implementations (n = 45 students), all of the questions showed significant shifts in the number of logical justifications, with large to moderate effect size (ES). For the pooled partial-semester cohort (n = 63), significant change was seen for three questions, with a large, moderate, or small ES. Thus, while significant change is seen in all groups, more and larger gains are made by the full-semester CREATE implementations. A similar pattern is seen for illogical statements, for which the full-semester implementations changed significantly (fewer illogical justifications postcourse than precourse) on three questions, while the partial-semester implementations changed on only one. Taken together, these findings suggest that full-semester implementations produce more changes with larger overall ESs than do partial-semester implementations.

Comparison of private (n = 42) and public (n = 66) student cohorts shows significant gains in logical statements made by both groups on all four questions (Table 4, A–D). ESs were large or moderate for private campuses and large, moderate, or small for public universities. Illogical statements decreased significantly on all questions in the private school cohort, and on a single question in the public school cohort. Overall, private school cohorts changed significantly in more categories and with larger ESs than did public school cohorts, but as the numbers overall are small and may be compounded by effects related to implementation duration, this conclusion must be considered tentative.

SAS.

With respect to SAS factors (1 = decoding primary literature, 2 = interpreting data, 3 = active reading, 4 = visualization, and 5 = thinking like a scientist), all cohorts changed significantly on factors 1, 3, 4, and 5, and all but the partial-semester cohorts also made significant gains on factor 2 (Table 5A). Comparing full- and partial-semester implementations shows larger ESs in full-semester cohorts, arguing for more significant gains (Coe, 2002; Maher et al., 2013) in the longer-duration implementations. Pooled private or pooled public implementations showed gains on each of factors 1–5 with more comparable ESs. No significant change was seen in any subgroup in a sixth category, “research in context,” which deals with students’ sense of the use of model systems and controls in experiments, suggesting that students were already familiar with these concepts (data not shown).

When we examined individual implementations with respect to these factors (Supplemental Material, SD 3a), we found that students in the majority of implementations changed significantly on statements assessing self-rated ability to decode scientific literature, science reading ability, and visualization, with large ESs. Multiple campuses also showed gains on self-rated data interpretation ability and ability to think like a scientist. The changes seen are in the same range as those seen in a comparison single class of CCNY students taught in a full-semester CREATE course by an experienced CREATE teacher (S.G.H.; see Supplemental Material, “CCNY 2009” in SD 3a).

The factors addressing epistemological issues showed fewer overall changes and more differential outcomes between cohorts (Table 5B). Four of the seven factors defined in the original SAAB survey as addressing epistemological issues, (referred to here as 1 = creativity; 2 = sense of scientists; 3 = sense of scientists’ motivations; 4 = research as collaborative), showed significant change in the implementations The pooled group of all institutions changed significantly with small ES on factors 1 and 2. Full-semester implementations changed on factors 1, 2, and 3 (large to moderate ES), while partial-semester implementations did not change significantly on any epistemological factor.

Outcomes in public versus private implementations were comparable, with each cohort changing with moderate to small ES on two of the four categories (Table 5B). Looked at individually, four of the seven cohorts changed significantly on at least one epistemological factor (Supplemental Material, SD 3b). Figure 2 plots precourse versus postcourse responses for pooled cohorts on factor 1, decoding primary literature. This set of scatter plots shows a pattern seen throughout the SAS responses, with full-semester and private school cohorts less dispersed overall than partial-semester and public school cohorts.

The changes seen at individual campuses on the SAS are in the same range as those seen in a comparison single class of CCNY students taught by an experienced CREATE teacher (S.G.H.) in a full-semester CREATE course (see Supplemental Material, “CCNY 2009” in SD 3). CCNY students made gains in three epistemological categories, with moderate to large ES. These findings are consistent with our hypothesis 3, that student groups at various types of institutions will respond to the CREATE method by undergoing positive shifts in self-rated science abilities and attitudes as well as epistemological beliefs.

SALG.

The SALG survey contained 45 prompts distributed across 10 categories. Table 6 summarizes data for each category, with scores reflecting combined averages for the substatements in that category. Notably, students in all implementations reported self-assessed learning gains in all categories. Responses to two open-ended SALG survey questions, on students’ views of scientists and of their ability to “think like a scientist,” are also discussed below (Table 7).

Of the SALG summary categories (Table 6), “the course overall,” “class activities,” and “support for you as an individual learner” were rated by two implementations each as the category in which students felt they made the largest learning gains. In one implementation, the score was highest for “your understanding of class content.” Within individual categories, we explored the substatements that received high scores from more than one implementation. For the majority of implementations, “the way the material was approached” (the course overall); “discussions in class” (class activities) and “opportunities for in class review” (assignments, graded activities, and tests) received the highest ratings.

Regarding the category “aspects of the class that might have affected learning,” students in multiple implementations singled out “the relationship between classwork and assigned reading,” “the modified scientific papers,” or “templates for analyzing figures” as the most helpful. With regard to the categories “support for you as an individual learner” and “your understanding of class content,” the majority of implementations rated “the quality of contact with the instructor” and “the relationships between concepts” as the most influential.

With regard to “increases in your skills” (15 substatements), we tracked the top three self-reported gains for each implementation. “Critically reviewing articles,” “understanding the ‘methods’ sections in papers,” “relating methods used to data obtained,” “reading about science,” “ability to explain concepts,” “understanding experimental design,” and “critical analysis” were among the top choices in multiple implementations. Regarding “impact on your attitudes,” most implementations scored “appreciating this field” as the area in which they had made the largest overall gains, with others noting their sense that they had gained transferable skills. Regarding “integration of your learning,” the majority chose “understanding the relevance of this field to real-world issues,” while responding students of the remaining implementations considered their largest gains to be in “understanding how ideas in this class relate to those in other science classes.”

On SALG open-ended questions, students were asked about their views of scientists and their confidence that they could think like scientists. Fifty-two percent reported that their views of scientists had changed postcourse. Open-ended comments associated with this SALG question indicated that views had changed favorably. In addition, 74% of student respondents described themselves as feeling “more confident” of their scientific thinking ability. Table 7 presents representative open-ended comments from each implementation that are related to these two questions. Undergraduates on multiple campuses noted a new awareness of scientific creativity as well as confidence that analytical skills they developed during the semester would be applicable in future academic situations.

DISCUSSION

We will discuss the results of our study with respect to the three hypotheses proposed at the outset of our study (Figure 1). We note, however, that our findings related to hypothesis 2 reflect on hypothesis 1, in that effective teaching by the CIs implies the workshops were successful. Similarly, gains made by the CIs’ students (hypothesis 3) reflect on the effectiveness of the CIs’ teaching (hypothesis 2).

Hypothesis 1: Our workshops would be an effective way to convey CREATE pedagogies and change participants’ views about science education and their classroom approaches.

Workshops have long been used to disseminate pedagogical improvements (Emerson and Mosteller, 2000; American Association of Physics Teachers, 2013; Gregg et al., 2013), but relatively few studies have evaluated workshop efficacy with respect to faculty participants, beyond participant self-report. The increasingly common “develop and disseminate” model, in which science educators develop novel pedagogical approaches and then train other faculty members to implement them, a strategy also used in this study, has to date not been supported strongly by data (Henderson and Dancy, 2007, 2011; Henderson et al., 2010, 2011). Even if a workshop successfully changes faculty attitudes and intentions, the process of changing how teachers teach and students learn is surprisingly difficult. For example, workshop-trained faculty members may self-report that they are implementing new approaches, yet not actually use the new methods as their developers intended (Ebert-May et al., 2011). In some cases, it may take several years postworkshop before faculty self-report applying the workshop training effectively (Pfund et al., 2009). Finally, it is unusual for projects involving faculty development workshops to combine faculty participant self-assessment with both independent evaluation of “workshop alumni performance,” such as repeated outside evaluation of their postworkshop teaching, and with separate anonymous assessment of participants’ students using cognitive as well as affective measures. As a result, there are few data on cognitive and affective changes in the student cohorts taught by workshop-trained faculty. While our workshops and subsequent implementations were relatively small scale, our multiple levels of assessment allow us to address both the effects of the workshop training on faculty participants and the response of their students to the CREATE strategy.

Our participants were a self-selected group of faculty members who applied to the workshop because of their strong interest in science education. Although the participants were already carrying out a number of CREATE-type practices, our results indicate the workshop series was effective in shifting their attitudes, particularly with respect to the use of specific CREATE tools such as concept mapping, as well as in the use of precourse student assessment and making reference to the science education literature. We attribute this success in part to the dual function of the workshops in providing both a theoretical basis for the CREATE tool kit and associated classroom activities, as well as practical hands-on experience; for example, “student teaching” in workshops 4 and 5. Participants’ anonymous postworkshop SALG surveys (average: 4 = “much help” for each of the five workshops) support this view, as does the OE's summary evaluation of the workshop series as “overall excellent” (Hurley, 2009).

Hypothesis 2: Workshop-trained faculty from diverse colleges and universities would teach CREATE effectively in their first attempt.

As described above, previous research suggests that effective workshop training does not guarantee successful implementation of new pedagogical strategies. Some projects have had strong starts with workshop training of faculty and additional plans to subsequently implement new teaching methods, but foundered when participants failed to follow through in the implementation process (Silverthorn et al., 2006). Our faculty implementers agreed to file paperwork with their campus IRB board, to be observed twice by the OE, to return to the CUNY Graduate Center for midsemester discussion and troubleshooting, and to administer pre- and postimplementation surveys and tests to their students. We think it is likely that the workshop leaders’ continued contact with the implementers contributed to the participants’ persistence in carrying out these essential activities. In addition, as this project was supported by a grant from the NSF, we were able to provide stipends to the implementers in recognition of the time and effort they contributed to the project.

The OE noted that faculty participants differed substantially in teaching experience and their preworkshop understanding of pedagogy and that it is likely that all participants experienced traditional teaching during their own scientific training. Despite these variables, the OE determined that each of the seven implementations was “largely successful,” and her assessment of individual CIs ranged from “moderately effective” to “extremely effective.” Differences were attributed largely to individual differences in the implementers’ levels of teaching experience and in their student cohorts, as well as in the particular approaches each implementer used when teaching CREATE. Some had larger classes or initially resistant students, which made introduction of a novel teaching strategy more challenging. Student resistance to change in teaching methods is not unusual (Dembo and Seli, 2004; Seidel and Tanner, 2013); change is perhaps particularly difficult for students who have achieved success in traditional classrooms. The duration of implementations as well as their timing within a given semester may have been an additional factor influencing CI effectiveness. We anticipate examining these issues in our ongoing expansion of CREATE implementations to 20 4-yr and community colleges in courses taught by a subset of the 96 faculty members we trained in intensive summer workshops in 2012–2013 (K. Kenyon and S. Hoskins, study in progress as part of DUE 1021443).

Although all student participants were aware that their class was participating in a science education study, the rationale behind the CREATE method was not explained by all of the CIs. This initially created anxiety for some students, as described in the following comments made in response to an anonymous survey administered by the OE (full data set not shown): 1) “At the beginning of the course the CREATE methodology was overwhelming, because I was accustomed to being expected to fully understand what I read very quickly. I was nervous about ‘not knowing’ until I realized that there would be time for me to explore the content and become familiar with it. This process of discovery has rarely been found in my other science courses … and is much more enjoyable and rewarding than being handed information and connections. It was empowering to realize that I can understand research in a field I was not familiar with, without the help of a textbook.” 2) “While at first, I was hesitant and relatively unnerved by the CREATE method, I have found it to be very helpful. Not only in terms of understanding scientific articles (via mapping/cartooning), but also in terms of the social aspects of the field and an introduction to the processes (via small group work/mini grant panels).” We suggest that students will adapt more quickly if faculty explain the pedagogical bases of the novel CREATE tools throughout the implementation period.

Hypothesis 3: Students on diverse campuses, in courses taught by CREATE workshop–trained faculty, would make cognitive and/or affective gains paralleling those seen previously at an MSI.

We used three measures, a CTT and SAS that were both administered anonymously pre- and postcourse, and an anonymous postcourse SALG survey to gauge student gains in cognitive and affective areas.

CTT.

As the CREATE approach had been successful in promoting gains in critical thinking in CCNY students (Hoskins et al., 2007), we wanted to assess the extent to which similar gains might be made by students in the CREATE implementations. Our CTT had no questions directly related to module content in any of the implementations and thus was designed to measure potential transferable thinking skills developed in CREATE courses. The implementers’ students, taken as a whole, made significant gains on the numbers of logical statements they made in support of their arguments on all CTT questions (Table 4, A–D). The results from individual implementations are not as strong statistically, because of the low numbers, but most made critical-thinking gains. No major differences were noted when comparing full/partial-semester or private/public implementations, although the full-semester implementations may have been slightly more effective (Table 4, A–D). This might be expected, due to the reiterative nature of the CREATE process. In ongoing work with a larger population of student participants, (K. Kenyon and S. Hoskins, DUE 1021443), we hope to address the relationship between implementation length and student outcomes with more statistical power.

In addition to the results reported here and in our original description of the CREATE method (Hoskins et al., 2007), we have documented critical-thinking gains, assessed by the Critical thinking Ability Test (CAT) survey (Stein et al., 2012), in first-year science, technology, engineering, and mathematics (STEM)-interested students during a one-semester CREATE course at CCNY (Gottesman and Hoskins, 2013). Others have shown that an innovative inquiry-based undergraduate microbiology laboratory course leads to critical-thinking gains, also assessed by CAT, during a single semester (Gasper et al., 2012). We have not compared the gains in critical thinking made by CREATE students with those of students in a more traditional class that is also focused on the primary literature, as was done in a recent study that used a modified version of CREATE (Segura-Totten and Dalman, 2013). These authors reported comparable critical-thinking gains for both groups. We do not consider these findings relevant to the present study, however, as these authors omitted numerous CREATE activities that we consider essential to the power of the approach and also evaluated critical-thinking abilities through an entirely different method (Hoskins and Kenyon, 2014). Although this comparison may be of interest in future studies, our focus here is on the ability of CREATE to bring about multiple gains, in addition to critical thinking, in diverse student populations.

SAS.

The SAS included statements addressing students’ sense of their academic abilities and statements reflecting epistemological belief factors (Hoskins et al., 2011). As described in Results, we saw significant pre/postimplementation changes with large or moderate ESs on multiple factors (Table 5A; changes on individual campuses are shown in Supplemental Material, SD 3). Student self-assessment data must be interpreted carefully, as students tend to overestimate learning gains, for example (Cloud-Hansen et al., 2008). However, when we administered a similar pre/postcourse survey to a comparison group of CCNY students in a semester-long physiology course with lab that was not taught using CREATE, those students did not self-assess as having made any significant gains either in the categories above or in any of the epistemological categories discussed below (Gottesman and Hoskins, 2013). These findings argue that the attitudinal and epistemological shifts that we documented in the CIs’ students did not occur simply as a consequence of student maturation during a typical college semester. Rather, we suggest that the epistemological and attitudinal changes seen in implementers’ students reflect their response to the cognitive challenges of the CREATE approach.

Of the seven epistemological categories defined in our previous study, four (1, science as creative; 2, sense of scientists as people; 3, sense of scientists’ motives; 4, science as a collaborative activity) showed significant change in some of the pooled groups, with the largest shifts in categories 1 and 2 (Table 5B). As in our previous study at CCNY (Hoskins et al., 2011), we saw more changes overall in the science attitudes/abilities categories than for epistemological beliefs; this finding holds both for the implementers’ students and a CCNY 2009 cohort (data not shown) taught by an experienced CREATE faculty member (SGH; see Supplemental Material, SD 3). These results are consistent with the well-known stability of epistemological beliefs (Perry, 1970; Baxter Magolda, 1992), but it is encouraging that changes in some aspects of students’ epistemological beliefs did occur in multiple CREATE implementations.

The positive shifts in students’ sense of science as creative are particularly interesting. The full-semester and private school pooled groups made significant gains on this issue, as did two of the three individual full-semester implementations. We are not aware of other studies in which an intervention focused on primary literature documented students’ self-reported changes in their sense of science as a creative process. In a recent review, DeHaan noted that “Evidence suggests that instruction to support the development of creativity requires inquiry-based teaching that includes explicit strategies to promote cognitive flexibility. Students need to be repeatedly reminded and shown how to be creative, to integrate material across subject areas, to question their own assumptions, and to imagine other viewpoints and possibilities” (DeHaan, 2009, p 172). In a discussion of programs aimed at developing creativity specifically, Scott et al. (2004, p. 363) state that divergent thinking, “the capacity to generate multiple alternative solutions as opposed to the one correct solution,” is an important component of creativity. We speculate that CREATE contributes to students’ increased sense of scientific creativity by providing repeated opportunities for experimental design and encouraging students’ realization that it is possible to interpret the same data in multiple ways and that more than one of the “next experiments” designed by their classmates may be valid.

Previous work with students in introductory college science courses indicates that, following instruction, introductory biology students may shift to a more naive rather than a more mature epistemology (Semsar et al., 2011). A similar trend of science students showing more naive epistemological beliefs postinstruction has been seen in introductory level physics (Adams et al., 2006; Gray et al., 2008) and chemistry (Adams et al., 2008) courses. In the Semsar study, students in upper-division biology courses did not revert to more naive epistemology during the semester, but neither did they show substantial maturation of beliefs or large gains in expert-like thinking. Rather, upper-level students largely retained their precourse beliefs throughout the semester (Semsar et al., 2011). The finding that students in multiple CREATE implementations developed more mature epistemological beliefs about science during their first CREATE course suggests that this teaching strategy, even when taught by inexperienced faculty, has the ability to influence diverse students’ core beliefs about science.

SALG.

The SALG survey designed by the PIs covers a wide range of aspects of the CREATE classroom experience (Table 6). Overall, students rated “the way the material was approached,” “the opportunities for in-class review,” and “the relationship between classwork and assigned reading” as substantially helping their learning. Self-assessed gains in confidence and learning were seen in all implementations. Thus, a single CREATE experience, which takes place in a classroom rather than a laboratory setting, can shift students’ views of research scientists while at the same time enhancing students’ self-assessed confidence in their ability to think scientifically. This is a particularly important finding, as students’ attitudes can positively influence their motivation and, in turn, facilitate both persistence (President's Council of Advisors on Science and Technology, 2012) and success in science (House, 1994, 1995; Oliver and Simpson, 1998; Pintrich, 2004). We are encouraged that, overall, students in the implementations reported positive views of CREATE pedagogy and its influence on their abilities

In Fall 2013, we surveyed the seven 2008 implementers regarding whether they were still using CREATE and its associated pedagogical tools. All reported continuing to teach with CREATE methods, using many or all of the CREATE tools. Some of the original implementing faculty participants described having designed additional creative enhancements of the core strategy, for example, building on the experimental design activities with a grant-writing project. On three of the five campuses where CREATE was tested, the 2008 implementers reported that additional faculty members (not trained in our workshops) have adopted CREATE strategies for their own courses (CREATE implementers #1, 3, 7, personal communications).

CONCLUSIONS

In a provocative recent essay, Charlene D’Avanzo argues that while the Vision and Change biology teaching reform process produced a strong set of recommendations (AAAS, 2011), it is unclear whether we “know how to change,” citing a paucity of evidence from models based in education research and noting that some influential models have achieved substantial traction in the absence of such evidence (D’Avanzo, 2013; see also Henderson and Dancy, 2011, and Henderson et al., 2011, on this point). Others have speculated that scientists’ sense of their professional identities (Brownell and Tanner, 2012) and/or resistance at the departmental or institutional level (Singer et al., 2012) are potential additional barriers to change. Thus, a large gap exists between defining goals for change and implementing them. The recent Discipline-based Education Research report “Crosscutting Themes” section calls for more studies to examine teaching innovations across multiple disciplines, in diverse cohorts of students, and in upper-division as well as introductory-level students. The report also recommends collecting data on “a wider range of outcomes” (Singer et al., 2012).

When a pedagogical innovation produces no impact on students, it is difficult to know whether the innovation is itself ineffective or whether the faculty members implementing it are ineffective, perhaps because they were not well trained. We have addressed these issues in the current study by assessing at each level (workshop, faculty implementation, student outcomes) and by coordinating assessments by the PIs with those of an independent OE. Based on our expansion of CREATE to seven courses taught by workshop-trained faculty on campuses in the New York/New Jersey/Pennsylvania area, we report that the CREATE strategy, aimed at demystifying and humanizing science through intensive analysis of primary literature, is broadly applicable beyond the MSI where it was originally developed and piloted. Further, our data indicate that CREATE is effective in topic areas (psychology and biochemistry) beyond those in which it was originally tested. Finally, our data indicate that both full-semester and partial-semester CREATE implementations produce substantial gains, but that epistemological gains are very rare in the shorter implementations. Thus, the CREATE strategy is effective in the hands of faculty at a broad range of institutions, producing gains in diverse cohorts of students. Faculty members can learn to teach CREATE by participating in interactive workshops and can effectively adapt CREATE strategies to their existing courses. The persistence of CREATE courses taught by the original CIs 5 yr after the initial implementation and the spread of CREATE courses on several implementers’ campuses argue for the strategy's adaptability and perceived value.

We posed three key hypotheses at the outset of our study: 1) Our workshops would be an effective way to convey CREATE pedagogies and change participants’ views about science education and their classroom approaches. 2) Workshop-trained faculty from diverse colleges and universities would teach CREATE effectively in their first attempt. 3) Students on diverse campuses, in courses taught by CREATE workshop–trained faculty, would make cognitive and/or affective gains paralleling those seen previously at an MSI. Each hypothesis was supported by data gathered in the study. We found that our workshops successfully prepared the CIs to teach effective CREATE courses and that students taught by CREATE faculty on a variety of campuses made critical-thinking and self-rated attitude/ability/epistemological belief gains while also self-reporting enhanced learning in multiple categories, deeper understanding of scientists, and more confidence in their scientific thinking ability. These changes occurred at all colleges/universities where CREATE was implemented, ranging from public universities to private liberal arts colleges and one private R1 university. Thus, CREATE is an effective strategy for diverse student cohorts. To our knowledge, CREATE is unusual in being easily learned, low cost, adaptable to multiple content areas, and effective in diverse student cohorts, as well as able to convey transferable analytical skills, provoke positive changes in student self-rated attitudes/beliefs, and provide novel insights into the motivations of working researchers. As such, we believe that this teaching strategy could contribute to solving long-standing problems of student STEM learning and persistence by providing faculty members a framework for change that builds on their existing research skills (see Hoskins and Stevens, 2009, for discussion of this point). CREATE strategies play to faculty strengths and are built on analytical skills that faculty members already possess. They have the potential to transform undergraduate science education.

REFERENCES