What Good Is a Scientist in the Classroom? Participant Outcomes and Program Design Features for a Short-Duration Science Outreach Intervention in K–12 Classrooms

INTRODUCTION

National concern is high over the quality and equity of science education (National Research Council [NRC], 2005; Bush, 2006), and efforts in both formal and informal K–12 education are needed to foster improved public science literacy and to maintain a strong and diverse science workforce. A variety of organizations, including professional societies, universities, government and industrial laboratories, and informal science institutions such as museums and planetaria, seek to aid schools and to engage the public in science, and scientists are increasingly asked to participate in this work by their funders and taxpayers (Dolan et al., 2004; Andrews et al., 2005). To judge the effectiveness of such investments, it is important to understand what outreach programs can achieve in the “best-case” scenario in which they are well run and well implemented.

The “scientist in the classroom” is a common outreach model that seeks to bring to schools the content expertise and enthusiasm of practicing professional scientists to stimulate student learning, interest in science, and consideration of science careers. We use this term to refer to programs that offer short-duration visits to a classroom where the scientist may give a presentation, lead a hands-on activity, or discuss scientific careers with students, and we distinguish these programs from longer-term partnerships (Alberts, 1993). Programs may be sponsored by universities, professional groups, or community organizations with external or internal funding, and they may stand alone or contribute to a portfolio of outreach efforts within an organization. The model of the visiting content professional is also widely applied in art, music, literature, and other fields.

Despite the popularity of this model, the positive effects of scientist in the classroom programs are largely a matter of faith: little research literature documents their effectiveness. Although both audiences and presenters are reported to be enthusiastic about their participation (Hood, 1994; Koehler et al., 1999; Swim, 1999), most of the published literature consists of outreach program descriptions and advice from experienced program developers (Hood, 1994; Koehler et al., 1999; Munn et al., 1999; Swim, 1999; DeLooper et al., 2000; Evans et al., 2001; Pelaez and Gonzalez, 2002; Dolan et al., 2004; Halford, 2005). These descriptions provide valuable information from experienced practitioners, especially in helping to improve program delivery, but many of the claims made are not supported by evidence gathered using methodologically sound research and evaluation approaches.

The reasons for this lacuna in the research base are not hard to understand. Like many other informal science education programs (Crane, 1994), the change goals of science classroom outreach programs are generally long-term and societal—for example, to increase the number of students pursuing education and careers in science, technology, engineering, and mathematics (STEM) fields or to increase representation of women and minorities in science. When the intervention strategy chosen is short in duration, like a scientist's visit to a classroom, the immediate outcomes of such events are primarily affective, compared with other strategies such as curriculum change or teacher professional development where deeper learning may take place. Short-duration intervention strategies are based on a change model (Seymour, 2002) with the premise that developing interest and enthusiasm around science, having positive experiences with science, meeting science role models, and learning about science careers will translate down the road to more students pursuing advanced science education and careers in high school, college, and beyond. As Bruce et al. (1997) point out, positive attitudes and early positive experiences with science have been shown to correlate positively with outcomes such as choice of high school science courses, college major, and careers, but this link has been shown largely for longer-duration, school-based experiences. Evaluations of informal science programs of any type are rare, and to date these evaluations focus on longer-duration, more intensive opportunities such as summer (Knox et al., 2003; Markowitz, 2004) or after-school programs (Fancsali, 2002).

Given the choice of short-duration change strategies, program outcomes are inherently difficult to evaluate. Although they take place in schools, scientist in the classroom programs share difficulties with other types of informal science learning (Crane, 1994): Because they are typically offered to a wide range of classrooms and schools, the student audience is thus inhomogeneous in grade, age, gender, ethnicity, and socioeconomic status. The course, teacher, and school hosting the presentation also differ. When there are multiple presenters and topics, the content, style, and quality of the presentations may vary as well; they are not part of a regular, annually repeated curriculum. In these circumstances, no common success criteria for students, such as a common assessment of student learning, can be established, nor is content learning necessarily the primary objective.

Methodological limitations also apply. Affordable methods, such as surveys, largely measure attitudes, or, if the instrument is sufficiently sensitive and administered both before and after the event, attitudinal changes. Yet, whether these changes will be permanent or lead to real, long-term behavior changes is unknown. We also do not understand fully what leads to attitude change. Responses to surveys “given at the end of the event when participants are revved up … are a better indication of a participant's enjoyment rather than whether the objectives of the activity were met. … While it is certainly important for the success of a program to have the participants enjoy it, from the point of view of larger goals and objectives, it is inadequate” and should be verified by other types of data (Bogue, 2005). Even survey data that indicate attitudinal changes may not be sufficient to establish what caused those changes (Lott, 2003).

In contrast, a very large, long-term tracking study with multiple control groups would be required to demonstrate statistically significant differences between participating and nonparticipating school-aged students in desirable outcomes such as high school completion rates, high school courses chosen, college attendance and major, or career choices. Such a “gold standard” study would be prohibitively expensive, and the likelihood would be small that all confounding variables, such as students' home and school environments, could be anticipated and controlled in the design to enable unambiguous attribution of the outcomes to any particular influence. Thus, the desired ultimate outcomes for short-duration outreach programs are not meaningfully measured by the evaluation tools most readily available. Yet, the very premise of many outreach programs is that these short-duration events can and do have beneficial results on long-term outcomes.

We attempted to address this problem in our study of a well-established and long-lived scientist in the classroom program. The Science Squad, developed by the Biological Sciences Initiative (BSI) at the University of Colorado at Boulder is an outreach program that supports university science and engineering graduate students to give interactive, hands-on science presentations in area K–12 schools. The BSI staff's long experience with the program and iterative use of formative evaluation data to refine it, the data gathered from internal evaluations, and the high demand for the program by local teachers indicated that the program was well executed and well received—successful by all apparent measures. The scientist presenters were extensively trained; their work was observed and refined to ensure that the programs were inquiry based and well delivered; and the program had encountered, but had already solved, some of the problems met by other scientist in the classroom programs (Bruce et al., 1997; Community Resources for Science [CRS], 2006). Together, these lines of evidence were persuasive that the program was functioning well, but we wanted to know more about the specific ways in which the program might be making a difference: What impacts of the program could be documented for K–12 students and teachers and for the Science Squad presenters, and how did these impacts come about? Thus, the program staff (J.G. and her colleagues) worked together with an independent research group (S.L., C.L., and H.T.) to design and conduct a summative study to examine the longer-term outcomes for participants and presenters and to elucidate the processes by which these outcomes arose (Laursen et al., 2004, 2005). We chose qualitative methods for their strength in detecting outcomes that are likely to be primarily affective and attitudinal rather than cognitive, and in understanding processes and interaction (Denzin, 1989).

Beyond its evaluative utility to the program itself, the study findings have broader applicability. The evidence indicates that this program represents a best-case scenario for its type, the short-duration scientist in the classroom outreach program. Thus, a study of this program offers the chance to determine whether such outreach programs, when well implemented, have any lasting benefit. The findings are of use beyond this program because the model is widely used, although little studied, and because we can link program outcomes to elements of the program design that can be incorporated or adapted by others.

In this report, we present study findings about the outcomes for students, teachers, and presenters. Student outcomes are most difficult to measure, yet the findings are consistent as seen by two groups of adult observers in indicating the types of benefits that may be anticipated, by using different types of measures than the typical postevent survey. Moreover, when added to the student benefits, the additional benefits to teachers and especially to the presenters present a powerful argument for the merits of this type of program as one variation within the portfolio of efforts across the outreach community.

Our aim here is to present the research findings in a manner useful for practitioners—the designers, organizers, and sponsors of outreach programs. To shed further light on how the program outcomes are achieved, we also include practical insight about the design and support of the program that may aid others. Thus, this article includes both findings from a qualitative study by the independent research team and advice distilled from years of experience by the program staff. These types of information are based on different kinds of evidence, and we are careful to distinguish them in our narrative. More information about the program design and logistics is available at the BSI website (http://www.colorado.edu/Outreach/BSI/k12/sciencesquad.html) and from J.G.

BRIEF PROGRAM DESCRIPTION

The Science Squad is one of several outreach programs sponsored by the BSI under a series of multiyear grants from the Howard Hughes Medical Institute. The Science Squad comprises four to six graduate students from biomedically related science and engineering departments who each visit K–12 school classrooms to lead inquiry-based, hands-on science presentations.

Science Squad members are selected in a rigorous application and interview process. Once chosen, they receive compensation comparable to teaching assistantships in their graduate departments; most participate in the Science Squad instead of working as a teaching assistant (TA) for a given year, while continuing their graduate research. Together with BSI staff, each member creates a set of four presentations that includes hands-on activities and that are related to their area of scientific expertise. They typically offer these presentations 2 days a week to several classes at one school each day. The program thus provides both an intensive teaching experience to Science Squad members and a science enrichment experience for many K–12 students and teachers.

According to its mission statement, the BSI seeks to “increase the number of students interested in careers in the biological or medical sciences, to strengthen their biology education, and especially to encourage minority and women students entering the sciences” (http://www.colorado.edu/Outreach/BSI/k12/sciencesquad.html). To achieve the latter goal within the Science Squad program (one of several run by the BSI), schools are prioritized for Science Squad presentations that have low-income and high-minority student populations, although other schools may and do participate. Science Squad members are deliberately selected for their potential to be role models for all students.

The program is advertised by mailed brochures and on the Internet. Teachers request program dates by phone or on a Web-based calendar system, and they may schedule up to four presentations during a school year. In a typical year, the Science Squad reaches an average of 14,980 students (46% minority, 56% female) and 273 teachers (data from program averages 1996–2004).

BSI staff members monitor the program through teacher feedback forms and reflective end-of-year surveys completed by Science Squad members. One-on-one coaching, classroom observation, and monthly team meetings provide feedback to members to help them improve their presentations and brainstorm solutions to problems. Additional program features are discussed below, where they bear on and give insight into the study results. Implementation details are the focus of the section titled Developer's Advice: Program Design Choices.

STUDY METHODS

In this report, we share the direct perspectives of some important constituent groups: the Science Squad presenters and the teachers, two groups who were interviewed in the study, and the program director, whose experience is shared directly in Developer's Advice: Program Design Choices. The choice not to include students' perspectives directly in the study design was made after discussion of the attendant challenges, including the general difficulties discussed in the Introduction and the limited funding available for this study.

This choice does not discount the importance of student outcomes, but rather it acknowledges the difficulty of measuring the outcomes that matter. We chose to solicit observations from two groups of adults about children's responses—the behaviors and changes in behavior that these adults noticed in participating students. The two groups observed students under different circumstances: Science Squad members encountered many students unfamiliar to them and for a short time only, whereas teachers could observe their own students in the days and weeks after the Science Squad presentation. That these observations are consistent across the two groups supports the validity of this approach. It is also clear that further research is required to understand the longer-term implications of these medium-term student outcomes.

The interview samples are intentional, reflecting patterns in the nature of the scientist presenters and the teachers and schools seeking their assistance. The interviews included two groups: Science Squad members and K–12 classroom teachers. A nonrandom, purposive sample (Berg, 1989) of 16 “frequent user” teachers was chosen from among teachers who had used the Science Squad for several years, often multiple times a year. Given the dearth of published evidence about the outcomes of short-duration outreach interventions, these teachers were chosen as those most likely to be able to report any effects on students, due to their multiple opportunities to observe students' responses to the program in their classrooms. This choice probably biases the sample toward teachers with a positive view of the program, but a positive view is also typical: Internal evaluation data reflect that few teachers report negative experiences with the program. Indeed, the only consistent negative report from teachers is of difficulties in scheduling, because there are too few Science Squad presenters to meet the demand. The study questions were summative; for formative purposes, it would have been critical to include teachers who discontinued use of the program.

The teachers in the study were all white and mostly female, typical of the teacher population in the schools served. The Science Squad primarily serves secondary schools: of the teacher sample, eight (half) taught high school (grades 9–12), six taught middle school (grades 6–8), and two taught elementary grades (K–5). Five taught in urban schools, and the remainder were in suburban settings of varied socioeconomic status. Three taught in high-minority (≥75% nonwhite) schools. In addition to using the Science Squad, most of the teachers interviewed participated in other BSI outreach programs, such as teacher workshops. Overall, the teacher sample has a high self-reported use of inquiry teaching strategies for science and high interest in providing role models and authentic science experiences for their students, more than typical in other teacher populations we have studied, and the findings reflect their perspective.

Interviews with 24 Science Squad members were conducted with former Science Squad members whom we could locate, out of a total pool of 34 past members who had participated during the 10-yr period from 1992–1993 to 2001–2002. The sample of 20 women and four men reflects the historical gender makeup of the program. Most interviewees and past Science Squad members were white. Members came primarily from biology departments but also anthropology, engineering, and geography departments. Their Science Squad experience ranged from one to six semesters; multiyear participation accounts for the low total head count. Most participated as graduate students, with a few postbaccalaureate or postdoctoral scientists. Many had extensive teaching experience with youth in a wide array of informal and experiential education settings, and all expressed high interest in teaching.

Interview protocols addressed the benefits to students, teachers, and Science Squad members of participating in the program and how these benefits were achieved. Benefits questions in particular asked for respondents' evidence for their claims: What behaviors did teachers or members observe? By which students? What examples could they provide? We also asked participants about their motivations for participating; their experiences with the program, including difficulties or costs to them; and we invited their advice to the program staff.¹ Teachers also described their classroom demographics, courses, and preparation for and follow-up to the programs. Science Squad members described their education and career paths and the Science Squad's role in their career decision-making. We also asked participants to respond to a common critique of programs like this program, that a one-time intervention may not yield any longer-term effects. These protocols were also adjusted throughout to incorporate newly emergent issues into subsequent interviews. Teacher interviews lasted 20–30 min; Science Squad member interviews lasted 40–60 min. The interviewers clarified the independence of their role relative to the staff and reiterated the confidentiality and anonymity of the interview data.

Our choice of ethnographic interview methods is well suited to this study. Grounded in methodological traditions from sociology, anthropology, and social psychology and commonly used in education, ethnographic approaches can uncover and explore issues that shape individuals' thinking and actions. Especially when previous knowledge is limited—as in this topic—findings from ethnographic data may be used to generate hypotheses for experimentation or to develop constructs for survey studies. Modern software tools allow ethnographers to disentangle patterns in very large text data sets and report them by using descriptive statistics. The results from careful sampling and consistent coding of text data can be very powerful, and these features distinguish qualitative research from collection of anecdotes. Readers more familiar with experimental methods may find the article by Green and Britten (1998) a helpful introduction to qualitative research.

Our methods are detailed in Seymour et al. (2004) (also see Berg, 1989). Briefly, interviews were conducted by phone, audiotape recorded, and transcribed verbatim, with the respondent's written consent. As transcripts were searched for information bearing on the research questions, text segments referencing distinct ideas were tagged by code names by using N'Vivo qualitative software (QSR International, Doncaster, Victoria, Australia). Codes were not preconceived but empirical: Each new code marked a discrete idea not previously raised. Because the codes and passages were linked in the software, a data set was amassed for each interview group. Groups of codes that clustered around particular themes were given domain names, and these clustered codes and domains defined the themes of qualitative analysis. In essence, coding an interview transcript may be compared with disassembling a necklace and sorting the beads by color, shape, size, and other variables.

For the Science Squad member interviews, ∼400 codes were developed and sorted into 18 major domains, such as motivations for participating, personal benefits, personal costs, teacher benefits, student benefits, career impacts, graduate school history, advice to project staff, and gender issues. Although many broad domains (e.g., personal benefits) could be anticipated from the research questions and interview protocols, they and their subdomains were developed from the data, as each broad domain was categorized and analyzed to build explanatory accounts—for example, sorting Science Squad member benefits into the types described below, or developing explanations for the high representation of women on the Science Squad. For the teachers, ∼160 codes were developed and categorized into domains such as classroom demographics, observations of student interest, differences in student reactions, their classroom practices, and observations about Science Squad presentations and about the Science Squad program as a whole. The analysis was conducted by the independent evaluators, but throughout the analysis, emergent findings were shared with the project staff (J.G. and colleagues), and their input provided clarification, advice, and a “reality check” on the findings.

It is sometimes useful to count the frequency of use for certain codes across the data set, by using conservative counting conventions to avoid overestimating the weight of opinion. Together, these frequencies describe the relative weighting of issues in participants' collective report. They hypothesize the strength of particular variables and their relationships that may later be tested by surveys or other means. These counts are not, however, drawn from random samples of identically conducted interviews; they are not subject to statistical tests. In this article, we count speakers rather than codes—that is, we report the number of individuals making an observation in a particular category rather than the total number of observations in that category. Following the bead analogy, we count the number of necklaces on which one or more beads of a certain type occur, rather than totaling all the beads of that type.

STUDY FINDINGS: OUTCOMES FOR K–12 STUDENTS AND TEACHERS

Student Gains

Evidence of student gains comes from both teachers and Science Squad members, through their observations of student response to the programs—classroom behavior, engagement in activities, verbal and affective responses, questions asked, and answers given by students. Teachers were able to observe and report evidence that emerged over the longer term, such as student comments or questions in later classes, or further investigation of the presentation topic in a paper or project. Their prior knowledge of their students enabled them to notice responses such as enthusiasm from a student not usually interested in science. However, they only saw the response of students in their own classroom. Science Squad members' observations had countervailing advantages: they had no preconceived notions about particular students, such as whether they were “good students” or not, and they saw many more students in a much wider variety of classrooms. The high corroboration in the types of student gains reported by these two groups is thus notable. Their different perspectives account for differences in the extent to which each group observed these gains.

The student gains fall into three categories, summarized in Table 1 and discussed in more detail below.

Enhanced Interest and Engagement.

Enhanced interest and engagement was reported by 14 of 16 teachers (88%) and 22 of 24 Science Squad members (92%). As evidence for these benefits, teachers reported student behaviors such as concentrating on the activities, asking questions, and stating their interest. No teachers reported lack of engagement or interest.

Several teachers reported that all their students were engaged—despite variations in grade level, course titles, audiences, and levels (e.g., introductory or advanced placement)—and ascribed this engagement to the inquiry style of the presentations.

Others noticed the “leveling” effect of Science Squad presentations, enabling both high- and typically low-achieving students to succeed.

Science Squad members also observed students' interested responses to their presentations. They attributed student interest to their choice of inherently interesting topics and the engaging, hands-on activities and authentic science materials and equipment that they brought. They hoped that this interesting one-time experience would spur students' longer-term interest in science: “I would do things on primate behavior and primate evolution and human evolution and spark some interest in things that they hadn't really thought about before.”

Also important was the novelty of a presenter different from the regular teacher. Both teachers and Science Squad members noticed that students perked up and paid attention to someone different from their everyday teacher, someone perhaps younger and more “cool.”

New Views of Science and Scientists.

A second category of gain, also broadly observed, was changes in students' views of science and scientists. Seven teachers (44%) and 24 Science Squad members (100%) reported this student gain. Moreover, many teachers stated that they deliberately used the Science Squad to provide scientific role models for their students. They wanted students to see that science was something real people did and enjoyed and that science provided many career and education options. This was something they could not easily provide in their role as teacher.

Science Squad members, too, described seeing changes in students' stereotypes about science and scientists. A fun and successful science experience could challenge students' belief that science was boring.

By sharing their enthusiasm about science and their personal stories of entering science, Science Squad members were able to broaden students' views about what scientists were like, who can be a scientist, and the many paths to pursue within science.

This benefit extended not only to members of the same group as the presenter but also to other groups. For example, stereotypes held by both boys and girls were dispelled when a woman Science Squad member presented.

The majority of presenters were women (20 of 24 interviewees), and some were members of ethnic minorities underrepresented in STEM fields. We discuss the high representation of women elsewhere (Laursen et al., 2005) and in a forthcoming article.

Presenters' perspectives could help students understand the concrete steps needed to pursue a career in science and could make these steps seem realistic and possible.

Both teachers and Science Squad members emphasized the importance of sharing career ideas with students, information not always accessible to all students, as this Science Squad member pointed out.

Understanding of Science and Its Relevance.

Finally, teachers and Science Squad members reported gains in students' understanding of science—concepts, skills, and new connections. However, consistent with the short duration of the intervention, this benefit was much less widely reported than gains in interest and new views of science—only eight of 24 Science Squad members cited it, and six of the teachers. Most often these learning gains were related to scientific skills, such as collaboration, critical thinking, and problem solving, rather than specific subject matter content.

Also important were the connections between science and everyday life that students made.

The relative paucity of learning gains as a student outcome observed by the adults is notable—but it is also not the primary objective of the program. This paucity is also consistent with the short duration and “special topic” nature of the presentations (even if aligned with the school curriculum through the teacher's efforts). Other studies also find that, although students may learn new vocabulary, the larger impacts are on their interest and enjoyment of science (CRS, 2006). These findings support our argument in the Introduction that student learning assessment is not the most appropriate outcome measure for programs of this type. However, the learning gains that were reported are well aligned with community-sanctioned science learning goals for students as represented in the national and state standards, particularly those emphasizing the abilities and understandings of inquiry (NRC, 1996).

Overall, the gains observed by adults among K–12 students emphasize development of interest and new views of science. These findings are consistent with those from studies of student outcomes for short-duration outreach programs that have used other methods (Bruce et al., 1997; Bottomley et al., 2001; CRS, 2006). For example, Bruce et al. (1997) found that even very young students found one-time scientist presentations special and memorable, as seen in their drawings and conversations, and were more likely to identify scientists as people like themselves. New views of scientists and novelty were student benefits reported by Trautmann et al. (2002). These gains also were those sought by the BSI staff in their program design.

Importantly, the K–12 student gains are all related to science literacy. Although these gains cannot be said to define science literacy (a complex construct), they do contribute to development of science literacy by engaging students' interest and providing views of science as an accessible, human activity (Eisenhart et al., 1996; Bybee, 2005). Hands-on, personally relevant learning experiences have been shown to be effective in inspiring students, especially girls, to continue to take classes in science and possibly pursue a career in the field (Fancsali, 2002; Eisenhart and Edwards, 2004). Students' gains are in domains where scientists' contributions have been argued to be greatest (Alberts, 1993; Bower, 1996; Bybee and Morrow, 1998; Colwell and Kelly, 1999; Laursen, 2006).

Teacher Gains

Teacher gains included those reported by teachers themselves and by Science Squad members who made observations or conversed with teachers about their gains. Again, the high degree of corroboration between the types of gains reported by these two distinct groups of observers increases the reliability of the findings. When asked about the benefits of the program to themselves, teachers often described benefits that were in fact realized by their students. Because these gains helped teachers be more effective in their work of teaching students, it was difficult for teachers to separate their own gains from their students'—they valued the student interest and learning that they observed and saw it as supporting their work; thus, they described this as a benefit to themselves as professionals.

As outsiders, Science Squad members were more easily able to identify benefits to teachers that were separate from enhancement of their classroom work. All 24 members, and four teachers, identified gains in teachers' understanding of science topics and ways to teach them effectively. Because teachers are responsible for a wide range of knowledge, their knowledge in any one area may not be deep, they may be teaching out of discipline, and it is difficult for them to keep up with cutting-edge science.

Science Squad members reported examples of helping teachers clarify particular concepts and providing technical advice to improve experiments or activities. They saw teachers benefit from seeing new approaches to teaching familiar content. Sometimes teachers gained specific ideas for classroom activities to use again, and other times simply a new perspective.

These benefits came to teachers through seeing how a content specialist approached the lesson, by direct participation in the activities, and by observing their students' reactions to the lesson.

Science Squad visits sometimes had lasting effects on classroom practice. Several teachers reported using Science Squad activities on their own later. Most teachers in this sample of frequent users reported that they already used inquiry pedagogies in their own teaching. Thus, only a few reported changes to their teaching style as a result of seeing inquiry-based teaching modeled in Science Squad presentations. However, teachers did report that observing Squad presentations reinforced their belief in the importance of active learning, because they saw their own students benefit from the hands-on, minds-on approach used by the Science Squad. The opportunity to step back and focus on student learning was valuable in itself. Although each Science Squad visit represents a single opportunity to observe students, the benefits reported by teachers are consistent with those cited by professional development opportunities, such as lesson study, that include collegial observation (Stigler and Hiebert, 1999).

Science Squad members' reports of their conversations with teachers, follow-up requests by teachers, and other teacher interactions indicated a broader degree of influence on teachers' teaching styles than did the teacher interviews. This difference is consistent with the differing perspectives of the two groups: Squad members interacted with a more varied group of teachers, not just the inquiry-supportive group represented by the frequent users.

In addition to their own professional learning, teachers benefited from a break in the routine. The Science Squad visit provided a respite from the everyday grind of planning and leading lessons. Last, teachers gained a variety of types of emotional support, including validation of their work through the support of a university program, collegial interactions with Science Squad members, and the intrinsic pleasures of learning with their students.

Negative Outcomes for Students and Teachers

We found little evidence of negative outcomes for students and teachers. In large part, the negative evidence reported was about missed opportunities or gains not made that might have been made in other circumstances. For example, a few teachers reported that their students did not consider Science Squad presenters as role models, because their contact time was too short to establish a meaningful personal connection or because the students were too young to view the presenter as a possible “future self.” An occasional case of poor fit occurred when a (usually novice) presenter aimed a presentation too high or too low for the student audience. Science Squad members reported classroom sessions to be less effective when teachers did not adequately prepare or supervise students. Most Science Squad members reported experiencing such situations, but as a small fraction of their total experience. Presenters and teachers alike reported that good advance communication clarified expectations and reduced the likelihood of problems.

The most frequent problem reported by teachers was the inability to schedule a particular presentation or preferred date because it was already booked. This is a negative for the individual, although it reflects the success of the program as a whole. Our data do not show whether scheduling was a significant issue for those who stopped using the Science Squad, or, conversely, whether reports of scheduling problems were greater among these teachers who wished to use the Science Squad often. A Web-based scheduling system was introduced recently, after our interviews, and the project reports that this has reduced scheduling problems substantially.

STUDY FINDINGS: PROGRAM ELEMENTS THAT SUPPORT BENEFITS FOR K–12 TEACHERS AND STUDENTS

Teachers were particularly insightful in identifying features of Science Squad presentations that led to the positive student outcomes they observed, and Science Squad members echoed these views. Often teachers discussed these features as factors in their decision to invite the Science Squad back to their classroom year after year. Key features included the following:

• Equipment and materials that enabled science learning experiences that were authentic and often novel for students (cited by 69% of teachers). Teachers valued the opportunity for students to experience new areas of science—for example, forensic botany or DNA fingerprinting—and use real scientific tools, such as bacterial cultures inoculated with a virus, or primate bones and skulls.

  “One of my major roles was bringing in concrete things to the classroom—either slides that I had of unusual environments, or lab demonstrations, microscope preparations…. The schools that I worked in, I felt, would not—the students would not have seen live organisms. They would not have seen these type of preparations under the microscope.” (Science Squad member)

• Science topics that were interesting, contributed to the curriculum, and taught both students and teachers (63% of teachers). Teachers valued Science Squad members' scientific expertise and the relevance and currency of the topics they chose.

  “In biology, you cannot be knowledgeable about everything, so that there are people who know a lot about diabetes, or know a lot about phosphorescence, or something of that nature, and they come as experts to your class. They would know more on that particular subject than I would, so it's a good thing to have them come.” (teacher)

• Style of presentation (50% of teachers). Teachers mentioned the hands-on and inquiry approaches that were integral to the Science Squad presentations, and the fact that Science Squad members were well prepared to lead their activity and adapt it to their audience.

  “The biggest impact is being able to provide the students with exposure to a variety of labs that we otherwise couldn't do. Because the labs are what makes science live…. The greater the variety of lab topics, the greater the possibility of reaching kids for science.” (teacher)

• A break from routine (50% of teachers). Teachers appreciated the way a novel presenter could provide new perspectives and bring students to a high level of alertness and interest.

  “I think it's really good for my kids to really be exposed to a lot of different people and not just hear one point of view, not just hear one teacher. I think they love having someone different talk to them.” (teacher)

These features gave rise to the student benefits observed and were valued by teachers. In parallel with their students, teachers benefited when they were actively engaged in the presentation themselves, participating as learners, and supporting the presenter's work with their students. By contrast, when teachers did not actively participate, Science Squad members reported that the presentations were less effective, and benefits to both students and teachers lessened. Emotional benefits to teachers arose from collegial interactions with the visiting Science Squad member—planning and follow-up, having lunch together—and their awareness of the effort and interest of Science Squad members in coming to schools and doing a good job.

In sum, both the pedagogy and content of the Science Squad presentations were crucial to the Science Squad's effectiveness in the classroom. These reflect choices made by the program—aspects of program philosophy, selection and preparation of Science Squad members, and program structure—that are discussed below. Teacher engagement was a critical variable in mediating the effectiveness of Science Squad presentations. Teachers were able to further enhance the benefits of the Science Squad benefits by preparing students for the session and following up on it later, making connections to their curriculum, and adapting effective activities for their own teaching.

STUDY FINDINGS: OUTCOMES FOR SCIENCE SQUAD PRESENTERS

Gains Reported by Presenters

In addition to the positive outcomes observed for K–12 teachers and students, the program also had substantial benefit for the Science Squad presenters. These benefits include the following types (with the percentage of members reporting any gain in each category in parentheses) and are discussed in more detail below:

• Gains of skills—in teaching, communication, and management (83%)

• Gains in understanding, particularly of issues surrounding education and diversity (92%)

• Personal gains, including growth in confidence and intrinsic or emotional rewards (83%)

• Career gains (96%)—primarily in the form of transferable skills and understanding, but also résumé enhancement, career networking, and materials used in later work. Career gains also include effects on members' career paths, such as clarification and confirmation of career plans or new career ideas based on the Science Squad experience.

Gains in Skills.

Gains in teaching skills were reported by most Science Squad members (83%). These included component skills, such as communication and organizational skills, but they were counted as teaching skills when described by interviewees as applied in a teaching context. Members gained skill in explaining and demonstrating complex scientific ideas to a variety of audiences, finding simple language and vivid analogies. They came to understand their own science better through explaining it to others with less background knowledge.

An important step was learning to adapt their presentations to the different audiences they encountered:

Members developed both a recognition of the importance of adjusting to their audience, and the ability to do it quickly in a wide range of settings. They gained skill in using interactive teaching strategies and developed a belief in their value. They developed practical teaching skills, such as lesson planning, estimating time, developing low-cost and age-appropriate materials, and managing classroom dynamics.

Perhaps most importantly, participants were able to link these skills together to develop a coherent, personal teaching style. Even those with previous teaching experience found in the Science Squad an opportunity to refine, reflect on, and improve their teaching style.

Although communication is an element of teaching, 15 members (63%) reported gains in communication skills that they distinguished from teaching skills. This included increased confidence in their public speaking abilities, lowered anxiety, and the ability to select appropriate visuals and language—skills seen to apply widely beyond the classroom. In addition, five members (21%) reported gains in organizational skills, such as managing a complex schedule and planning ahead.

Gains in Understanding.

Gains in understanding were a second large category of benefit to Science Squad members, encompassing a range of issues affecting public education: diversity, social inequities in educational access, articulation between K–12 and higher education, student learning and development, and the work of teachers and schools. Gains in understanding of diversity and equity issues were particularly strong, reported by 17 members (71%). As they interacted with students and teachers in schools served by the Science Squad, members were exposed to broader diversity than they had encountered in their upbringing or in the university and were disturbed by the inequities they saw.

They came to realize the importance of high-quality education for changing students' economic and career prospects, and to understand the potential impact of cultural differences in the classroom—for example, in the different ways students interacted with their teacher. One interviewee articulated particularly well how greater awareness of diversity could also increase motivation to solve these social problems.

Members also reported gains in understanding how students learn at different developmental stages, and how to reach diverse learners. Greater understanding of the nature of teachers' work and school systems helped them to appreciate the difficulties faced by teachers and respect the teachers who persisted under challenging conditions. Understanding the varied backgrounds of their future students was helpful to those planning to teach at the college level.

Personal Gains.

Gains in confidence were reported by 17 members (71%). These were not gains in general self-esteem, but confidence in their ability to do the work at hand—to communicate science to others, to manage a classroom, to represent their STEM discipline.

Confidence gains transferred to their professional lives, providing new ideas about future work options and the ability to compete effectively for teaching positions.

In addition to confidence gains, members reported intrinsic emotional benefits: “I got a lot of personal warm fuzzies, which are important, you know.” It was gratifying for them to see students learning, having fun, and appreciating science. Members met students who remembered them, received thank-you notes, and felt inspired by seeing teachers do good work in difficult settings. They had fun with the students and enjoyed collegial connections with their Squad cohort and the BSI staff. Several members reported these “warm fuzzies” as a change from their research environment, where positive feedback was harder to come by.

Career Gains.

Last, Science Squad members who had entered careers² reported a number of ways in which these benefits had transferred to their work. They applied their teaching and communication skills to their jobs, reused teaching materials developed for the Science Squad, saw positive reception of their résumés during job seeking, and developed useful professional networks.

In addition to these concrete career benefits, members gained understanding about their career options and interests that helped them to make better decisions. Squad members described a variety of ways in which their experience had confirmed, clarified, refined, or altered their career goals and paths. All interviewees were still working in STEM fields or STEM education—one-third (8) in higher education and one-quarter (6) in K-12 education or outreach. Importantly, 20 of the 24 interviewees (83%) reported some significant effect of Squad participation on their career path—a figure much higher than we expected. We describe our extensive data on these career path outcomes and the role of Squad participation elsewhere (Laursen et al., 2005) and in a forthcoming article. The extent and depth of career influence from Science Squad participation for these individuals, who are already highly interested in teaching, may also reflect the general lack of other experiences supporting career development in teaching for STEM graduate students (Smith et al., 2002).

Negative Outcomes for Presenters

Although Science Squad members were unanimous in stating that the benefits to them outweighed the difficulties, they did report three types of negative outcomes of their participation:

• Problems of time, travel, and logistics (reported by 71% of members)

• Emotional costs, such as discouragement and frustration (79%)

• Marginalization or lack of support by advisors and colleagues (79%)

Problems of time, travel, and organization stemmed from the challenging schedules that Science Squad members maintained. In addition to giving presentations and traveling to schools, members had to plan their calendar, respond to teacher requests, and prepare their materials and equipment. Logistical glitches were a fact of life, and traffic conditions or bad weather could add stress to travel.

Science Squad members reported that they kept their work time to match their paid commitment (typically half-time), but found participating on the Science Squad to require more than the nominally equal time commitment of a teaching assistantship. They had to use time efficiently to make progress on their research and satisfy their advisors. This was challenging after a full day in a school.

Emotional costs were incurred when students did not respond to their efforts or when teachers did not support them. Some were discouraged by the low quality of science education in the schools they visited, and they felt their efforts could not ameliorate this daunting problem. Others expressed frustration at seeing new groups of students each day, without a chance to build relationships with students or see their progress.

Last, 19 members reported some level of marginalization or negative response to their participation in the Science Squad, or to the career choices that it implied, by their graduate advisors, other faculty, or peers. They received both overt and implicit messages that teaching, especially in K–12 schools rather than in the university, was less valued than research. Although marginalization may be a factor that deters some from applying, it did not deter this group of actual participants; however, they did exert some energy in coping with and responding to these negative messages. We discuss this issue in detail elsewhere (Laursen et al., 2005) and in a forthcoming article. Although some members reported changes in career path, such as ruling out the possibility of teaching high school science after trying it as a Science Squad member, they did not count this as a negative outcome but rather as positive clarification of their career interests.

STUDY FINDINGS: PROGRAM ELEMENTS AND PROCESSES THAT GIVE RISE TO PRESENTER GAINS

The interview data revealed the Science Squad experience to function as a particularly effective form of experiential learning about teaching. Repetition was one crucial element: By offering a few programs in many classrooms, Science Squad members could refine their presentation, experiment with different approaches, and observe students' responses closely.

Participants were also able to observe “a whole flock of different teachers” in many schools. Insight came from both positive and negative examples of how teachers presented material, interacted with students, managed classroom dynamics, and handled problems.

In the best cases, this was accomplished not just by observing teachers, but by collegial interaction with them. This occurred in coteaching lessons, in discussing a lesson together afterward, and in collegial feedback from the teachers to the Science Squad presenters, as each brought mutual respect to the other's differing expertise.

Formal training sessions also contributed to skill development.

Together, their training, classroom repetition, observation of skilled teachers, and group discussions amounted to an intensive teaching practicum for Science Squad members. Members could apply their ideas to real teaching situations and then analyze and discuss them afterward. Even members with formal training in teaching found this combination beneficial.

The structure of the program, with visits to a large number of classrooms, fostered these benefits, but it also had some negatives for Science Squad members, primarily the inability to develop lasting relationships with students and to observe their learning and progress. Programs that build in more sustained engagement in a single classroom likely address this—but also run risks of reducing impact in other ways, such as loss of the novelty and excitement of a visitor, and reach many fewer classrooms with the same resources. Moreover, programs where a university student works with one classroom all year have reported difficulties in relationships between the student and teachers—when long-term classroom planning and pedagogical skills become more prominent in the student's role, their science expertise may seem less relevant or appreciated, and lessons may become traditional rather than inquiry based (Coates, personal communication; Bruce et al., 1997). Tradeoffs such as this are discussed in the following section.

DEVELOPER'S ADVICE: PROGRAM DESIGN CHOICES

In this section, program features are shared by the program director (J.G.), whose long experience designing, running, and refining the program gives additional insight into how design choices affect the observed outcomes.

Recruitment, Selection, and Training of Presenters

Recruitment begins with information sessions led by program staff and current Science Squad members, assuming that the better informed the applicant, the better the applicant pool. Previous presenters are often the best recruitment tool. Although applicants well outnumber the available positions, a shortage of minority candidates reflects the lack of diversity in the university's graduate student population. The written application includes several essay questions, two letters of recommendation, and a curriculum vitae, and it is followed by personal interviews.

Careful matching of applicant characteristics and program goals selects for presenters who are likely to succeed with students and to benefit themselves. Motivation is critical: alignment with BSI's mission, desire to “give back” to the community, and dedication to communicating science. Also important are sensitivity to educational and cultural differences and an ability to communicate well with a variety of people. The staff considers past teaching experience and judges potential to be a role model for youth, to develop novel, engaging presentations in their scientific field, and the “fit” and appeal of those topics in the K–12 curriculum. Also valued are maturity, flexibility, and organization. Over time, this extensive review process has reliably yielded effective presenters.

Applications are typically due in February, with interviews in March. April notification means new members have secure funding and time to prepare. New members shadow current presenters to get a clear picture of the work, learn in situ, and spur concrete thinking about their own science presentations. The staff leads group sessions emphasizing science inquiry, best teaching practices, classroom management, and role playing. Monthly group meetings provide a collegial forum to practice presentations and troubleshoot challenges. The program staff aims to set a tone of accessibility, collegiality, and support for the presenters. Research findings concur, as Science Squad members report they appreciate and feel inspired by the support and professionalism of BSI staff.

Development and Support of Presentations

After selection and shadowing, staff members meet with individuals to discuss their ideas for presentations and share resources, including an annotated library of past presentations with outlines, activities, and notes about implementation and logistics: timing, pacing, differentiation for student groups, and solutions to common problems. Each member prepares four or five presentations in their area of expertise, which are publicized in August. From that menu, teachers request specific presentations for their classroom via e-mail or phone directly to each presenter. Presenters compile, prioritize, and schedule those teacher requests on their own.

Staff members help new Science Squad presenters translate their initial ideas into hands-on activities and think through the logistics of conducting them with multiple classes of 30 students in a day. A lab coordinator helps each member to identify and acquire the needed materials and supervises undergraduate assistants who help with lab prep, restocking consumables, and photocopying; lab and storage space is set aside for this. To stretch the supplies budget, the program shares with and borrows from other campus units.

Focusing on High-Need Schools

Because increasing diversity in science is central to the program's mission, Science Squad efforts are concentrated in greater Denver schools with high-minority (≥50%) populations, as determined from state enrollment data. Presenters are asked to prioritize such that 75% of their presentations go to these schools. This approach generally ensures that ∼50% of students reached are ethnic minorities.

Teachers and Science Squad members suggest that the student outcomes may be most important in these high-need schools. This choice also benefits presenters: Because they visit both resource-rich and resource-poor schools, they often report that the Science Squad experience is an “eye-opener” in revealing disparities in public education and students' preparation for college.

With the high demand, prioritization and scheduling can be a challenge. A policy limiting presentations to four per year per teacher helps to increase fairness and spread the benefit. Most teachers are understanding—although they advocate for their own students, they recognize others' needs and support the broader goals. That many teachers participate in other BSI programs (e.g., teacher workshops and outreach grant awards) probably also helps.

Teacher Engagement

Science Squad presentations work best when teachers are engaged and set an example for their students. Guidelines mailed before the session emphasize tips that “help make the Science Squad visit the best learning opportunity for students.” Teachers are asked to communicate their students' background and needs to the presenter in advance, prepare their students, introduce the presenter, and remain in the classroom at all times to assist and monitor student behavior. It is made clear that the Science Squad member is a specialist in science, not in classroom management: that responsibility remains with the teacher. Likewise, the guidelines indicate what the teacher may expect of the presenter, such as punctual arrival and respectful treatment of students. To inspire complete reading, important scheduling information is included. If guidelines are not followed, teachers may become ineligible for future Science Squad visits. This is rare but invoked if problems are recurrent. More commonly, the program director helps to resolve problems reported by teachers or presenters, which are typically misunderstandings spurred by incomplete communication and lack of time.

The Choice of Focused or Distributed Intervention

A significant decision for any outreach program is the choice to offer a focused intervention to a small number of participants, or a less extensive intervention distributed to a larger number of participants—“a lot to a little, or a little to a lot,” as one Science Squad member put it. Initially, the BSI staff experimented with both approaches, giving members the option to work in one school with one or a few teachers, in addition to the current, distributed model. Several challenges to the focused model became clear to program staff from evaluation data. Teachers' desires for help with their science curriculum were ongoing, but it was challenging for members to keep coming up with something new, especially outside their area of expertise. With their pedagogical role unclear, members could devolve into just another adult set of hands, doing jobs such as grading and supervising small groups, and the benefits of their subject matter expertise were lost. When students came to view the Science Squad member as a “regular” teacher, the benefits of novelty, specialness, and role modeling of a science career were diminished.

This issue was also investigated in the interview study. Program participants recognized the inherent tensions of distributed versus focused interventions but also offered arguments supporting a distributed intervention: the high numbers of students reached, the possibility of multiple encounters in a year, the intensity of learning during a special event, the potential to inspire, and the importance of a single counterexample to stereotypes, showing students that a woman can be an engineer or a Latino can be a biologist. It is also clear from the interviews that visiting a large number and variety of schools was crucial to presenters' development of teaching skills and increased understanding of diversity and education. Although development of some skills—such as classroom management and individualizing instruction—might be greater from a longer-term classroom placement, other fundamental and transferable teaching skills were strongly reinforced by exposure to many classrooms and the opportunity to practice and refine. Without repetition, such growth would not otherwise be realized in a time as short as a year.

Last, the longevity and impact of Science Squad members' work is increased by other program elements. In addition to contributing their presentations to the support library, members may lead a teacher workshop at the end of the year. Workshops highlight the strongest aspects of the presentations and offer background information to teachers who want to use these elements in their own teaching. The chance to work more extensively with a group of teachers benefits Science Squad members as well.

CONCLUSIONS AND IMPLICATIONS

We have used a qualitative approach to demonstrate that a scientist in the classroom outreach program has benefits for K–12 students and teachers. Teachers and presenters observe students' engagement and interest in science, exposure to new science learning opportunities, and changing ideas about what science is and who can do it. Teachers benefit by learning new content and new ways to teach it, and they feel supported by the presence of interested individuals from the university. We conclude that, when well run and carefully structured, scientist in the classroom programs can have a positive impact on students' interest in science and thus their eagerness to learn it. As noted in the Introduction, increased interest is argued to increase students' potential to pursue further education or a career in science. Long-term longitudinal studies to follow the actual educational and career outcomes of students exposed to such programs would be of interest but difficult to conduct. At the same time, the program reaches a large number of students with these benefits.

Perhaps more importantly, the program has substantial benefits for its scientist participants, a group of STEM graduate students who seek preparation for teaching as well as for scientific research. Finding insufficient opportunities within their own departments and degree programs, they discover the Science Squad as an opportunity to learn to teach well and creatively. A high fraction of Science Squad alumni pursue academic careers in institutions with a balance of teaching and research or apply their skills to other types of public science communication. For some, the Science Squad experience confirms their career goals and enhances achievement of them; for others, it introduces new possibilities for careers using their science expertise. In a few cases, Science Squad participation seems to support students to complete a graduate degree and pursue a professional science career who may otherwise be at risk to drop out.

Collectively, studies such as this study offer lessons for the design of classroom outreach programs, helping designers to anticipate potential consequences of certain design elements. For example, could the program be equally successful if offered by undergraduates? (e.g., Bruce et al., 1997.) This approach has appeal as a means to recruit and prepare students to be science teachers and to reduce costs. Our findings indicate some likely tradeoffs: The success of the presenters depends on their inquiry approach and disciplinary expertise, which in turn require sound science knowledge and confidence in it. Research experience seems to be crucial, because presenters must deeply understand the nature of science to be able to effectively communicate the fundamentals of inquiry—gathering and evaluating evidence, drawing conclusions, building and communicating arguments—that are common to all the Science Squad presentations, regardless of topic.

Likewise, some programs are based on the hypothesis that engaging graduate students in a longer-term interaction with a single school will lead to greater student content learning, or to formation of individual relationships rewarding to both students and scientist (e.g., Trautmann et al., 2002). Many of the programs currently funded by the National Science Foundation's GK–12 program are based on this model (National Science Foundation, 2006). But, as we have argued, repetition and exposure to a range of classrooms is crucial in presenters' rapid gains of teaching skills and understanding; thus, this benefit to presenters would likely be reduced in return. Programs that support practicing scientists could provide the needed scientific expertise from otherwise untapped talent pools, not just universities, but they also must solve the challenge of preparing scientists to develop interactive, hands-on presentations and make presentations age appropriate (e.g., CRS, no date; Project Astro, 2001). Our point is not to rank these program design choices but to emphasize that program designers must make appropriate design choices depending on their goals and that they should use research findings to anticipate the outcomes of their choices.

Last, our findings suggest issues for further research. More direct measures of student outcomes are obviously needed, and the qualitative data suggest some domains of student outcomes that might be fruitful for development of other types of assessment. Given the program goals and the experiences of teachers and Science Squad members, it is not surprising that affective and attitudinal gains are much more pronounced than more easily measured gains in science content learning. Long-term impacts of outreach or informal education are of great interest to practitioners who design and run such programs, but they are very difficult to measure. Although it is argued that the impact of a single event may be great, it would be daunting and expensive to design a study to control for every possible relevant factor. The ultimate benefit of such a study is also questionable: As much as we might like a magic intervention that must be administered only once, surely common sense would suggest that what is most likely to work is a series of interventions—related or not—throughout every child's school and youth years—some of which reach children at times when they are primed to benefit most. Society cannot rely on a single event to inspire a future scientist, but it must provide a range of opportunities for excellent science education, in school and outside it.

FOOTNOTES

REFERENCES