Promoting Undergraduate Interest, Preparedness, and Professional Pursuit in the Sciences: An Outcomes Evaluation of the SURE Program at Emory University

INTRODUCTION

Since the late 1990s, undergraduate science education policy, programming, and funding in the United States have shifted increasingly toward the integration of teaching and research. Earlier proponents of undergraduate research (Okorodudu, 1970; Fields, 1971; Pace, 1979; Carsrud, 1980; Peppas, 1981; Palladino, 1982; Prentice-Dunn and Roberts, 1985; Malanga, 1988; Kremer and Bringle, 1990; Mauger, 1990; Delagarza et al., 1991; Schowen, 1998) saw their recommendations validated in the Boyer Commission on Educating Undergraduates in the Research University (1998) report call for a modern undergraduate education blueprint featuring inquiry-based, hands-on learning. Fueled by National Science Foundation's (NSF) Integration of Research and Education grants and with support from foundations such as the Howard Hughes Medical Institute, appreciation for the multiple pedagogic, institutional, and professional benefits conferred by undergraduate research has grown (Table 1) and so has the number of undergraduate research (UR) programs in major research universities and small liberal arts colleges (Mervis, 2001a,b).

Following calls in the late 1990s for evaluation components in UR programs (Manduca, 1997; and Spilich, 1997; Schowen, 1998) and concerns over the quality of previous evaluations (Adhikari and Nolan, 2002; Bauer and Bennett, 2003), a growing number of authors have sought to quantify the benefits conferred by UR. Studies range from quantitative before-after surveys to ethnographic accounts, and include both individual case studies and large-sample studies that pool data from multiple UR settings (Table 1). The growing corpus of evaluative studies of UR to date, however, focuses minimally on the intensive, mentor-based summer program model (cf. Kardash, 2000; Seymour et al., 2004). To be sure, summer UR programs abound and many of them have been described or, in some manner, assessed (Kremer and Bringle, 1990; Foertsch et al., 1997; Alexander et al., 1998; Kardash, 2000; Adhikari and Nolan, 2002; Burnley et al., 2002; Schowen, 2002; Zydney et al., 2002; Shachter, 2003; Lopatto, 2004b; Page et al., 2004; Frantz et al., 2006; Seymour et al., 2004; Gum et al., 2007; Butler et al., 2008). Few studies, however, use methodologically rigorous designs and include both program participants and nonparticipants in assessing outcomes linked to participation.

This article reports on an evaluation of the Summer Undergraduate Research Experience (SURE) Program at Emory University in Atlanta, GA. The evaluation began in the spring of 2005 to assess short- and long-term outcomes associated with participation for the program's first 15 summer cohorts. By means of two methodologically distinct studies—one study grounded in self-reported follow-up survey data (1990–2004 cohorts) and the other study in academic performance data from course work transcripts from a subsample from the 1990–2008 cohorts—we sought to gauge levels of interest in, preparedness for, and actual pursuit of graduate study and professional careers in the sciences. To address gains by more recent participants, we present data gathered by David Lopatto's SURE II-III surveys (www.grinnell.edu/academic/psychology/faculty/dl/sure&cure) that summarize self-perceived gains by 2007–2009 Emory SURE program respondents. We also briefly discuss measures of participant satisfaction per our internal end-of-program survey.

THE SURE PROGRAM AT EMORY: PARTICIPANTS AND COMPONENTS

SURE is a 10-wk residential program that provides full-time, mentored research training in mathematics, life and natural sciences, psychology, and anthropology to undergraduates from Emory and other U.S. undergraduate institutions. SURE is organized and administrated out of the Emory College Center for Science Education (ECCSE), which focuses on the promotion of undergraduate access, interest, and participation in the sciences. Funded primarily by the Howard Hughes Medical Institute (HHMI), and with variable support from sources such as the NSF, Emory University, and the National Institutes of Health, the SURE program has been in operation since 1990. The ECCSE focuses on recruitment and training of U.S. citizens in science fields (NSF, 2005) and encourages “young people's interest in science, prepare(s) them for science-related careers, and increase(s) science literacy among all students, including nonscience majors” (HHMI; www.hhmi.org/grants/institutions). SURE began by serving students working with biomedical sciences faculty. In 1995, we began supporting students in psychology and anthropology; an ethics component was added that year as well. For the period addressed in this study, 218 colleges were represented, and >320 Emory faculty served as mentors. Internal, formative assessments have been conducted each year to review program content and participant satisfaction.

Although applicants can seek appointments with any Emory science faculty, we maintain an online directory of potential research mentors. The majority of our mentors have hosted multiple students through the years, sometimes in tandem. Our program's success hinges on our mentor's belief that SURE activities and infrastructure facilitate and enhance their ability to provide meaningful, well-rounded research experiences.

Program Characteristics and Components

SURE takes place from late May until early August, during which time participants reside in an on-campus dormitory and devote 40 h/wk to program activities. After orientation and training in laboratory safety and research ethics, participants spend the bulk of their time at their mentor's research facility (typically a laboratory). In this setting, students devise, refine, and implement an individual research project, with supervision and training in methods and data analysis from the faculty mentor. Fellows share their research proposal (which includes background on their project, a clearly stated hypothesis, methodology, a 10-wk timeline, discussion of foreseeable differences and data analysis plan, and references; see Supplemental Material 1) on the second week of the program via our intranet. Sharing research plans early in the program paces student progress and allows students to learn about one another's work. Students receive additional preparation and guidance at weekly “Science Careers” meetings, which address career options, mentorship and professional skills, graduate school funding options, and how to prepare and present a scientific poster. To encourage audience participation and keep contents current, whenever feasible, we use a panel format (e.g., separate graduate student panel and faculty recruiter panels present a layered, multiple-perspective view of the graduate school application process; nonacademic scientists discuss their training, career paths, and current employment trends in their areas of expertise).

The program's ethics training component involves approximately 15 h of contact time, and begins with a day-long role-playing activity that introduces topics such as authorship, collaboration, data collection and ownership, and research integrity; these topics are discussed more fully in weekly small group meetings. These meetings are student-led, with a directive that the presentation should include background on the topic and include audience participation (e.g., via the use of a case study). We encourage students to query their research team as they research their topic (e.g., ask each laboratory member questions such as “how is authorship determined in this laboratory?”); mentors report these inquiries often trigger discussions that might otherwise not happen and which benefit all lab members. Fellows also have the option of attending weekly “Lunch with a Scientist” meetings (informal networking opportunities with faculty), participating in library skills workshops, attending a skills session for the Graduate Record Examinations, and meeting with MD/PhD program students. The final week concludes with a Research Poster Symposium that is judged by advanced graduate students and open to the Emory community. Both participants and mentors have access to judging criteria early in the program. Participants receive the judges' feedback by mail and the best posters receive cash prizes that are announced at our closing banquet and on the program website.

Eligibility and Selection Criteria

Applicants must be enrolled as full-time students in either a 4-yr college (working toward a bachelors degree) or a community/preparatory college (working toward an associates degree) and cannot have graduated before program participation. Applicants from Emory must identify a research mentor before applying and submit a mentor-approved proposal as part of the application as well as a letter of recommendation from the supporting mentor (a requirement that dramatically improved the quality of submitted proposals when we implemented it in the late 1990s). We typically review >300 external applications, and 60–70 applications from Emory students. Approximately half of applicants accepted for SURE participation are from Emory, and the remaining half typically represent 20–25 schools. Following the priorities of the U.S. government and HHMI to increase racial and gender diversity in the sciences, ECCSE also endeavors to accept a sizable proportion of women, underrepresented minorities, and small liberal arts college students into its annual participant pool.

Profile of SURE Participants, 1990–2004

During its first 15 yr, 822 individuals participated in SURE, with cohort size increasing from 10 students in 1990, to between 20 and 30 students for 1991–1994, and to 50–70 students from 1995 to 2004 (overall mean cohort size, 54.8 students). Most participants are rising seniors (70.4%) or rising juniors (23.7%). Over the years, approximately half (50.4%) of SURE participants have been from Emory and half from other schools. Of non-Emory students, more than two-thirds (68.9%) came from schools outside of Georgia, and more than one-quarter (28.5%) were from small liberal arts colleges. Overall, slightly more than one-quarter of participants (26.9%) were minority students and 15.9% came from historically minority colleges. More women participated in SURE than men (59.0 vs. 41.0%) and mean age overall was 21.3 yr old (range, 17.7–51.0, SD 2.6).

Emory students were more likely to be men (49.1 vs. 32.4%; p < .001) and less likely to be of a minority race/ethnicity (13.0 vs. 41.7%; p < .001). Comparison by cohort cluster revealed that the proportion of minority SURE participants increased significantly during the 1990–1994/1995–1999/2000–2004 clusters (20.0, 26.1, and 30.7%, respectively; p = 0.050). More than 90% of SURE participants pursue science-intensive majors at their home institutions (with more than three-quarters in the life and natural sciences, and with biology the most common major at 40.7%). One-fifth of participants (19.8%) pursue double majors, and the degree most typically completed by SURE participants is a bachelor of science (79.2%). The overall self-reported undergraduate grade point average (GPA) of SURE graduates is 3.63.

Within a subsample of 2002–2004 SURE participants who completed an intake questionnaire (n = 95), most (96.8%) had laboratory-related course work backgrounds. However, less than half (47.4%) had any prior experience in a mentor-supervised UR setting, and two-fifths (40.0%) have no UR experience whatsoever. Finally, two-thirds of the 2002–2004 cohort (67.4%) of SURE participants have no background in research ethics training.

METHODS

This evaluation draws upon four sources of data: 1) a study of follow-up data for 1999–2004 participants gauging interest, preparedness, and pursuit (as related to science careers outcomes); 2) transcript analysis of Emory SURE fellows relative to peers who did not participate in SURE (1999–2008) participants; 3) postprogram survey data (SURE 2007–2009) provided by the SURE II-III team (D. Lopatto, Grinnell College, Grinnell, IA); and 4) participant satisfaction data from exit surveys completed by SURE participants (2002–2009).

Study 1. Online Follow-Up Survey

This study gauges associations between SURE participation and three sets of outcome variables, conceptualized as “interest,” “preparedness,” and “pursuit.” Interest and preparedness variables relate to short-term outcomes, namely, the extent to which SURE stimulates interest in the sciences and provides skills and experience germane to graduate study or a career in the sciences, respectively. Pursuit variables concern SURE's impact on postundergraduate academic training and professional career trajectories.

Self-report data come from an online follow-up survey (hereafter referred to as OFS) conducted during spring 2005. (See Supplemental Material 2 for survey questions.) To recruit respondents, all SURE graduates for whom a viable email or street address was available were contacted and invited to complete the online questionnaire (following an informed consent protocol approved by Emory University's Institutional Review Board). Of 743 individuals contacted, 250 completed the survey (with a response rate of 33.6%). Survey completion rates were marginally higher among racial/ethnic minorities (32.6 vs. 25.0%; p = 0.040) and noticeably higher among more recent SURE cohorts (with more than half of respondents coming from the 2000–2004 cohorts). For other variables considered (i.e., gender and undergraduate home school), the OFS sample was comparable with the overall SURE participant base.

The OFS addressed each of the project's three key themes: interest, preparedness, and pursuit, each described briefly below.

Interest. The OFS addresses interest as a subjective variable, asking respondents to recall their level of interest (“not interested at all,” “slightly interested,” “interested,” and “very interested”) in a variety of science-related graduate and professional careers at entry and completion of SURE (baseline/exit). Table 2 lists the 12 career options individually rated by each respondent. (See Supplemental Material 2, question 17 for exact wording.) We compared baseline and exit proportions of expressed interest for each of the options mentioned above, using a population proportion test to gauge statistical significance (the population proportion test is a normal approximation of a binomial distribution). Two types of interest are conceptualized: First, overall interest is calculated by comparing baseline and exit proportions of respondents who selected “interested” or “very interested” for the given option. Second, high-level interest is calculated by limiting the comparison to “very interested.” To test for significant change in (recollected) interest level at baseline and exit, we used a Mantel–Haenszel chi-square test, which examines the significance of linear relationship between two ordinal variables (baseline and exit proportions of response categories). This test is nonparametric, that is, it does not assume normal distribution of the data, only of standard deviations. Using a one-tailed test, the criterion for statistical significance is a p value of <0.050.

Preparedness. Preparedness is assessed using a similar approach. Specifically, respondents were asked to recall how prepared they felt (“no ability,” “weak ability,” “basic ability,” and “strong ability”) at the beginning and conclusion of SURE with respect to a range of research design, methodology, and presentation skills, as well as familiarity with research ethics. Skills considered fall into three categories: research design skills, data presentation and publication skills, and graduate study skills. Baseline and exit preparedness comparisons were made using the same statistical measures and significance criteria as with interest (assessing proportions of overall preparedness and high-level preparedness for each skill).

Pursuit. Finally, the OFS assessed the pursuit of graduate training and professional careers in the sciences after completing college. In the absence of a non-SURE comparison group, outcomes identified among SURE graduates cannot be definitively attributed to program exposure. These outcomes are, however, contextualized with subjective assessment of the level of importance attributed to their SURE experiences for subsequent academic/professional development and career paths.

Study 2. Course Work Transcript Analysis

In contrast to the self-reports in study 1, which are subjective in nature (asking participants how interested and prepared they felt), study 2 provides a more objective measurement of interest demonstrated through course selection and academic performance. Specifically, we use academic records (transcripts) for Emory students to gauge associations between SURE participation and three outcome variables related to interest in science: 1) number of science courses taken, 2) number of advanced-level science courses taken (i.e., 300- or 400-level courses, or graduate-level courses), and 3) academic performance (GPA) in science course work. (“Science course work” refers to any courses in the natural and life sciences, as well as empirically oriented courses in psychology and anthropology.) Because the majority of SURE students participate in the program during the summer between junior and senior years, we would expect to see program impact during the following academic year. Therefore, these three variables are calculated for the senior year only. Our principal research hypothesis is that SURE participants will have higher levels of interest as measured in each of the three response variables—that is, they will have taken more science classes, have taken more advanced-level science classes, and have done better in science classes during their senior year—than students who did not participate in SURE. We emphasize that due to data availability, this analysis applies only to Emory students who participated in SURE.

Data Sources. Transcript data (courses taken, grades), as well as gender and ethnic minority status,¹ were provided by the Registrar's Office at Emory University. These data were available for a total of 21,364 students, including 475 SURE participants, who entered the Emory College in fall 1987 or later and who graduated in or before spring 2008. We limited our analysis to students who graduated within 4 years, because most students participate in the SURE program during the summer after the third year. Data on high school GPA and SAT math and verbal scores were obtained from the Emory Admissions Office. Admissions data were only available from the year 2000 onward, for a total of 5073 students, including 114 SURE participants.

For students who did not take any science classes during their senior or freshman year, science GPA for that year was coded as zero, rather than as missing. We believe the resulting adjusted science GPA (aGPA) captures course work choice (a real-time science interest indicator) and allows us to retain these students in our sample.

Analysis. To compare differences between SURE participants and nonparticipants with respect to number of science courses taken, number of advanced-level science courses taken, and science course work GPA, we used two different statistical measures: bivariate two-sample t tests and ordinary least square (OLS) multivariate regression. Each of these tests produces an odds ratio and a measure of association (p value). The OLS regression model accounts for possible confounding factors.² Comparing our measures of interest in science (science GPA, number of science classes, and advanced-level science classes taken during senior year), we would expect that students who chose to enroll in SURE would score higher than students who did not enroll in the program, simply because those students who did enroll were more interested in science to begin with. (That is why they decided to participate in the program in the first place.) Thus, any association we detect between participation in SURE and higher scores on our measures of interest in science might simply be due to students' prior interest in science, rather than to their participation in SURE (selection bias). One way to avoid selection bias is in the program design stage by assigning subjects randomly to the intervention group. This clearly was not an option in the case of SURE; students chose to participate in the program, or, in other words, self-selected.

Another way to deal with the selection bias is to include control variables in a multivariate regression model in the analysis stage. In particular, we need to control for (or, in other ways, “take out of the equation”) students' prior interest in science, which would make them enroll in the program and have better senior-year science GPA and take more science classes their senior year. Therefore, we included the following three variables in the multiple regression model variables that we believe capture well the extent to which a student was interested in science before enrolling in SURE: number of science classes taken freshman year, freshman-year science GPA, and math SAT. (We restrict the first two indicators to the freshman year, because beyond this point, some students will have participated in SURE.) Additional controls include the student's gender, ethnic minority status, and high school GPA.

Thus, simply put, a multiple regression model compares students who had high existing interest in science (measured as their SAT math score and freshman science GPA) who participated in SURE and students with similar prior interest (similar math SAT score and freshman science GPA) who did not participate in the program. By including control variables into a multiple regression model, we are able to isolate the association between SURE and interest in science and ensure that this association is not driven by students' prior interest in science (thus limiting the possible effects of selection bias).³

Because data on high school GPA and SAT scores were available only for a subsample of the students, we estimated two sets of regression models: The first set of models included all 21,364 records from 1987 through 2008 and did not include Admissions' Office data. The other set of models did include admission data but only a subsample of records (n = 5065) for which the data were available was included.

Study 3. SURE II-III Survey (External)

The SURE survey is available to all summer research programs, and its scope is outlined in Lopatto (2007). The survey focuses on learning gains identified in Lopatto's prior research, which included queries to summer program directors on areas of training explicitly or implicitly pursued by their programs. Program directors receive a report that compares their participants' answers to those of students attending programs hosted at institutions with similar profiles (broadly defined, colleges vs. universities). Emory SURE students are invited to complete the survey on the last week of the program, and receive at least one additional email reminder during the fall semester. We have participated in this survey since 2003. The response rate for our program participants has ranged from 40 to 67%, with response rate decreasing as the program has grown larger (for 2003–2009, cohort size ranged from 51 to 84 participants).

Study 4. Annual Evaluation (Internal)

Participants are invited to complete an internal evaluation tied to student identification. This survey asks students to rate their satisfaction with various administrative program aspects (organization, adequate communication) and services (accommodations, dining facilities, social activities, parking, transportation options); perceived usefulness of various program activities and presentations; satisfaction with the assigned mentor and laboratory; asks respondents to give the program an overall grade; and asks respondents for ways in which the program may be improved. We also collect information about whether students have prior research experience and prior formal training in research ethics.

RESULTS

Study 1. Online Follow-Up Survey

Interest (Table 2).

For overall interest, the highest baseline levels were reported for “scientific research” (72.4%), “an academic career” (51.6%), and “doing a Ph.D. in a scientific field” (48.4%), with only negligible interest in careers in dentistry (0.8%) and allied health (3.2%). For overall interest, baseline-to-exit changes are statistically significant (at p < .050) for half of the 12 options considered, namely, “doing a Ph.D. in a scientific field” (p = .001), “a job in science education/college” (p < .001), “a job in science writing/journalism” (p < .001), “an academic career” (p < .001), “a public health career” (p < .001), and “an allied health career” (p = .001). Among those options with significant change, the highest degree of change were for “an academic career” (14.0% increase), “a public health career” (11.2% increase), and “a job in science education/college” (10.0% increase). For high-level interest, baseline-to-exit changes were highly significant for all options (at p < .001) except “going to medical school,” “a job in science education/K–12,” “a career in dentistry,” and “an allied health career.” Those with the highest degree of significant change were “scientific research (generally)” (16.8% increase), “an academic career” (14.8% increase), and “doing a Ph.D. in a scientific field” (14.4% increase).

Preparedness (Table 3).

At baseline, more than half of respondents indicated “basic” or “strong” overall ability in the following areas: “conceptualizing research objectives/hypotheses” (60.4%), “collecting data” (56.8%), and “understanding published research” (53.6%). The weakest skills at baseline were “writing grant applications” (13.2%), “using a statistical software package” (23.6%), and “identifying graduate funding” (26.4%). For overall ability (that is, baseline and exit proportions of respondents selecting “basic ability” or “strong ability”), substantive and highly significant increases were reported for all skills addressed (no p values >0.001). The three most notable increases were for the ability to prepare a poster presentation (45.2% increase), to discuss research at a graduate school interview (34.4% increase), and to select an appropriate data analysis strategy (34.0% increase). The smallest increase (at 16.4%) was for the ability to write a grant application. For high-level ability (that is, baseline and exit proportions of respondents selecting “strong ability”), increases were reported for all skills with statistical significance at p < .001. The largest increases were observed for the ability to prepare a poster presentation (45.2% increase), to apply principles of research ethics (38.4% increase), and to collect data (36.4%). The lowest increases (although all significant) were for the ability to write a grant application (4.8% increase), to use a statistical software package (11.2% increase), and to identify graduate funding (13.6%).

Postundergraduate degree pursuit and completion (Table 4).

We first report on the proportion of SURE alumni who, subsequent to completion of their undergraduate degree, have pursued graduate-level degree training. Nearly one-third (31.6%) completed a graduate degree, 86.5% of which were in a scientific field. Master's' degrees were completed by 15.4% of SURE graduates and 17.6% completed a doctoral degree. The most common graduate degrees completed were the M.D., M.A., and Ph.D., accounting for 30.1, 23.7, and 16.1% of all graduate degrees completed, respectively; graduate degrees were most commonly in a health-related field (41.9%) or in a life or natural science (26.9%). Of the 141 SURE graduates who had not yet completed a graduate degree at the time of the survey, 21 (14.9%) reported being currently enrolled in a graduate program. Accordingly, we estimate that as many as 40% (79 + 21/250) of SURE alumni pursue graduate training of some form. Of the 120 individuals who had neither completed a graduate degree nor were enrolled in a graduate program, eight (6.7%) said they intended to pursue another degree in the near future.

Employment (Table 5).

At the time of the survey, approximately one-half of the entire OFS sample was employed on a full- or part-time basis (44.4 and 6.0%, respectively). Among employed respondents, the two most common work categories were research and development (44.6%) and professional services (35.6%); nearly a half of respondents (48.9%) were working in an academic setting (i.e., a college or university) and more than one-quarter (28.9%) in industry. Most (83.7%) were working in a science field; slightly less than one-half (44.6%) were in a research field. Three-quarters (77.7%) of employed respondents reported being either “satisfied” or “very satisfied” with their current positions. Among SURE graduates who completed their undergraduate degree in 2004 or earlier (an exclusion criterion imposed to allow at least 1 yr for the given outcome to occur), the most common pursuit during the first year after graduation was to attend graduate school (61.1%). When inquired on level of importance attributed to SURE for subsequent academic development and career paths, nearly three-fifths (56.8%) considered SURE to be “very important” in this respect. More than nine of 10 respondents felt SURE to be at least “important” (data not shown).

Study 2. Course Work Transcript Analysis

Profile of Study Sample. Demographic and baseline characteristics of the study sample (n = 21,364) as well as data on science course work and performance are presented in Table 6. The overall sample (Emory graduates from the classes 1990–2004) includes 475 SURE participants and consists of slightly higher proportions of women than men (54.5 vs. 44.5%). In terms of ethnicity, the sample was primarily (89.0%) nonminority status. The average overall number of senior-year science courses was 0.78, and 0.49 for advanced level courses. GPA for science course work was 1.90 and 1.19 for freshman and senior years, respectively. (This figure, as noted above, is artificially driven down due to our having coded the GPA of students who took no science courses as 0.) For the subsample for whom admissions data were available (n = 5065), the average high school GPA was 3.52; average math and verbal SAT scores were 645 and 623, respectively. Overall undergraduate GPA was 3.21.

Associations between SURE Participation and Interest in Science (Table 7).

Bivariate analysis shows that participating in SURE was associated with higher interest in science: Students who participated in the program took on average 1.16 more science classes and 1.11 more high-level science classes during their senior year than students who did not participate. Their senior-year adjusted science GPA was also higher by 1.65 points. Multivariate analysis, which controlled for students' prior interest in science, confirmed those results. The model which controlled for freshman adjusted science GPA, and number of science classes taken during freshman year, and the graduation GPA showed that SURE participants took on average 0.97 more science classes and 0.92 more high-level science classes than their peers who did not participate in the program, and their senior-year adjusted science GPA was higher by 1.38 points. The model which also included SAT math and verbal scores as well as high school GPA showed that SURE participants took on average 0.82 more science classes and 0.83 high-level science classes than their peers who did not participate in the program and their senior-year adjusted science GPA was higher by 1.15 points. All results were statistically significant at the 99% confidence level.

Results of our analysis of control variables, principally prior interest in science, are presented in Table 8. This analysis shows that senior-year interest in science was associated with the proxy indicators for prior interest in science: number of freshman-year science classes, freshman-year science GPA, and math SAT. (Neither gender nor racial/ethnic minority status were significant in our regression models.) Analysis of the subsample for which admissions data were available (bottom portion of Table 8) revealed that higher SAT verbal scores were associated with a lower number of science classes and advanced- level science classes taken but not with lower senior-year adjusted science GPA.

Studies 3 and 4. SURE II-III Survey (External) and Annual Evaluation (Internal)

Figure 1 reports aggregated responses for our 2007–2009 participants as provided by the Lopatto SURE survey team (Grinnell College). We have noted gains in recent years relative to previous years.

Response to our internal survey has ranged from 50 to 76.7% for years 2002–2009. For this period, >87% of survey respondents indicated the mentorship they received was superb and exceeded their expectations (52.5%) or met their expectations (35.4%). Eighty-four percent state the program met or exceeded their expectations, and 85% give the program an overall grade of “B+” or above. More than 84% of respondents indicate the program's clear communication of timelines is excellent or very useful and regard program organization similarly. Our most popular activity has been our “Lunches with Scientists,” with >89% of respondents rating these activities as excellent or quite useful. The primary criticism of this activity is that demand is higher than supply, and some students are not able to attend as many lunches as they would like. Although less attended, the M.D./Ph.D. program information sessions were also highly rated (85% judged it excellent or very useful). Information presented in panel format was typically rated as being excellent or very useful by ∼60% of respondents, and so was our presentation on funding graduate careers. Our Mentorship and Professional skills session was judged excellent or very useful by 43% of respondents and as somewhat useful by 42% of respondents.

More than 60% of respondents judged our ethics training to be very good or excellent, with 13% suggesting there is room for improvement. Approximately 67% of our 2002–2004 participants did not have ethics training before participating in SURE; for 2005–2009 participants, this percentage rises to 74%, probably reflecting the increase in younger first- and second-year participants.

DISCUSSION

SURE's Reach

During its first 15 yr, the SURE Program at Emory provided UR experience to >800 students. Among the roughly half of SURE participants not from Emory, more than two-thirds were from schools outside of Georgia and 28.5% were from small liberal arts colleges, underscoring that SURE's reach is national and not limited to students from large research universities. SURE has also been successful in enrolling growing numbers of minority students, with increases over the years from approximately one-fifth to one-third of each summer cohort. Although the majority of SURE participants have been science majors, a steady presence of nonmajors (∼1/10) suggests that SURE has also contributed to HHMI's goal to increase science literacy among the general undergraduate population. The average self-reported undergraduate GPA of SURE graduates (3.63, corresponding roughly to “B+”) also suggests that SURE is successfully involving not only the highest academic achievers but also students with room for improvement in grade performance. Importantly, we note that the majority of participants come to SURE with no prior experience in a mentor-supervised research setting, a gap which SURE fills.

Study 1: Online Follow-Up Survey

Methodological Reflections on Alumni Surveys. Existing evaluation studies have addressed the benefits and shortcomings of alumni surveys (Hakim, 1998; Bauer and Bennett, 2003). On the one hand, such surveys may be constrained by recall limitations (i.e., memory issues) or a greater interest in responding among those participants who had a positive experience. On the other hand, alumni may have less temptation to speak well of their UR program post facto, because they are more distant from ongoing funding concerns on the part of program organizers. Alumni may also possess a broader critical perspective or insight into the positive or negative effects of their UR experience that could only come from the passage of time since completion of that experience. Finally, our confidence in the alumni survey approach is increased in light of previous studies that have shown correlation between a person's assessments of their achievements gauged during their undergraduate years and as alumni, and between their own assessments and those of faculty mentors (Pace, 1979; Seymour et al., 2004).

SURE's Influence on Interest in the Sciences. Even in the absence of a comparison group, the range and degree of reported increases in both general and high-level interest suggests that SURE is effectively stimulating interest in graduate study and professional careers in the sciences. Especially noteworthy are increases in general and high-level interest in an academic career and high-level interest in scientific research generally. (We attribute the lack of significant change in general interest in scientific research to the fact that baseline interest level, at 72.4%, was already quite high, leaving less room for improvement.) We are encouraged at the lack of significant change in general interest levels for medical school, a career in industry, and dentistry, as these career trajectories fall outside the purview of SURE's programmatic foci as well as the aims of HHMI and NSF. It is also worth underscoring SURE's effect on general interest in college-level science education and, to a lesser degree, K–12 science education, suggesting the program's contribution to training the next generation of students in the sciences.

Calculations of change in high-level interest (meaning, differences between baseline and exit proportions of students who were “very interested” in the given option) are useful to gauge SURE's capacity to help students identify those professional trajectories that hold particularly intense appeal. Beyond scientific research and an academic career, we note the relatively high degree of significant change in high-level interest in carrying out a Ph.D. in a scientific field (a 150% increase from 26.4 to 40.8%). It is extremely difficult to ascertain whether a student will pursue a medical degree or a Ph.D. based on an application. A savvy student can target his or her essay to mask a pre-med focus. Applicant interviews are not feasible in most cases, and even when possible, a student (or letter writer) may feel it is not unethical to secure a research fellowship that will impress a medical school admissions committee. It can also be argued that basic research skills are an asset to any physician and as such, our program doesn't fail science at large. Recent publications (McGee and Keller, 2007; Villarejo et al., 2008) have begun to tease out possible pre-M.D. versus pre-Ph.D. indicators, and we will consider adding some kind of rubric addressing these metrics to the recommendation letter that is currently part of the application.

SURE's Influence on Preparedness in the Sciences. Significant increases in all response options, for perceived overall and high-level ability, suggest that SURE is effectively improving participants' abilities in key scientific skill areas and hence their preparedness for graduate study and professional careers in the sciences. By the same token, the degree of increase varied substantially, suggesting greater effectiveness in some training areas over others. We note, for example, that the skills for which respondents reported the lowest general ability at baseline—“writing a grant application” (13.2%), using a statistical software application (23.6%), and “identifying graduate funding” (26.4%)—were also those for which the smallest positive change was reported from baseline to exit (29.6, 26.8, and 23.2%, respectively). This finding suggests important areas for programmatic improvement. It is also important to remember that some skills sets (grant application skills, for example) require years to fully develop and, therefore, that SURE may impel its students in a positive pedagogic direction even in the absence of a sense that their “ability” has improved.

In terms of overall ability, the strongest increase was reported for the preparation of a poster presentation (a 228% increase from 35.2 to 80.4%). We are also encouraged by appreciable increases for the selection of an appropriate data analysis strategy (+34.0), discussion of research at a graduate school interview (+34.4), and applying research ethics principles (+30.4%). The latter increase (research ethics) strikes us as particularly significant as this area is all too often left out of undergraduate science education.

SURE's Influence on the Pursuit of Graduate Study and Careers in the Sciences. Notable in this set of results is the finding that nearly three of 10 (28.0%) of SURE graduates go on to complete a graduate degree in a scientific field. This rate substantially exceeds rates reported in a recent alumni survey of former science majors at Emory (unpublished data). Whereas Ph.D.s and M.D.s were completed by 5.4 and 11.8% of alumni surveyed, respectively, SURE graduates completed these respective degrees at the rates of 16.1 and 30.0%. Moreover, the proportion of our sample that completed M.P.H. degrees was nearly double that of the science alumni survey (6.5 vs. 3.0%). That the M.D. was the most frequently pursued degree raises questions about what sorts of careers SURE graduates ultimately aim to pursue (i.e., research vs. clinical), and our finding that research and development was the most commonly reported employment category (at 44.6%) assuages concern that SURE might be a research-oriented detour leading to a clinical medical career.

Among respondents who were employed at the time of the survey, we found academic settings to be most common (at 44.6%), with industry (28.9%) and federal government settings (11.1%) also prevalent. Results also indicate high levels of employment in science fields (83.7%) and job satisfaction (with more than three-quarters reporting “satisfied” or “very satisfied”). Finally, the majority (>90%) consider SURE to have been important or very important to their academic development and career paths.

Study 2: Course Work Transcripts

The major finding from the regression analyses (limited to Emory students) is the statistically significant association between SURE participation and three outcome measures of interest in the sciences, even after controlling for prior interest in science, gender, and minority status. Specifically, we found that during their senior year former SURE participants take more science classes (both introductory and advanced level) and receive higher grades in these courses than their nonparticipating peers.

It is worth underscoring the ability of a regression design to address a common shortcoming of comparative outcome studies between program participants and nonparticipants, namely the inability to address selection bias. The particular bias we were concerned with was the likelihood that SURE participants are an undergraduate subset with a high pre-existing interest in science, which we would therefore expect to take more science courses and to do better in those courses irrespective of SURE participation. The data indeed showed that students who enrolled in SURE had higher freshman-year adjusted science GPA and SAT math scores and took more science classes their freshman year than students who did not participate. We addressed this issue by including indicators of students' interest in science before their enrollment in SURE (freshman adjusted science GPA, number of science courses taken during freshman year, and SAT math scores) in the regression models. Our analysis showed that SURE participants had higher interest in science after completing the program compared with their peers with similar pre-existing interest in science, but who did not participate in SURE. Thus, the breakthrough in this study was our ability to control for these variables and still detect significant associations with the three given outcomes. This methodological innovation lends powerful new evidence that participation in SURE is stimulating interest and enhancing academic performance in senior-year science course work.

Our online survey and transcript analyses also point toward important areas for future research. Absent in our own research, and seldom addressed in UR scholarship generally, is systematic attention to the role of faculty mentors (Hathaway et al., 2002; Zydney et al., 2002; Gafney, 2001; Frantz et al., 2006). All too often, mentors are presumed to be a homogeneous group, each of whom devotes equal time and energy to their student researcher. Moreover, some faculty develop collaborative projects with their student researchers, whereas others leave the student to undertake their project more or less independently. As these variations have obvious implications for the students' qualitative experience of UR programs, they should be integrated into formal evaluation studies. A related area concerns the distinction between UR programs at public versus private institutions (see Merkel, 2003) and at large research universities versus small liberal arts colleges. Also called for in future evaluation studies is greater attention to those attributes that distinguish the summer intensive model from those UR programs taking place during the school year. What, in other words, might be the value added by summer research as a particular form of UR experience? Finally, there is a paucity of evaluation research on ethics training components of UR programs (but see Sweeting 1999; Shachter, 2003).

Studies 3 and 4. SURE II-III Survey (External) and Annual Evaluation (Internal)

Combined with our internal evaluation, SURE II-III (Lopatto) survey data offer a powerful, annual opportunity for reflection on our program offerings and how we may adapt activities to ensure explicit discussion of all topics we consider relevant to a UR experience for future scientists. This explicit articulation of objectives is important for all students (awareness of developing expertise inspires confidence) but possibly more so for students who study at institutions that are not as research-intensive as ours and who may not have a family history of participation in science careers. We also believe that this increased awareness helps students better reflect on their experience, and in turn allows them to offer more robust program feedback. It is also important to review activity feedback with an eye to student development and experience: the benefit of an activity (e.g., discussion on the importance of mentorship or how authorship is determined) may not be immediately apparent to a student in the short-term but may become more so upon experiencing new learning and work environments.

CONCLUSIONS

Despite limitations in the available data, this study strongly indicates that participation in the SURE program enhances research skills and interest in academic research careers. More importantly, the data indicate that participation in our summer research program significantly increases the number of science courses taken and GPA, postparticipation, in particular among women and minorities. The results suggest that planning SURE-like experiences for first- and second-year students might improve success in science and interest in academic research careers. We plan to conduct a study of first- and second-year SURE participants to test this hypothesis since in recent years we have added such students to our program. First- and second-year students may require different kinds of preparation, perhaps a research apprenticeship during the academic year or a research course. Future studies should also include attention to the quality of mentoring and to preparing current and future faculty to mentor undergraduates in research. We also hope to study different outcomes for students who participate in research for credit versus those who complete an honors thesis.

FOOTNOTES

REFERENCES