| HOME | HELP | FEEDBACK | ARCHIVE | SEARCH | TABLE OF CONTENTS |
|
|
|
||||||
|
||||||||
ARTICLES |
Department of Psychology, Grinnell College, Grinnell, IA 50112
Submitted July 6, 2004; Accepted September 3, 2004
 |
ABSTRACT |
|---|
 TOP ABSTRACT METHODRESULTSDISCUSSIONREFERENCES |
|---|
Key Words: undergraduate • research • learning • career • science
Attempts to determine an empirically established set of benefits generated by undergraduate research experiences in the sciences are fairly recent (Lopatto, 2003a; Seymour et al., 2004). Lopatto surveyed science undergraduates at four liberal arts colleges over a period of three summers. The quantitative results of these surveys fit well with qualitative data drawn from student interviews at the same four institutions by Seymour et al. (2004). Having grounded the benefits of undergraduate research in both quantitative and qualitative data in these pilot studies, it remains for researchers to establish that the findings apply to a broader range of institutions. The opportunity for a more extended study was presented by the Howard Hughes Medical Institute (HHMI), which funds grant activity for undergraduate science education at a variety of institutions.
The undergraduate research experience is widely touted as an effective educational tool for enhancing the undergraduate experience (Mogk, 1993; Tomovic, 1994) with multiple benefits (Lopatto, 2003a), the most instrumental of which is an increased interest in a career in the science, technology, engineering, and mathematics workforce (Fitzsimmons et al., 1990; Zydney et al., 2002). Undergraduate research experiences are associated with increased persistence in pursuit of an undergraduate degree (Nagda et al., 1998); increased levels of pursuit of graduate education (Hathaway et al., 2002; Kremer and Bringle, 1990); and alumni retrospective reports of higher gains than comparison groups in skills such as carrying out research, acquiring information, and speaking effectively (Bauer and Bennett, 2003). Several studies have supported the hypothesis that undergraduate research helps promote career pathways for members of underrepresented groups by increasing the retention rate of minority undergraduates (Nagda, et al., 1998) and increasing the rate of graduate education in minority students (Hathaway et al., 2002).
This study was motivated by three research questions, primarily concerning but not limited to HHMI-funded undergraduate research experiences in the sciences. These questions are: 1) Is the educational experience of undergraduates being enhanced? 2) Are undergraduate research programs attracting and supporting talented students interested in a career involving scientific research? 3) Are undergraduate research programs retaining minority students in the "pathway" to a scientific career?
In terms of this survey, question 1 was answered by investigating both the general response to the undergraduate research experience and the specific gains reported on 20 potential benefits. Question 2 was examined by asking students their plans for postgraduate education. Are students in undergraduate research programs declaring that they intend to continue their science education or to seek a science-related career? Furthermore, can we identify differences in the experiences of those students who intend to continue in science compared with those students who assert that they have lost interest in science careers? Question 3 was examined by looking for differences among ethnic groups in proportion to each group that intends to continue in science. Included in this examination were differences between women and men because women are an underrepresented group in some sciences.
|
METHOD |
|---|
|
TOPABSTRACTMETHODRESULTSDISCUSSIONREFERENCES |
|---|
|
As is true of every tabular or statistical presentation in this report,
there are missing cases. Students might have failed to indicate their
institution, declined to specify personal characteristics, or left an
evaluative question unanswered because it did not apply to them.
Table 1 shows the categories of
institutional type, sex, and ethnicity. Overall, including cases missing from
Table 1, 258 respondents
attended a college, 59 respondents attended a master's university, and 787
respondents attended a research university. Thirty-one respondents did not
specify their institution. Analysis of the data in
Table 1 indicates that the
distribution of men and women is uniform across institutional types. The
distribution of men and women is also uniform across ethnic categories. The
distribution of ethnic categories across institutional types is nearly
uniform. A statistical analysis of ethnicity by institutional type yields a
significant chi square (
2 = 34.4; df = 12; p, .05),
with higher than expected frequencies of Asian American and Hispanic students
at Research Universities.
Respondents were asked to cite the source of the funding that supported their research. Fifty-five percent of the respondents reported being funded by HHMI programs. Local university or college funding accounted for 12% of the respondent support. Other sources of funding were each less than 10% of the total. Non-HHMI-funded students were eligible to take part in the survey because institutions with HHMI grant programs often fund other students for similar activities in the same time period, so it was deemed sensible to invite these students to participate.
Approximately 59% of the respondents are women. Table 2 shows the research fields of the respondents crossed with the sex of the respondent. As might be expected from national trends, women outnumber men in biology, chemistry, and biochemistry, but men outnumber women in physics, mathematics, computer science, and engineering.
|
About 48% of the respondents reported their graduation year as 2003 or 2004. Juniors graduating in 2005 made up 32.6% of the total, whereas sophomores made up 16.3%. The remaining 2.7% reported their graduation year as 2007, implying that they were entering first-year students. Specific ethnic categories conformed to this pattern, with the exception of Hispanic respondents, who had a lower proportion of rising seniors (34.5%) and a higher proportion of rising juniors (40%). Only about one-third (35%) of the entire group reported no prior experience in undergraduate research. Older students tended to have more prior experience than younger students.
The Survey
The Survey of Undergraduate Research Experiences (SURE) consisted of 44
items, including demographic variables, learning gains, and evaluation of
aspects of summer programs. Items regarding learning gains were suggested by
previous survey research. Before the main survey was conducted, a brief
checklist survey was sent to HHMI program directors. A list of learning gains
concepts was presented, and program directors were asked to report whether
each was primarily taught explicitly (e.g., in a seminar or workshop) or
implicitly (e.g., a behavior modeled by the student's supervisor).
The SURE survey was located online on server at Washington University in St. Louis, MO.
Procedure
Notices of survey availability were sent to each program director.
Participating program directors (PDs) were asked to specify the number of
students from their school eligible to take the survey and the date on which
they would be asked to do so.
The target date was immediately after the end-of-program symposium or other "summing up" activity. Two weeks after that date, PDs were informed how many students had participated, giving the PDs the option to contact their students to remind them to participate in the survey. Students were provided with a name and password for access to the survey. Within the survey, students identified their school and provided demographic information, but anonymity was maintained. Student names were collected for a raffle that awarded gift certificates to the winners, but the names were separated from the survey material. Students answered items on the survey by either selecting from a pull-down menu or choosing a number on a rating scale. A "no answer" option was available. At the end of the survey, students were provided with a text box for written comments, which were directed to the PD for that institution by e-mail. After the site closed, all PDs received the aggregate results for their school.
|
RESULTS |
|---|
|
TOPABSTRACTMETHODRESULTSDISCUSSIONREFERENCES |
|---|
|
|
Reported Learning Gains and Overall Evaluation of the Undergraduate Research Experience
To evaluate the educational experience of undergraduate researchers, 20
evaluative questions on specific learning gains, drawn from previous research,
were presented to the respondents. Table
5 summarizes the results by listing the means for each set of
responses. These means are listed in descending order. The highest rated item
is "Understanding of the research process in your field" (X =
4.13), followed by "Readiness for more demanding research" (X =
4.03), "Understanding how scientists work on real problems" (X =
4.0), and "Learning laboratory techniques" (X = 4.0). The lowest
rated item is "Learning ethical conduct in your field" (X = 3.15),
followed by "Skill in science writing" (X = 3.32), "Skill in
how to give an effective oral presentation" (X = 3.42), and
"Clarification of a career path"
(
= 3.42). A measure of interitem consistency,
Cronbach's alpha, on these 20 items is .92, indicating a high degree of
consistency. Students who reported being funded by HHMI also reported higher
means on all 20 items compared with the overall survey means
(Table 5).
|
In addition to the 20 specific learning gains, respondents were asked to evaluate five general questions about their experience. Student attitudes toward their research supervisor, peers, and overall sense of summer research as a learning experience were decisively positive. The frequency distributions of responses to these questions are found in Table 6. The responses reveal a high degree of satisfaction with the undergraduate research experience. Eighty-seven percent of the respondents rated their experience as good or better than they expected. Seventy-eight percent of the respondents rated their supervisors as above average or outstanding, an impressive finding. The supervisors included faculty, postdoctoral students, and graduate students. Sixty-three percent of the students rated their experience with other students in a positive way; 19% of the respondents did not answer this question, presumably because they did not work with other undergraduates. Eighty-four percent of the respondents indicated they were likely or very likely to choose another research experience if they could, and a similar percentage gave a positive overall evaluation of their summer research as a learning experience. These five overall evaluative questions are correlated with each other (Table 7). For example, the overall sense of summer research, perhaps the most global of these questions, is significantly related to prior expectations (rs = .47), to supervisor ratings (rs = .39), to student ratings (rs = .20), and to choosing another research experience (rs = .36). Thus, the responses are generally consistent.
|
|
To investigate the question regarding the attraction of talented undergraduates to science, the data were analyzed for students who reported positive or negative experiences. One variable for such exploration concerns the influence of the research experience on plans for postgraduate education given that one of the goals of undergraduate research is to promote graduate education in science. As described earlier, the undergraduate research experience sustained or confirmed the student's plans for graduate education. Over 83% of the students continued to plan for postgraduate education in the sciences following their undergraduate research experience. Two groups of students did change their plans (see Table 4). Forty students initially had no plans for postgraduate education, but the research experience changed their minds. Fifty-one students initially had plans for postgraduate education, but their research experience convinced them otherwise. In what ways did their experiences differ? To find out, the 20 learning gains questions were analyzed. When the mean scores for these two subgroups are inspected, the students who now plan to further their education have the highest means of any group displayed in Table 5 on 18 of the 20 items. In contrast, the respondents who decided not to pursue further educations have the nominally lowest means of any group displayed in Table 5 on 15 of the 20 items. When the two groups are directly compared to each other via independent group t-tests, 13 significant differences emerge at the a = .005 level, all in the direction of higher learning gains by the group that is now planning graduate education.
The two groups that were influenced by summer research to change their plans for postgraduate education also differed on some of the overall evaluation questions. An inferential statistic, the Mann-Whitney U-test (evaluated at a = .01), was employed to analyze the data. Students who changed their plans in favor of graduate education had higher ratings about summer research meeting or exceeding their expectations, evaluated their supervisors more favorably, were more likely to choose another experience, and had a more positive overall sense of summer research than students who changed their plans to forego graduate education. The two groups did not differ in their evaluation of working with other students. The two groups did not differ in the type of institution at which they had their experience. Although the results can be interpreted in various ways, the SURE survey nevertheless appears to capture the different experiences of students who become attracted to, compared with those who turn away from, science careers.
Differences between Women and Men
Women constituted almost 60% of the sample of respondents. As noted in
Table 2, women are the majority
in the biological sciences, whereas men are the majority in the physical
sciences, math, computer science, and engineering. This distinction is
repeated when students were asked about their postgraduate education plans. A
higher percentage of women than of men plan to continue in biology (21.9% of
all women vs. 19.3% of all men) and medicine (24.7% of all women vs. 21.1% of
all men), whereas a lower percentage of women than of men plan to continue
with a Ph.D. in the physical sciences (9.5% of all women vs. 21.6% of all
men). Women and men did not differ overall in prior experience, in the
influence of the research experience on their future plans, in their
evaluation of their supervisor, or in whether they would choose another
research experience. Women reported higher gains than men on 14 of the 20
learning items. These results, however, are entangled with institution (there
are two women's colleges in the group), research area (biological vs. physical
science), and previous experience indicating that women have a positive
response bias when compared with men.
On overall evaluation questions, women reported higher ratings than men on
two items. When asked to compare their experience to their prior expectations
about the experience, more women than men rated the research experience as
"Much better than I expected" (35.5% of women vs. 26.2% of men;
2 =13.7; p, .05). Women also rated their experience with other
students more favorably than men did (45.1% of women vs. 37.6% of men rated
working with other students as "One of the best parts of the summer
research experience" (2 = 14.0; p, .05).
Differences among Ethnic Groups
To address the issue of retention of minority students in a pathway to a
scientific career, the reported experience of minority students was analyzed.
Nearly half of the respondents in the sample were not members of the Caucasian
majority. The better represented minority groups include Asian-American
(14.9%) and African-American (10.0%). Hispanic students made up 4.9% of the
total, whereas foreign national students constituted 6.1%. The memberships of
some ethnic groups are so small (e.g., 1 Native American) that statistical
comparisons are not meaningful.
Ethnic groups did not differ in their distribution of women and men, with women constituting the majority in every ethnic group. The research fields chosen by each group were dominated by the biological sciences (Table 8), as might be expected because the major source of student funding in this study was from the HHMI. Ethnic groups did not differ in prior experience. Asian-Americans showed relatively less interest in postgraduate education in biology-related Ph.D. programs (15%) and relatively more interest in medical school (35%) than comparison groups (see Table 9). There were no statistically significant differences among ethnic groups in the influence their summer research experience had on their postgraduate plans (Table 10), an indication that the undergraduate research experiences are retaining minority students in the pathway to a science career as well as they retain Caucasian students.
|
|
|
An analysis of the five general satisfaction questions revealed no differences among ethnic groups in their expectations of summer research being met, their evaluation of their supervisors, their evaluation of their student coworkers, their inclination to have another research experience, or their overall sense of research as a learning experience. Analysis of the 20 learning gains questions indicates that ethnic groups statistically differed on three items ("Learning ethical conduct", "Skill in oral presentations", and "Becoming part of a learning community"), however, with Hispanic students reporting scores that were higher than at least one other group.
Institutional Type
The means and standard deviations for each institutional type for the 20
learning gains items are presented in
Table 11. Although some
ratings of the institutions differ significantly on some items, the pattern of
the differences is difficult to determine. Analyzing differences in student
learning with reference to Carnegie Classification categories does not reveal
clear differences among institutions.
|
|
DISCUSSION |
|---|
|
TOPABSTRACTMETHODRESULTSDISCUSSIONREFERENCES |
|---|
The present findings replicate pilot work on assessing undergraduate research (Lopatto, 2003b; Seymour et al., 2004). Lopatto surveyed 384 science undergraduate researchers working on summer research programs at four liberal arts colleges. As in this study, students rated their gains on potential benefits on a scale of 1 (no gain) to 5 (very large gain). The results showed that "Learning laboratory techniques" and "Understanding the research process" were among the highest rated benefits. Personal development items, including "Readiness for more demanding research" and "Tolerance for obstacles," were also rated highly. The overall pattern of rated benefits was similar to that seen in this study. Seymour and her colleagues (2004) analyzed the results of 76 interviews with students participating in undergraduate research experiences at the same four sites examined by Lopatto (2003b). A qualitative analysis, analyzing the text of the transcripted interviews, revealed the same pattern of benefits reported by students as are seen in the results presented here.
This study does not include a control group and does not sort out all the influences, including precollege career intentions, general academic achievement, and family influence, that might account for the choices students make to pursue careers in science. A control group is selected with the goal of creating a comparison group that is like the treatment group in all characteristics except the treatment. This goal is elusive in the in situ study of undergraduate students. Undergraduate research programs are affected by both institutional selection procedures, which filter the best applicants for research positions, and self-selection by students motivated to explore science. An attempt to create a quasi-control group, such as a group composed of students who applied for research positions but were turned down, does not escape the criticism that the control subjects were not identical to the treatment subjects before the undergraduate research experience commenced. Differential academic credentials or interpersonal skills might have sorted the undergraduate researchers and unselected students into nonequivalent groups. An attempt to create a true control group by traditional means, such as random assignment, would be unlikely to succeed in the face of ethical concerns regarding withholding education from talented students and regarding traditional student autonomy in choosing courses of study.
Should a reasonable control group be provided for the study of the undergraduate research experience, a second obstacle renders comparisons between treatment and control groups difficult. Undergraduate research experiences contain features that are not exclusive; laboratory courses, "research-like" experiences and other pedagogies can share some of the features of undergraduate research. The presence of non-exclusive features means that treatment and control subjects are likely to differ in amount of treatment rather than to differ by the presence or absence of treatment. Furthermore, the undergraduate research experience is a molar treatment (Campbell, 1986), consisting of an array of treatment components, including the mentoring skill of the supervisor, the social interactions of the working group, the sophistication of the instrumentation related to the research experience, and so on. Therefore, this study cannot meaningfully compare the learning gains of students who experienced undergraduate research relative to those who did not, except for the small group that left the science career pathway as a result of their experience. The results do show, however, that most respondents report an enhanced educational experience through a variety of learning gains. The experience sustains the interest of many students already planning science careers and attracts some students who did not previously plan a science career. These outcomes are consistent across gender and ethnic group.
Further information is needed to support these findings. The current research includes a 9-month follow-up survey, currently in progress, that will contribute more information to the robustness of student plans and to the specific enhancement of later educational experiences following the summer research experience. Ideally, a further research effort should include tracking the cohort of student respondents to determine how many continued on pathways to science careers.
|
ACKNOWLEDGMENTS |
|---|
|
FOOTNOTES |
|---|
E-mail address: lopatto{at}grinnell.edu.
|
REFERENCES |
|---|
|
TOPABSTRACTMETHODRESULTSDISCUSSIONREFERENCES |
|---|
Campbell, D.T. (1986). Relabeling internal and external validity for applied social scientists. In: Advances in Quasi-Experimental Design and Analysis. New Directions for Program Evaluation, No. 31, ed. W.M.K. Trochim. San Francisco: Jossey-Bass.
Fitzsimmons, S.J., Carlson, K., Kerpelman, L.C., and Stoner, D. (1990). A Preliminary Evaluation of the Research Experiences for Undergraduates (REU) Program of the National Science Foundation. Washington, D.C.: National Science Foundation.
Hathaway, R.S., Nagda, B.A., and Gregerman, S.R. (2002). The relationship of undergraduate research participation to graduate and professional education pursuit: an empirical study. J. Coll. Stud. Dev. 43,614 -631.
Kremer, J.F., and Bringle, R.G. (1990). The effects of an intensive research experience on the careers of talented undergraduates.J. Res. Dev. Educ. 24,1 -5.
Lopatto, D. (2003a). The essential features of undergraduate research. Counc. Undergrad. Res. Q.24 , 139-142.
Lopatto, D. (2003b). What undergraduate research can tell us about research on learning. Invited address presented at the meeting of Project Kaleidoscope, University of Richmond, VA. Text posted online by Project Kaleidoscope. http://www.pkal.org/template2.cfm?c_id=1002 (accessed 15 June 2004).
Mogk, D.W. (1993). Undergraduate research experiences as preparation for graduate study in geology. J. Geol. Educ.41 , 126-128.
Nagda, B.A., Gregerman, S.R., Jonides, J., von Hippel, W., and Lerner, J.S. (1998). Undergraduate student-faculty partnerships affect student retention. Rev. Higher Educ.,22 , 55-72.
Seymour, E., Hunter, A.-B., Laursen, S., and DeAntoni, T. (2004). Establishing the benefits of research experiences for undergraduates: first findings from a three-year study. Sci. Educ., 88,493 -594.[CrossRef]
Tomovic, M.M. (1994). Undergraduate research—prerequisite for successful lifelong learning. ASEE Annu. Conf. Proc. 1,1469 -1470.
Zydney, A.L., Bennett, J.S., Shahid, A., and Bauer, K.W. (2002). Impact of undergraduate research experience in engineering. J. Eng. Educ. 91,151 -157.
| ||||||||||||||||||||||||||||||||||||||||||||||||||||||||||
| HOME | HELP | FEEDBACK | ARCHIVE | SEARCH | TABLE OF CONTENTS |
