Effect of Time on Perceived Gains from an Undergraduate Research Program

INTRODUCTION

The benefits of undergraduate research experiences (UREs) cannot be overstated. Research and evaluation studies continue to show that UREs are effective for enhancing student overall baccalaureate experience and for recruiting, retaining and graduating students, especially underrepresented minorities, in science, technology, engineering, and mathematics (STEM) majors (Hernandez et al., 2012). For example, Bauer and Bennett (2003) found that University of Delaware alumni who participated in URE programs reported higher satisfaction with their overall undergraduate education experience and were more likely to pursue graduate education than their counterparts who did not participate in URE programs. URE participants in a study by Korkmaz and colleagues (2011) reported higher levels of academic engagement and student–faculty interaction. A recent study by Slovacek and colleagues (2012) also reported that participants in the Minority Opportunities in Research programs at California State University, Los Angeles, had lower dropout and higher graduation rates and were more likely to pursue graduate or professional education than a matched sample of nonparticipants. Similarly, Nagda et al. (1998) reported that participation in the University of Michigan's Undergraduate Research Opportunity Program increased retention rates of minority students, in particular, low-achieving African-American students.

UREs have also gained popularity for their role in enhancing student research skills, research self-efficacy, and understanding of research processes (Lopatto, 2003, 2004). Hunter and colleagues (2007) reported that student participation in UREs resulted in enhanced research skills (i.e., abilities to perform techniques required for laboratory and research work), understanding of science literature, and ability to find and retrieve scientific literature. Hunter et al. (2007, p. 64) also reported that “nearly half the gains in particular skills discussed by faculty and students concerned communication skills.” In a similar study focused on undergraduate research experiences at a research university, Thiry and Laursen (2009) reported that student outcomes were remarkably consistent with those found by Hunter and colleagues in the previous 2007 study. In the study by Bauer and Bennett (2003), alumni who participated in URE programs reported significantly higher ability to carry out research and greater enhancement of communication skills than their counterparts who did not participate in URE programs.

In recent years, researchers have begun to address the issue of whether or not the length or intensity (e.g., Summer only vs. full- or multiyear experiences) and entry time (e.g., first year vs. later in one's undergraduate career) influence the benefits of UREs to participants (Thiry et al., 2012). Some studies have documented positive correlations between the length of time spent in URE programs and the benefits to students (Bauer and Bennett, 2003; Russell et al., 2007; Jones et al., 2010). In their evaluation of the impact of University of Delaware undergraduate research programs, Bauer and Bennett (2003) conducted a survey of alumni and found a positive link between the numbers of semesters spent in UREs and the level of self-reported benefits. Similarly, in their study of the importance of URE for minority persistence and achievement in biology, Jones and colleagues (2010) found a strong positive correlation between the number of terms spent in URE and persistence and performance in biology, as well as a positive correlation between longevity in research and college retention. Russell and colleagues (2007) reported positive effects of participation duration on aspirations for graduate education and research careers. Although these studies suggest that URE duration influences student self-reported outcomes, questions regarding the trajectories of student development in URE programs remain.

Using data from participants in two similar URE programs administered by a research center at a midwestern research university, Adedokun and colleagues (2012a) compared program outcomes between Summer URE participants and three other groups: academic-term participants (i.e., students who participated in either a Fall or Spring semester); two-term participants (i.e., students who participated in a Fall and a Spring semester); and full-year participants (i.e., students who participated for 12 mo). Their analyses revealed no significant differences in outcomes between the Summer and single-semester groups. However, they found that the students who participated for two academic semesters reported higher levels of awareness of available research career opportunities and higher skills in writing research papers for publications than did Summer participants. Adedokun and colleagues (2012a) also found that participants in yearlong experiences reported greater understanding of research procedures; awareness of available research career opportunities; and research skills, including organizing research ideas in writing, working independently on research projects, conducting literature searches, writing literature reviews, writing research papers for publications, and overall research skills.

In summary, the studies reviewed above suggest that longer-term UREs produce higher gains. However, the trajectories of student development in URE programs remain unclear. Although there are some indications that students experience different URE benefits “at developmentally different stages of their academic careers” (Hernandez et al., 2012, p. 14), little is known about student development within these programs. As noted by other researchers (e.g., Sadler et al., 2010; Adedokun et al., 2012b), a major limitation in URE research and evaluation is that the focus is often on program outcomes, with little to no focus on the processes and conditions under which the goals or outcomes are achieved. There is the need for research to clarify and understand how students develop in URE programs, and what outcomes are achieved at what time. Although some recent studies (e.g., Hernandez et al., 2012) have examined the longitudinal effects of UREs, more studies are needed on the trajectories of student development in URE programs, in particular, how program duration might influence student achievement of expected program outcomes.

The purpose of the present study was to examine the effect of time on students’ perceived gains (e.g., student awareness of research careers, understanding of research processes, and various research skills) from a URE. The key question guiding the study is: Do students report higher levels of perceived gains as they spend more time in the program? That is, do student self-reported scores on outcome variables differ significantly across time?

Although Bauer and Bennett (2003) examined the relationship between program duration and reported outcomes, the current study differs in two regards. First, participants in the Bauer and Bennett study included graduates drawn from a database that contained “records of individuals who were known to have conducted undergraduate research and to have received some service from the [undergraduate research program], ranging from an initial advisement meeting, to funding, to production of a senior thesis” (p. 218), suggesting that the participants were in a variety of URE models that could differ in program components and dynamics. The current study, on the other hand, uses data from students who participated in the same URE program and experienced similar program components and dynamics. Second, while Bauer and Bennett used data collected at a single point in time (i.e., postparticipation) to examine the relationship between program duration (i.e., number of semesters spent in URE) and perceived benefit, the current study focuses on trajectories of student development of outcomes and uses data collected at three points (i.e., pre-, mid- and postparticipation) to examine whether students reported higher levels of outcomes as they spent more time in the program.

METHODS

Participant Description

To address our research question, we collected data from participants in a URE—Purdue University's Cancer Prevention Interdisciplinary Education Program (CPIP). The goal of the CPIP program is to prepare students for future careers in interdisciplinary cancer prevention research. CPIP aims to provide students with the knowledge, experiences, and professional networks that will support them in conducting research across disciplines and in diverse contexts. The program is a 12-mo, multifaceted training model that includes experiential components in research and service, a traditional course-based component covering current knowledge in cancer prevention research across disciplines, and mentoring components that provide each student with multiple mentors from multiple disciplines and career stages. CPIP undergraduate interns participate in four programmatic components: 1) yearlong research experiences (i.e., Summer to Spring semesters); 2) a seminar class to learn basics of cancer biology and cancer prevention; 3) discussion sections to learn more about communication of research, career goals, and research in cancer prevention; and 4) service learning to understand the social relevance of their research. The CPIP program is described in detail in Teegarden et al. (2011). Briefly, the selection process begins with the submission of cancer prevention research projects from faculty. Research projects involve faculty from science, engineering, and behavioral sciences. Each submitted project must be interdisciplinary and involve faculty mentors from at least two disciplines. Undergraduate students, who must have a cumulative grade point average of 3.0 or higher, are recruited to work on these projects. Faculty members review applicants and select students for their project through an interview process. Students represent disciplines from across campus (including science, engineering, health sciences, behavioral sciences, and liberal arts). Data for the current study come from participants in the first three cohorts of the program. Of the 32 participants in the first three cohorts of the program, 27 completed the three surveys. Table 1 describes the academic and demographic characteristics of the students.

CPIP Program Components

A cornerstone of the CPIP program is participation of undergraduate students in research focused on cancer prevention. Undergraduate interns devote a minimum of 400 h of research during the Summer months and continue their research during the academic year on a part-time basis. Students are required to present posters communicating their research at a minimum of three on-campus symposia. Separate preparations of material for the poster sessions afford students the opportunities to practice their research and presentation skills, including analyzing and summarizing data, documenting research procedures, writing results of experiments, and orally discussing findings.

In addition to conducting interdisciplinary research projects, students work in teams to design and conduct a service activity in the local community. This activity must be related to cancer prevention and can involve, but is not necessarily limited to, educational, public health, or philanthropic efforts. Students are mentored by faculty, staff, and graduate students in this effort; however, the determination of the domain and scope of the projects remains the responsibility of the students. These research and service experiences run throughout the full year of the students’ participation in the program. Students are required to complete reflective journaling assignments that prompt them to think about their personal and professional growth and their perceptions of interdisciplinary research.

Moreover, the interns participate in a one-credit course focused on interdisciplinary approaches to cancer prevention research during the academic year. The goals of this course are to introduce students to the diversity of cancer prevention research and to assist them in developing as researchers. Through presentations by cancer prevention researchers working in multiple disciplines and contexts, students learn about a wide range of ongoing research and career options in cancer prevention. Sessions include speakers from a wide variety of disciplines (biology, communications, engineering education, nutrition, pathology, psychology, and statistics) and research settings (laboratory, clinical setting, and community) and include discussions with cancer survivors and oncologists that demonstrate the human face of cancer and the importance of cancer prevention. The seminars also provide students with the opportunity to interact and network with researchers from various disciplines. Additionally, students work together in pairs to plan an interdisciplinary research study and discuss what skills and resources are required to work in interdisciplinary teams.

Throughout the year, CPIP interns also participate in professional development activities including seminars on general topics, such as communication of research findings, research funding sources, interpreting primary literature, and the expectations of graduate school. Interns also participate in informal discussion groups to foster the exchange of research ideas and the development of a learning community. During these meetings, students practice oral research communication and receive feedback from their mentors and peers. These events allow participants to become comfortable discussing their own research, to reexamine their research goals and strategies, and to learn about the research interests and work of participants from different disciplines. This venue also allows for informal discussion of research with participants, which enhances research skills, professionalism, and an understanding of interdisciplinary research efforts. Finally, the CPIP program provides opportunities for social networking with faculty mentors and fosters the creation of multiple mentor–mentee relationships. Students are paired with primary and secondary research mentors from different disciplines for their interdisciplinary research project. In addition to these research mentors, the leadership faculty and a small group of cancer prevention graduate researchers form a network of additional mentors and role models for students (interested readers should see Teegarden et al. [2011] for further descriptions of the structures and processes of the CPIP program).

Instrument Description

Data for this study come from a survey designed to examine the impact of the program on student research skills, research confidence, and understanding of research processes. This survey was administered to the students at three separate times during their participation in the yearlong program. The first measurement (hereafter termed “pre”) was taken immediately before their participation in the program (i.e., at the start of the intensive 10-wk Summer research program); the second measurement (hereafter termed “mid”) occurred at the end of the 10-wk intensive Summer research experience; and the third (hereafter termed “post”) occurred during the last week of the program (i.e., the end of the Spring academic semester, roughly 1 yr after the preparticipation measures). The survey items were adapted (or modified) from Bieschke et al. (1996), Kardash (2000), and Russell (2006). We conducted reliability analysis to examine the internal consistency of the survey items. The overall reliability estimate, as measured by Cronbach's alpha was 0.94, 0.93, and 0.92 for the pre-, mid-, and postsurveys, respectively. However, the limited sample size of 27 did not allow for factor analyses to reduce the number of variables. For the purposes of this study, each of the survey items is treated as a separate variable. To simplify the discussion of results, we sorted the items into four distinct outcome categories (described in the following four sections) based on face validity and similarities in the concepts addressed by the items. Reliability estimates (i.e., Cronbach's alpha) for each category of outcomes at the each time period are also discussed.

Research Skills.

The survey asked the students to rate their skills/abilities in performing various research skills, including writing a literature review, documenting a research procedure, observing and collecting data, interpreting data, and so on. Response categories for these items ranged from “no ability” = 1 to “excellent ability” = 5. Cronbach's alphas for the items were 0.92, 0.87, and 0.87 for the pre-, mid-, and postparticipation data points, respectively.

Awareness of Career Options.

The survey asked students to rate their awareness of “what graduate school may be like,” “expertise of the faculty you worked with,” “various research-oriented career opportunities available to you,” and “various research career options you could specialize in.” Response categories for these items ranged from “not at all” = 1 to “a great deal” = 5. Cronbach's alphas for the items were 0.76, 0.76, and 0.70 for the pre-, mid-, and postparticipation data points, respectively.

Research Confidence.

Students were asked to indicate their confidence in: “your research skills generally,” “your ability to succeed in graduate school,” and “your qualifications for jobs in related fields.” Response categories for these items ranged from “not at all” = 1 to “a great deal” = 5. Cronbach's alphas for the items were 0.76, 0.78, and 0.83 for the pre-, mid-, and postparticipation data points, respectively.

Understanding of Research Processes.

This group of items measured “student understanding of the nature of scientific knowledge” (Hunter et al., 2007). Students were asked to rate their understanding of “how to formulate a research question,” “how to plan a research project,” “how scientific knowledge is built,” and so on. Response categories for these items ranged from “no ability” = 1 to “excellent ability” = 5. Cronbach's alphas for the items were 0.89, 0.87, and 0.78 for the pre-, mid-, and postparticipation data points, respectively.

Data Analysis

To address our research question, we used repeated-measures analysis of variance (ANOVA) with a Greenhouse-Geisser correction to examine the effect of time on the outcome variables, with the overarching goal of understanding the effect of time on students’ self-reported scores on each variable.

Typically, repeated-measures ANOVA assumes sphericity (i.e., that the variances of the differences between pairs of conditions/time periods are equal). Violations of this assumption affect the accuracy of the ANOVA F statistic by reducing its power (i.e., type I error). Hence we applied the Greenhouse-Geisser correction for sphericity, a conservative criterion of statistical significance (Stevens, 1996). For variables for which statistically significant changes were detected, contrasts/post hoc tests using the Bonferroni correction were conducted to identify specific differences among the three time points. Given that the data were pooled from students across three program cohorts, we conducted tests of parallelism (Heck et al., 2010). Specifically, we added the variable “cohort” into the ANOVA models as a between-subjects predictor to examine whether the observed growth patterns were the same across cohorts and to test whether nesting into cohorts did influence the findings.

RESULTS

Overall, the analyses revealed four patterns of change: 1) statistically significant immediate gains defined as changes between the pre- and midparticipation scores only; 2) statistically significant long-term gains defined as changes between the pre- and postparticipation scores only; 3) statistically significant immediate and long-term gains defined as changes between the pre- and midparticipation scores, as well as between the pre- and postparticipation scores, but with no statistically significant changes between the mid- and postparticipation scores; and 4) statistically significant consistent growth across the three time periods defined as significant changes from pre- to mid- and mid- to postintervention. Figures 1–4 provide examples of variables representing the four patterns of change, respectively.

With regard to possible cohort differences in the observed changes, tests of parallelism were not statistically significant, suggesting the observed patterns of changes are similar across cohorts.

Research Skills

The results (see Table 2) showed statistically significant immediate increase in observing and collecting data at the end of the Summer experience; the average midparticipation score for this variable was statistically significantly higher than the preparticipation scores. The analyses also revealed statistically significant long-term (i.e., pre-post) gains in student skills in organizing research ideas in writing, interpreting data by relating results to the original hypothesis, orally communicating the results of research projects, and writing a research paper for publication. Further, the results showed statistically significant immediate and long-term gains in student abilities to document research procedures, conduct a literature search, work independently on a research project, conduct statistical analysis using software, relate results to the bigger picture in their fields, write the results of their experiments, and prepare research posters. For these variables, the mid- and postparticipation scores were significantly higher than the preparticipation scores, but there were no statistically significant differences between the mid- and postparticipation scores.

Moreover, the analysis showed statistically significant changes across the three time periods in student ability to write a literature review and understand journal articles. However, while the pre→mid→post changes in student ability to write a literature review could be described as a consistent growth across the three time periods, the results showed that, for student ability to understand journal articles, there was a statistically significant increase between the pre- and midparticipation scores, but a statistically significant decrease between the mid- and postparticipation scores.

Awareness of Career Options

As shown in Table 3, the analyses revealed statistically significant immediate (i.e., pre→mid) gains in student awareness of what graduate school may be like. Moreover, our analyses revealed statistically significant immediate and postparticipation gain in student awareness of the expertise of their faculty mentors. The analyses also showed consistent growth in student awareness of various career opportunities available to them and research options in which they could specialize.

Research Confidence

The results (see Table 3) showed that students reported statistically significant greater confidence in their qualifications for jobs in related fields on the posttest than on the pre- and midmeasurement points. Also, compared with the preparticipation scores, students reported higher confidence in their research skills at the midmeasurement and at the postmeasurement. Postparticipation scores were also higher than the midparticipation scores, but not strong enough to attain statistical significance.

Understanding of Research Processes

Table 3 shows that students made statistically significant long-term gains in their understanding of: 1) how to plan a research project, 2) how to conduct a research project, and 3) their overall understanding of the research process. The mid- and postparticipation scores for these variables were statistically significantly higher than the preparticipation scores, although there were no statistically significant differences between the mid- and postparticipation scores. Moreover, the analyses revealed statistically significant consistent growth in student understanding of how scientific knowledge is built and their understanding of the nature of the job of a researcher.

DISCUSSION

Overall, the findings of this study are in line with those of previous studies and also reflect the specific contexts of the CPIP program.

As indicated in the Results, we observed immediate statistically significant increase in observing and collecting data at the end of the Summer experience. Researchers (e.g., Sadler et al., 2010) have observed that data collection is a basic task that URE interns perform when they are novices in their research groups. Sadler and colleagues noted that new URE students, as apprentices, often begin with menial or “legitimate peripheral activities” (e.g., data collection) that “tend to support” other complex tasks (Sadler et al., 2010, p. 236) tasks. Given that the interns in the CPIP program started their internship in the Summer (i.e., Summer was their novice phase, when they were likely assigned to data-collection tasks in their labs), it is not surprising that that they reported statistically significant gains in their abilities to observe and collect data at the end of the Summer.

The interns reported statistically significant long-term gains in the research skills that require time to develop and that often occur at the tail end of research projects (e.g., interpreting data by relating results to the original hypothesis, orally communicating the results of research projects, working independently on a research project, and relating results to the bigger picture in their fields). These also appear to be skills that the undergraduate research interns may not be able to fully develop during their Summer experiences. Our findings are thus in line with Thiry and colleagues’ (2012) assertion that URE students are more likely to report gains in higher-order skills (such as presenting at conferences and writing papers) at the advanced stages of their research, because it typically takes time to generate preliminary data. Sadler et al. (2010) also noted that URE students learn the trade and culture of scientific research in an incremental manner, whereby they start off with menial or basic tasks and are assigned to more central and complex activities and skills as they become more familiar with the tools and culture of their research groups.

These findings are also a reflection of the program requirements. As indicated earlier, CPIP students were required to participate in a minimum of three poster sessions. While student performances were not graded, their preparations for the poster sessions gave them the opportunity to analyze and summarize their data, document research procedures, write the results of their experiments, and design scientific posters. The actual poster presentation also afforded them the opportunity to orally communicate the results of their studies. Hence, it is not surprising that the students reported statistically significant levels of these skills in the mid- and posttests.

As reported, the results also showed consistent growth in student ability to write a literature review. Intuitively, writing literature reviews is a skill that students often have to perform/practice multiple times to improve. Even graduate researchers get better in literature review with practice. We suspect that this explains why the students reported continuous growth in this area. However, the significant increase followed by significant decrease in perceived ability to understand journal articles is much more difficult to interpret, especially because the students were required to read multiple papers across various research and content areas.

With regard to student awareness of research career options, we observed statistically significant immediate (i.e., pre→mid) gains in student awareness of what graduate school may be like. This is not surprising, given that the interns began their URE in the Summer and were likely mentored by graduate and postdoctoral researchers. Researchers (e.g., Dolan and Johnson, 2009; Thiry and Laursen, 2011; Thiry et al., 2012) have observed that when URE students are new to their labs (i.e., as novice researchers), they are typically mentored by graduate students and postdocs, whom they (undergraduates) often view as professional role models. These mentors help URE interns to socialize into the research community of practice, provide guidelines and teach them the laboratory techniques, and often provide interns with career guidance/encouragement to pursue graduate studies and “broaden their understanding of educational and career options in their field” (Thiry and Laursen, 2011, p. 782). As discussed in the program description, the students attended weekly brown-bag sessions in the Summer, during which they participated in discussions about research career choices and graduate education. Thus, it is not surprising that our study participants, at the end of their Summer experiences, reported significantly higher awareness of what graduate school may be like.

We also observed immediate and postparticipation gain in student awareness of the expertise of their faculty mentors. A possible explanation for this is that the interns became more aware of the expertise of their faculty mentors as they progressed in their research apprenticeships and became more “socially and intellectually integrated” into their respective research groups (Thiry and Laursen, 2011, p. 780). The analysis also yielded positive changes in interns’ awareness of the various career options and specializations. This is not surprising, given that the interns participated in a one-credit course that introduced them to the diversity of cancer prevention research and researchers working in multiple disciplines (e.g., biology, engineering education, nutrition, pathology, psychology, and statistics) and research settings (laboratory, clinical setting, and community). We suspect that this interdisciplinary experience contributed to the observed statistically significant increase in student awareness of various research careers.

Turning to the findings regarding student research confidence, our results suggest that as students progress beyond the novice stage of their research experiences, they begin to develop higher confidence in their research abilities. The level of confidence also increases as students spend more time in the research environment and gain “responsibility and independence within their research groups” (Thiry et al., 2012, p. 268). Students also reported statistically significant long-term gains or consistent growth in items assessing their understanding of research processes (e.g., understanding of how scientific knowledge is built), suggesting that these abilities take time to develop. The 1-yr CPIP research experience, coupled with the poster sessions and seminar series that provided students with the opportunity to discuss issues related to scientific processes, enhanced their understanding of research processes and overall research confidence. These findings are also in line with Sadler and colleagues’ (2010) statement that “it takes time for apprenticeship participants to feel comfortable in their research settings to really understand the science being conducted” (p. 252). Thiry and colleagues (2012) also argued that experienced students (i.e., students who have spent well over a semester in their research apprenticeships) are more likely to gain in their conceptual understanding of their research project and are also more likely to understand the big picture of their research, because it takes time to develop these skills.

SUMMARY AND CONCLUSION

As stated by Hernandez et al. (2012), “Students experience different benefits at developmentally different stages of their academic careers” (p. 91). Our data suggest that the CPIP interns experienced different gains/benefits at developmentally different stages of their UREs. Our study participants reported gains in fewer areas at the end of the Summer compared with the end of the yearlong experience, thus supporting the notion and argument that longer UREs offer students the most benefit. The findings suggest that the CPIP program offered the participants excellent opportunities to practice and improve their research skills. Our findings are also in line with previous studies (e.g., Thiry and Laursen, 2011) that reported that interns often start their research apprenticeship performing menial or basic tasks (such as data collection and entry) and gradually move on to more technical and complex activities.

Sadler and colleagues (2010) noted that it takes time for student to fully grasp the underpinnings of research processes; they encouraged program designers to “consider ways of sustaining these experiences” by providing extended research opportunities (p. 252). They added that “if the primary aim of an apprenticeship program is to foster sophisticated understandings and practices, then length of experience, instructional supports and opportunities to engage in epistemically demanding practices ought to be prioritized design principles” (p. 253). Overall, the findings of the current study suggest that length of experience is related to student growth and that further studies using larger samples are needed to further understand the trajectories of student development in URE programs.

LIMITATIONS

This study is not without limitations. First, the small sample size, coupled with the nonrandom nature of the data, limits us from conducting additional statistical analyses to examine the effects of other variables (including previous research experiences, gender, and academic major). Another limitation to our study is that we were unable to control for the effect of research group on student development. Although all the students had a URE experience, the specific or lived experiences in the program would differ by lab. Students who had positive interactions with graduate and faculty mentors in their groups are more likely to have a growth pattern that is different from students with fewer interactions. This is a limitation that is common and noted in other URE studies (e.g., Lopatto, 2004). Finally, the lack of a control group hinders us from capturing the possible effect of maturation (i.e., biological or psychological changes within students) on student self-reported skills across the three data points. Overall, these limitations offer opportunities for further analysis as we collect additional data in future years.

REFERENCES