Evaluating Psychosocial Mechanisms Underlying STEM Persistence in Undergraduates: Scalability and Longitudinal Analysis of Three Cohorts from a Six-Day Pre–College Engagement STEM Academy Program

INTRODUCTION

Many national reports predict a need for qualified individuals to fill nearly one million new science, technology, engineering, and mathematics (STEM) jobs over the next 10–20 years (American Association for the Advancement of Science, 2011; Carnevale et al., 2011; Hossain and Robinson, 2012; President’s Council of Advisors on Science and Technology, 2012; Fayer et al., 2017). The United States lags far behind other countries in the proportion of students graduating with STEM degrees (Kuenzi, 2008). The need for more qualified talent, coupled with pioneering work by Seymour and Hewitt (1997) on the loss of undergraduate students from STEM, sparked STEM reform efforts at many universities that endeavor to increase the retention of STEM majors, especially those from underrepresented minority groups. STEM reform efforts often focus on retaining underrepresented minority students, because high proportions of proficient female students and students of color switch to non–STEM careers (Griffith, 2010), resulting in the loss of qualified talent. Recent research efforts are focusing on the efficacy of STEM retention programs, because these programs differ in the types of activities they offer, scale, cost per student, and length and timing of the programs. Ashley and colleagues (2017) evaluated 30 different STEM bridge programs to highlight informational gaps and provided recommendations for increased publication of program descriptions, program implementations, and any issues encountered during a program. Of particular importance, they stressed the need for more comprehensive assessment of program impacts on student outcomes. To address the gaps outlined by Ashley et al. (2017), we present results from three STEM Academy (SA) cohorts. The SA is one type of pre–college engagement (bridge) program aimed at recruiting and retaining underrepresented minority groups as STEM majors. Findings from the initial SA cohort (Findley-Van Nostrand and Pollenz, 2017) demonstrate promising program impacts such as higher retention rates and positive attitudinal changes such as increased sense of belonging and science identity. The present study expands upon these previous findings by evaluating the impact of two more SA cohorts.

The SA and STEM Retention Efforts

The SA early-engagement bridge program was implemented with 116 first-year STEM undergraduate students in August 2015 at the University of South Florida. The SA is immersive, but short, lasting only 5.5 days and occurring the week before the start of the semester. Students residing on campus move into their assigned residence a week early to complete the SA and then transition directly to the first day of classes. The SA program serves a diverse population of STEM majors who enter the university with average Scholastic Aptitude Test (SAT)/ACT scores below the general first-year population. The purpose of the SA program is to create an initial cocurricular community-building experience for incoming students that feeds into an academic small-group, first-semester experience. Ultimately, SA students’ experiences will align with active-learning environments in the gateway STEM courses. Thus, the SA is distinct from the boot camp–style programs (such as the BIOS program at LSU) that typically occur during the summer and have a significant break before the students matriculate (Wischusen and Wischusen, 2007; Wheeler and Wischusen, 2014). However, like many bridge programs, the SA was designed to include high-impact practices (Kuh, 2008) and includes 18 programming modules that focus on writing, hands-on lab exercises, career planning, undergraduate research, math competencies, and team building, all with significant one-on-one engagement and collaboration with peers, graduate mentors, and faculty. Many of these practices have been implemented to various degrees in longer academic programs such as the Meyerhoff Scholars Program (Pender et al., 2010; Stolle-McAllister et al., 2011; Maton et al., 2016), the Biology Scholars Program (Matsui et al., 2003), the Freshman Research Initiative (Rodenbusch et al., 2016), and the SEA-PHAGES program (Caruso et al., 2009; Harrison et al., 2011; Hanauer et al., 2016). Although many of these programs helped inform the selection of SA content, a primary objective of the SA program design was to promote community building and belonging through small-group learning and collaboration with graduate and undergraduate mentors.

The Importance of Belonging and Community in STEM Retention

A central hypothesis driving research on the SA proposes that, through increased science identity, self-efficacy, and especially sense of belonging, students will experience more STEM success. Self-determination theory (SDT) posits that belonging (along with competence and autonomy) facilitates intrinsic motivation underlying positive educational outcomes (Deci and Ryan, 2012). To illustrate, students fare better when they experience positive social interactions and peer relationships (Fass and Tubman, 2002; Dennis et al., 2005; Estrada et al., 2018). In addition, the ability to develop a sense of belonging with their institutions is associated with student retention (O’Keeffe, 2013). Because needs for relatedness are associated with a host of positive academic outcomes and improved psychological well-being, one goal of the SA is to cultivate a strong sense of belonging to the university and STEM. Closely related to sense of belonging is identity formation. The desire to belong is translated into an identification with a given context such as STEM, and identifying with relevant contexts predicts increased persistence and success in that context (Chemers et al., 2011). Importantly, stronger science identities predict the likelihood of a student pursuing a career in science (Estrada et al., 2011). Therefore, a second goal of the SA is to cultivate incoming students’ science identities throughout the duration of the program.

In line with SDT’s emphasis on the role of competence, social cognitive theory proposes that self-regulation is achieved through developing strong feelings of self-efficacy (Bandura, 1982, 1986, 1991, 1997). Research consistently demonstrates that high self-efficacy is a predictor of academic and STEM-specific achievement (Andrew, 1998). Because feelings of competence are critical for achievement domains, a third goal of the SA is to increase students’ STEM self-efficacy. Finally, the SA may also benefit students by increasing the level of perceived parental support. While evaluating actual parental support is beyond the scope of this study, perceived parental support is linked to academic persistence (Cabrera et al., 1992), academic performance (Bank et al., 1990; DeBerard et al., 2004), and decreased academic stress (MacGeorge et al., 2005). Because a student’s perception (whether accurate or not) of familial support is closely tied to academic success, we sought to evaluate how participating in the SA is related to perceived support. Though it is unclear whether participating in the SA actually increases parental support, students may feel stronger support post-SA due to a myriad of positive attitude changes.

To measure attitudinal changes, more undergraduate STEM retention programs are employing validated psychosocial instruments to address each program’s impact. For example, the academic Meyerhoff Scholars Program and cocurricular Ohio Science and Engineering Talent Expansion Program report that participants’ sense of belonging increased, which may impact retention (Maton et al., 2016; Tomasko et al., 2016). In addition, the SEA-PHAGES course-based undergraduate research program shows that science identity and belonging to the community positively correlate with intent to become a research scientist (Hanauer et al., 2017). Research on the SA builds on these previous findings by evaluating similar constructs such as sense of belonging, science identity, and retention rates. By comparing SA students’ pre- and posttest attitudes, Findley-Van Nostrand and Pollenz (2017) identified increases on measures of belonging and science identity. Further, students from the SA reported stronger scores on these measures compared with a matched comparison group (MCG). Therefore, two goals that guide the present study include replicating the findings from the original report (e.g., Findley-Van Nostrand and Pollenz, 2017) on additional and larger cohorts (scale) and beginning to unravel the mechanisms behind these positive attitudinal changes.

Present Study

The 2017 report focused on the implementation and assessment of a single cohort of 116 students from 2015 (Findley-Van Nostrand and Pollenz, 2017). Since that time, two additional cohorts of students completed the SA program. Thus, the present study includes four aims: 1) report on increasing the scale of the original SA program to more than 200 students; 2) replicate the attitudinal changes evaluated pre- and post-SA program over two additional cohorts; 3) assess the mediating mechanisms that predict students’ increase in science identity and belonging; and 4) demonstrate the impact of the program through longitudinal analysis of university and STEM retention for the SA and MCG students in the 2015 and 2016 cohorts.

METHODS

Participants

All participants in this project were first time in college students (FTIC) at the University of South Florida (USF) who matriculated in Summer/Fall of 2015, 2016, or 2017. There are five different cohorts of students described in this report: three SA cohorts (2015, 2016, and 2017) and two MCG cohorts (2015 and 2016). The specifics of the recruitment for each group are summarized in the following sections. The university’s institutional review board approved all procedures involving human participants.

SA Participants.

All admitted FTIC students who had a quantitative SAT score of <650 or ACT of <25 and who selected a STEM major from the disciplines of biology, chemistry, physics, math, engineering (chemical or biomedical only), or geosciences, were invited to apply to the SA program. In any given year, approximately 2000 admitted students meet the eligibility criteria and are invited to apply through direct email and parent letters. Those who apply are reviewed and accepted into the program on a first-come-first-served basis. Typically, 98% of applicants are accepted. Those not admitted to the program are usually from an outside major not supported by the SA (e.g., computer science). The sizes of the full SA cohorts were 116 students in 2015, 222 in 2016, and 149 in 2017. However, for this report, only students who completed both the pre and post identity surveys were included in the results. Therefore, the 2015 SA cohort is 109 students, the 2016 SA cohort is 190 students, and the 2017 SA cohort is 140 students. The MCGs for 2015 and 2016 include 109 and 190 students, respectively. The demographics for each cohort are summarized in Table 1, and these values align with the overall FTIC population across the majors described above. In each cohort, ∼80% of the population is enrolled in a life science major (biomedical sciences or biology) and ∼20% in the physical sciences and biomedical or chemical engineering majors. In addition, the SA cohorts included ∼25% first-generation students, and ∼30% of the cohort was eligible for Federal Work-Study. Additional details of the SA recruitment and admission process can be found in the Supplemental Material for the previous report (Findley van-Nostrand and Pollenz, 2017).

Matched Comparison Group.

Participants were recruited from the introductory chemistry courses, because most STEM majors at the university take this course in their first semester. For the 2016 MCG, more than 1000 students completed the identity survey as a part of their course for extra credit. Of the initial pool, 693 were considered part of the “full comparison group” based on the criteria of first-year status and STEM major (i.e., transfer or more advanced students and non–STEM majors were excluded from analysis). Because random assignment to the program and a control condition was not possible, we used a propensity-scoring procedure to create the MCG group. To establish the 2016 MCG, we followed relatively common procedures (see outlines by Ho et al., 2007; Lane et al., 2012; Thoemmes, 2012; Rodenbusch et al., 2016) and selected the same covariates described in Findley-Van Nostrand and Pollenz (2017). This included using several demographic variables (high school grade point average [GPA], SAT score, race, gender, and Federal Work-Study status) to calculate a score reflecting the probability of being assigned to the treatment (i.e., program) condition and subsequently matching students based on similarity in scores. This resulted in an MCG of 190 students. The individual difference variables selected for the 2015 MCG are described in Findley-Van Nostrand and Pollenz (2017), and the demographics for the 2015 and 2016 MCGs are summarized in Table 1. Various institutional issues, such as changes to faculty who were teaching the chemistry course sections we used to deliver the survey, prevented our establishment of an MCG for the 2017 SA.

Measures and Procedure

To evaluate the 2016 and 2017 cohorts, we administered the same measures described in the previous study (Findley-Van Nostrand and Pollenz, 2017). Surveys were completed online via Qualtrics. All measures were reliable across the SA and MCG samples (Table 2). Factor analysis also replicated the factorial validity established in the first cohort (for the details and importance of this approach, see Findley-Van Nostrand and Pollenz, 2017). The measures are discussed in the following sections.

Self-Efficacy for Academic Course Work.

Four items based on the SDT literature (Williams and Deci, 1996; Ryan and Deci, 2000) assessed self-efficacy for course work (e.g., “I feel confident in my ability to learn the material in my STEM courses”; five-point scale from “not at all” to “a lot”). To create an academic self-efficacy composite, we averaged the four items.

Self-Efficacy for Science-Related Tasks.

In a seven-item scale developed by Chemers and colleagues (2011), students were prompted to think about a project that they are involved in or may get involved in, and indicate the extent to which they are confident they can complete several tasks (e.g., “use scientific literature to guide research”; four-point scale from “not at all/a little” to “a lot”). All seven items were averaged to create a task-related self-efficacy composite.

Science Identity.

Identity as a scientist was examined using five items developed by Chemers and colleagues (2011; e.g., “My interest in science is an important reflection of who I am”; four-point scale from “not at all/a little” to “a lot”). The five items were averaged to create a STEM identity composite.

Expectancy for Career in STEM.

Four items assessed positive expectancies for a career in STEM. We partially modified a scale by Stake and Mares (2001) to be specific to “STEM” rather than “science.” Participants read the prompt “Please think about yourself and rate how true the following statements are” and then indicated the degree to which the student agreed with each statement (e.g., “I would enjoy a career in STEM”; seven-point scale from “not at all true” to “very true”). The four items were averaged to create a career expectancy composite, with higher scores indicating stronger career expectancies.

Sense of Belonging to University.

We modified the eight-item sense of community scale (Peterson et al., 2008) to reflect sense of community in the university context. Participants read the prompt “Please think about yourself as a member of this university, and rate your level of agreement with each statement. If you are new to the university, please use your initial feelings” and then rated the level of agreement with statements such as “I belong at this university” (five-point scale from “strongly disagree” to “strongly agree”). All eight items were averaged into one university belonging composite.

Sense of Belonging to STEM.

Belonging to STEM was measured using a modified version of a 30-item scale developed by Good and colleagues (2012). The original scale was modified only in that the original context of math settings/courses was edited to reflect “STEM” settings/courses. This scale includes several subscales assessing belonging in terms of membership (e.g., “I consider myself a member of the STEM community”), acceptance (e.g., “Regarding the STEM community, I feel respected”), affect (e.g., “Regarding the STEM community, I feel anxious”), desire to fade (e.g., “Regarding the STEM community, I try to say as little as possible”), and trust in instructors (e.g., “I trust my instructors to be committed to helping me learn”). Given that this project entails a pre-college program, we did not analyze the trust in instructors subscale, as these items assume that students have begun their course work. Note, “desire to fade” items were reverse-scored in order to be included in the composite sense of belonging scale (i.e., low scores indicated greater sense of belonging); that is, higher scores indicated lower desire to fade (and higher belonging). We also created a STEM belonging composite by averaging all 30 items.

Participants first read the prompt: “Today we have some questions we would like you to answer about your experience in the STEM community at this university. When we mention the STEM community, we are referring to the broad group of people involved in these fields, including the students in your STEM courses. Given this broad definition of the STEM community, please respond to the following statements based on how you feel about your membership in it. There are no right or wrong answers to any of these statements; we are interested in your honest reactions and opinions. Please read each statement, and indicate the number that reflects your degree of agreement.” Then, before the membership subscale items, they read “When I am in a STEM setting at (university)…,”and before the other subscale items, they read “Regarding the STEM community, please indicate your degree of agreement with the following…,” and then rated their level of agreement with each statement (seven-point scale from “strongly disagree” to “strongly agree”).

Intention to Leave STEM.

We used a scale developed by Perez et al. (2014), which included seven items assessing students’ current intention to remain in or leave their STEM majors (e.g., “I am likely to switch to a major that is not in a STEM field”). Participants responded using a scale ranging from 1 (strongly disagree) to 6 (strongly agree). We averaged across all seven items to create one intention to leave STEM composite. A higher score indicates stronger intention to leave a STEM major.

Perceived Parental Support.

Participants were asked to rate their perceptions of how much their families encourage their academic careers in STEM (e.g., “My family is enthusiastic about a STEM career for me”). Responses were recorded on a scale from 1 (not at all true) to 7 (absolutely true). Four items were averaged; higher scores indicate stronger perceived family support.

Survey Data Collection

SA students were introduced to the Qualtrics survey through email communication, during their USF orientation sessions, and through postings to the SA organization page on the university’s online learning platform. The SA students were evaluated at two time points. SA students completed the pre survey in the summer 4–8 weeks before the start of the SA. Post surveys were completed on the last day of the SA program before the start of the semester. The comparison group was evaluated at one time point approximately 2 weeks into the semester and could complete the survey over a 2-week window. Though the comparison group completed the survey slightly later than the SA posttest, both groups still completed the surveys at the start of the Fall semester.

SA Program Design and Scaling

The SA program was designed as a pre-entry, low-cost, cocurricular program that could be theoretically scaled to large numbers (240–500) of incoming first-year students based on its modular design and employment of graduate students and peers in leadership roles. All the undergraduate peers in the 2016 program were 2015 SA scholars or undergraduate researchers who worked with the graduate mentors. Importantly, selecting graduate student and undergraduate peer mentors reduces costs associated with the program. These students are provided 20–30 hours of training in mentoring and facilitated teaching, so that they gain key skills that enhance their career preparation and direct evidence of mentorship. The pilot SA program in 2015 was set at 120 first-year students (five groups of 24) and required five graduate mentors and 18 undergraduate peers. For 2016, the program capacity was increased to 240 (10 groups of 24) and employed 10 graduate mentors and 30 undergraduate peers. The increase from 120 to 240 did not present staffing or cost problems, because a benefit of running the initial SA program was a pool of potential undergraduate peer mentors who could be recruited and graduate mentors who could also advocate for the program and assist in recruiting others. The SA director’s department hired the graduate mentors and undergraduate peers. The mentors and peers are paid a total of $3000 and $500, respectively, over the summer before the SA program. From a logistical perspective, the program begins 4 days before the general student population arrives on campus. This can create issues with resources, so it was critical that the university provided early access to the students’ assigned rooms, dinning, security, and library space (where many of the full SA sessions were delivered).

Exploratory Analyses for the 2016 and 2017 Cohorts

In addition to investigating attitude changes from before and after the SA program, we also wanted to test what underlying mechanisms may predict these changes. Montoya and Hayes’ (2017) MEMORE macro was used for these analyses with 10,000 bootstrap samples to investigate what specific attitude changes from pre- to post-SA resulted in the increased feelings of belonging, science identity, and self-efficacy. Owing to the exploratory nature of these analyses, we investigated each combination of the primary variables of interest. That is, of the five primary attitudes of interest (i.e., science identity, self-­efficacy, STEM belonging, university belonging, perceived family support, and intentions to leave STEM), we explored within-subjects mediation models in which we selected each variable as either the mediator (M) or the outcome variable (Y). The analyses presented in the Results and Discussion section reflect only the models that yielded significance.

Longitudinal Analysis of University and STEM Retention

All students entering the university as FTIC in Summer/Fall 2015 or 2016 were evaluated for declared major at matriculation. STEM majors included biomedical sciences, cellular and molecular biology (Biology Department), chemistry/biochemistry and environmental biology (Biology Department), environmental microbiology (Biology Department), environmental sciences, geology/geosciences, and integrated animal biology (Biology Department), marine biology (Biology Department), mathematics and microbiology (Biology Department), physics, statistics, all engineering majors, and management information systems. Review of individual transcripts and STEM course trajectories were evaluated for university enrollment and declared major in March 2018. Students who did not enroll in the Spring 2018 semester were scored as “loss from the university,” and students who changed majors and did not re-enroll were also scored as “loss from the university.” Students changing out of one of the listed STEM majors to a non–STEM major who were not taking core STEM courses for at least two semesters after the change were scored as “loss to non–STEM major.” Notably, students who changed to a non–STEM major but were still fulfilling their STEM core requirements for at least two semesters past the major change were scored as “STEM retained,” because many pre-medical students major in non–STEM disciplines but still complete the required STEM curriculum. This included 11 students in the SA 2015, six students in the MCG 2015, 21 students in the SA 2016, and seven students in the MCG 2016, as noted in Table 1.

RESULTS

Psychosocial Gains for the 2016 and 2017 SA Cohorts

The SA program was delivered to 222 scholars in 2016 compared with 116 in 2015. To determine whether the psychosocial gains we observed for the first SA program were replicated with an increase in program scale, we present 1) mean-level differences in the variables across the duration of the 2016 and 2017 SA cohorts, including follow-up analyses to better understand unexpected results; 2) mean-level differences in the variables in the SA relative to an MCG of students for the 2016 cohort; and 3) exploratory analyses that investigate what mechanisms predict positive STEM outcomes for the 2016 and 2017 SA cohorts. Discussion of results and a comparison with the 2015 SA cohort findings are included.

Paired t tests assessed mean-level changes from the beginning to the end of the program for the 2016 SA cohort (see Table 3) and for the 2017 SA cohort (see Table 4). SA scholars showed a marginal increase in academic self-efficacy among the 2016 cohort and a significant increase among the 2017 cohort. Both cohorts demonstrated no change in task-specific self-efficacy across the 1-week program. SA scholars from 2016 also exhibited a marginal increase in science identity, while the 2017 cohort yielded a significant increase in this measure. The increase in science identity from pre- to post-SA among the 2017 cohort is consistent with the findings for the 2015 cohort (Findley-Van Nostrand and Pollenz, 2017). The lack of increase in science identity among the 2016 students may reflect a unique cohort that required more hands-on training to see this effect. Alternatively, the larger size of the 2016 cohort may have required more resources to promote stronger science identities. Because a smaller number of scholars participated in the 2017 and 2015 cohorts, these students gained more one-on-one contact with mentors, providing them with stronger support and resources. For the 2018 SA program, we assigned four peer undergraduates per graduate mentor and plan to assess whether this improves the formation of science identities.

Participants from both the 2016 and 2017 cohorts showed significant increases in belonging to the university and belonging to STEM. Increases were found for the composite measure and all subscales of the belonging to STEM scale (Tables 3 and 4). Specifically, students reported increased feelings of membership, acceptance, positive affect, and lower levels of desire to fade. These findings are consistent with the results from the 2015 cohort and suggest that the program builds community and peer connection as intended, even with a significantly increased peer-to-mentor ratio. The SA introduces students to other like-minded students and requires them to interact on group projects to work toward common goals. These interpersonal dynamics likely foster feelings of closeness and a stronger sense of belonging to a large university where they would otherwise have limited personal interactions with other STEM students and faculty. Similar to results from the 2015 cohort, scholars from 2016 and 2017 did not show changes in positive expectancies for a STEM career. Finally, contrary to expectations, scholars from both the 2016 and 2017 cohorts reported a significantly higher intention to leave STEM majors across the program. While this finding was surprising and does not appear to align with the other measures, additional data discussed in the section Recalibrating First-Year “Pre-med” Students’ Perceptions of the Pathway to Medical School provide context to this result.

Follow-Up Analysis: Intentions to Leave to STEM among Pre-medical Students

When prospective students apply to the SA program, they must complete a categorical question regarding their career choices and ∼75% self-select that they desire to earn an MD following completion of their degrees. This answer is consistent with their choice of the presumed “pre-med” major: biomedical sciences. We hypothesized that, due to participation in several of the medical school preparation/admissions modules that are part of the SA program, the SA scholars changed their perception that a STEM major is the only path to medical school. To provide support for this hypothesis, we evaluated scholars who selected the MD career path separately from scholars aiming for all other postgraduation career paths (i.e., professional school, graduate school, immediate employment). The results reveal that intention to leave STEM for the 2016 and 2017 cohorts increased across the SA program for only the scholars who aimed for medical school (2016: t(124) = −5.13, p < 0.01; 2017: t(82) = −3.59, p < 0.001). Intention to leave STEM did not change across the 2016 and 2017 SA programs for scholars who were not aiming for medical school (2016: t(60) = −1.36, ns; 2017: t(56) = −1.32, ns). Thus, we interpret this finding to suggest that SA scholars are more inclined to switch to non–STEM majors while still pursuing relevant STEM course work needed for medical school admission. This finding is validated by content analysis of transcripts from scholars who changed out of STEM majors, which shows that 90% of these students were still taking core STEM courses, even if those courses were not a requirement for the new major. This was not observed in students who switched to non–STEM majors in the MCGs, as also reported for a small population in the 2015 cohort (Findley-Van Nostrand and Pollenz, 2017).

Differences between 2016 SA and MCG Participants

In the first report, we showed that the 2015 SA cohort began their first semester with higher STEM self-efficacy, sense of belonging, science identity, positive career expectancies, and lower intention to leave STEM than students in an MCG. To determine whether these results replicated across the larger 2016 SA program, we established an MCG using propensity scoring (as described in the Methods section) and compared the mean-level values of the survey measures across both groups. Table 5 demonstrates that, relative to the MCG, the 2016 SA cohort scored higher in academic and task-specific self-efficacy, science identity, positive expectancies for a STEM career, belonging to the university, and belonging to STEM (composite and all subscales). The 2016 SA scholars showed a higher intention to leave STEM majors compared with the MCG. Thus, we compared differences between the SA cohort and the MCG for those participants who reported aiming for medical school. Among the participants (both within the SA and the MCG) who reported aiming for medical school, SA scholars were more likely to indicate they intended to leave STEM majors relative to the medical career–oriented students in the MCG (t(225) = −2.22, p < 0.05). Among the participants who were not aiming for medical school, there was no difference in intention to leave STEM majors between the SA and the MCG groups (t(180) = −1.54, ns). This finding is consistent with the data reported in the preceding section. Collectively, these results support the findings from the 2015 SA and indicate that scaling the program did not have a negative impact on the ability to build a community of scholars who began their college careers with an increased sense of belonging, self-efficacy, and science identity.

It is always challenging to establish a true “control” group when working in programs that require applications for admission. Therefore, it is important to note that, as reported in the initial analysis of the 2015 SA program (Findley Van Nostrand and Pollenz, 2017), the 2016 SA students scored higher in their pre scores of academic self-efficacy, self-efficacy for science tasks, science identity, positive expectancies for STEM career, and belonging in STEM and lower in intention to leave STEM than MCG students. Thus, there are clearly unobserved characteristics in the SA group that were not captured by the demographic variables selected for the propensity-scoring procedure but that may have impacted self-selection into the SA program. Though some individual differences in attitudes between the SA and the MCG students could not be accounted for with the propensity scoring, we still assert that the MCG provides a meaningful comparison, because we matched students on key demographic characteristics (e.g., SAT/ACT score, high school GPA, gender, race/ethnicity, and first-generation status).

Exploratory Research Analyses

To expand the scope of this study beyond the assessment of attitudinal changes from pre- to posttest, we performed exploratory research analyses on the data for the 2016 and 2017 SA cohorts. Repeated-measures mediation was used to assess what underlying mechanisms facilitate positive changes in STEM attitudes. Accordingly, we conducted exploratory analyses that tested the indirect relationships among key constructs. To evaluate the proposed models, we used Montoya and Hayes’ (2017) MEMORE as described in the Methods section. While the 2016 and 2017 SA cohorts were analyzed separately, the same analyses were conducted on both cohorts. The exploratory analyses demonstrated three significant models, although more than three models were tested. In the first model, university belonging was treated as the mediator and science identity as the outcome variable. In the second model, STEM belonging was treated as the mediator and science identity as the outcome variable. In the third model, family support was treated as the mediator and university belonging as the outcome variable. The first and second models yielded significance for the 2016 SA cohort, and all three models were significant for the 2017 SA cohort. Figures 1–5 display the outcomes of these analyses.

Among the 2016 cohort, the SA program directly increased science identity from pre- to posttest, and this effect was mediated by increases in university belonging, b = −11, SE = 0.03, 95% CI [−0.17, −0.05]. See Figure 1. Similar to the first model, the second model (Figure 2) demonstrated a direct increase in science identity mediated by STEM belonging, b = −11, SE = 0.04, 95% CI [−0.18, −0.04]. Within the 2017 cohort, all three models yielded significance. For the first model (Figure 3), increases in university belonging mediated increases in science identity from time 1 to time 2, b = −12, SE = 0.05, 95% CI [−0.25, −0.04]. For the second model (Figure 4), STEM belonging mediated an increase in science identity from time 1 to time 2, b = −0.07, SE = 0.03, 95% CI [−0.12, −0.02]. In the third model (see Figure 5), family support indirectly predicted an increase in university belonging from time 1 to time 2, b = −0.03, SE = 0.02, 95% CI [−0.06, −0.002]. The results of this analysis are consistent with other reports linking belonging and community to increases in self-efficacy and identity (Maton et al., 2016; Estrada et al., 2018).

Retention of 2015 and 2016 SA Cohorts and MCG Participants

To assess retention, we evaluated academic enrollment and major changes for all SA scholars and MCG students in the 2015 and 2016 cohorts through the Spring 2018 semester. Two factors contribute to STEM retention: 1) loss of a STEM student from the university, and 2) students who persist at the university but change from a STEM to non–STEM major. Table 6 shows the results for the 2015 SA, MCG, and full 2015 STEM FTIC. Current retention rates (data from 2018) demonstrate that the 2015 SA lost 7.3% of the cohort, compared with 9.2% for the MCG and 19.2% for the 2015 STEM FTIC (not including the 109 SA students). Loss from STEM due to transfer to non–STEM majors was also reduced in the SA (19.3%) compared with the MCG (44.9%) and the total FTIC (43.8.%). Thus, total STEM retention for the 2015 SA was a robust 73.4% compared with 55.1% for the MCG and 56.2% for the full STEM FTIC (although students who switched to non-STEM but were still taking the STEM core we not evaluated for the full STEM FTIC). Similar trends were observed for the 2016 cohorts (see Table 7). The 2016 SA lost 6.3% of the cohort from the university compared with 10.0% for the MCG and 12.5% for the 2016 STEM FTIC (not including the 190 SA students). Loss from STEM due to transfer to non–STEM majors was also reduced in the SA (9.5%), compared with MCG (16.3%), and the total FTIC (19.9%). Thus, total STEM retention for the 2016 SA is 84.2% compared with 73.7% for the MCG and 66.6% for the full STEM FTIC (although students who switched to non-STEM but were still taking the STEM core were not evaluated for the full STEM FTIC). Several additional points are noteworthy regarding these results. First, several students from each cohort switched to non–STEM majors but are still taking their STEM core courses. This includes 11 students from the 2015 SA, six from the 2015 MCG, 21 from the 2016 SA, and seven from the 2016 MCG. We count these students as “retained in STEM,” as detailed in the Methods section. Second, the 2015 and 2016 SA programs contributed 0.9% and 1.68% to the overall university retention statistics. This suggests that, with additional scaling, even higher retention rates may be achievable. Finally, the SA program serves a demographic that is highly diverse. Detailed demographic analysis of the 2015 SA cohort shows that 68% of the URM students (Hispanic and Black) are retained in STEM compared with 49% for the MCG. With respect to gender, 84% of SA males and 68% of SA females are retained in STEM compared with 57 and 54%, respectively, for the MCG. Similar trends are observed for the larger 2016 SA cohort, but because this cohort has only 1.5 years of progression analysis, it is too early to determine whether these increases will be sustained.

DISCUSSION

In this report, we present results from the second and third cohorts of the USF-Howard Hughes Medical Institute (HHMI) SA program. One advantage of this report over the original assessment of the 2015 SA program (Findley-Van Nostrand and Pollenz, 2017) is that we expanded the scale of the program to more than 200 participants but still maintained the program impact on key psychosocial measures that contribute to undergraduate retention. The increases in participants’ sense of belonging to both STEM and the university across three different SA cohorts demonstrate a robust finding that is crucial for STEM persistence. These findings validate the SA model as an initial high impact cocurricular program and suggest that the SA scholars begin their undergraduate careers “primed” for success. Therefore, SA students start their STEM trajectories empowered to take advantage of resources and mentoring relationships that contribute to higher retention in both the university and STEM disciplines.

Considerations for Increasing Program Scale

An important factor in scaling the program from 116 to 222 was to maintain the small-group learning environment for each program module. Most of the 18 SA program modules include small groups of 24 students to ensure that the scholars receive an active-learning experience and can engage directly with faculty and graduate student and undergraduate peer mentors. Each group of 24 scholars is mentored by one STEM PhD student and three or four undergraduate peers. This creates a student:mentor ratio of no more than 6:1, allowing for individual interactions, enhanced team building, and group problem solving. We based the choice of 24 scholars on the assumption that adding additional groups of 24 would not negatively influence the community-based outcomes that the program was designed to achieve. In addition, 24 represents the maximum capacity of the teaching labs used for the programming modules. Moreover, we had access to up to six teaching laboratories for the program. Because we split the program on some days and ran the same exercises in the morning and afternoon sessions, our capacity was limited to 12 labs (six in the morning and six in the afternoon). This sets our SA program capacity at 288 students (12 labs × 24 students = 288). There are also several sessions for the full SA cohort; thus, it is critical to assure that facilities are available to support interactive programming that can be delivered to groups of 200–300 students.

Recalibrating First-Year “Pre-med” Students’ Perceptions of the Pathway to Medical School

Although we replicated the majority of the psychosocial gains across the three SA cohorts, one of the most notable differences that emerged from the 2016 and 2017 cohorts, compared with the 2015 cohort, was that pre-med students increased in their intention to leave their STEM major from pre- to posttest. At first glance, this finding appears to be inconsistent with the goal of the SA program and the positive attitude changes observed with belonging and identity. However, deeper analysis of the 2016 and 2017 cohorts shows that the intention to leave STEM was only found for students who described themselves as pre-medical upon entry to the university. So why might this pre-medical population reports increased intention to leave STEM after the short SA program?

When prospective students apply to the SA program, they must complete a categorical question regarding their career choices. The options include 1) attend graduate school and earn a master’s or doctoral (PhD) degree, 2) attend pharmacy/dental other professional school, 3) attend medical school and earn an MD, 4) get a job first and determine my next steps, and 5) not sure at the moment. In any given SA cohort, ∼75% of the scholars choose option 3, and this is consistent with the degree programs selected by the incoming SA scholars. Prospective students who indicate a desire to go into health fields are steered toward the biomedical sciences major by academic advisors, and the majority of the biomedical sciences majors indicate a pre-medical track. Owing to the high number of scholars who desire a medical career, scholars can participate in the optional Mythbusting Medical Admissions workshop on the fifth day of the SA program. This workshop requires attendees to complete a 20-question questionnaire that gauges their current understanding of the medical admissions process, the most important credentials, GPA and MCAT expectations, residency information, and a review of the most recent matriculation data pulled from the American Association of Medical Colleges. In addition, scholars have an opportunity to meet with MDs, DOs, and other health professionals during a career speed-dating session on the last day of the program. Some consistent messages the scholars receive from these sessions are that 1) the pathway to medical school is not always through a STEM degree; 2) it is essential to have a diversified portfolio beyond STEM experiences; 3) creativity and the liberal arts are important in success and in cultivating creative problem solving; and 4) life experience is valued, because the average age of admitted students is increasing. These messages create significant challenges to many of the “pre-medical” SA students who typically rely on anecdotal information about the pathway to medical school and believe that they should follow a linear path through a pre-medical degree program (see commentary from interviews of pre-med students in Lin et al., 2014). The results from the 2016 and 2017 cohorts may be due to a change in the timing of the workshop, which was moved from the first to the fifth day of the SA program. In addition, for the 2016 and 2017 cohorts, the career speed-­dating module was also expanded from one to four practicing physicians (both MD and DO) who further validated the lessons covered in the workshop. For scholars who switched to non–STEM majors but are still pursuing relevant STEM courses, we can track their intentions through their course enrollments.

Mechanisms That Predict Positive STEM Outcomes for SA Students

The current study supports our hypothesis that community and belonging are important contributors to STEM retention (Maton et al., 2016; Tomasko et al., 2016; Ashley et al., 2017; Hanauer et al., 2017). As an exploratory research question, we investigated what mediating mechanisms were associated with SA students’ increase in sense of belonging and science identity from the 2016 and 2017 cohorts. Among the 2016 cohort, these exploratory analyses yielded two significant models: 1) positive changes in university belonging indirectly predict an increase in science identity from when they were surveyed before the SA program compared with after the program; and 2) positive changes in STEM belonging indirectly predict an increase in science identity. These models suggest that the SA increases scholars’ science identity, in part, because the program increases their sense of belonging at the university and to a STEM discipline. The SA affords scholars many opportunities to interact formally and informally with STEM faculty, graduate students, and successful undergraduate peers. Once students start identifying with STEM mentors, this presumably facilitates their own group membership as scientists. Finally, the short duration of the SA program (5.5 days) still facilitates long-term mentoring and peer relationships.

Within the 2017 cohort, a third model also yielded significance: positive changes in perceived family support indirectly predict an increase in university belonging. The third model suggests that one reason students are increasing their sense of university belonging after completing the SA program (at least among the 2017 cohort) is because they experience an increase in perceived family support. Stronger perceived family support may increase students’ sense of university belonging by knowing that their families will continue to support them emotionally or even financially during their time at the university. For example, students who report stronger family support also report stronger feelings of self-efficacy, which predicts academic persistence (Torres and Solberg, 2001). Perceived family support is consistently associated with educational success, particularly among first-generation and ethnic minority college students (Arellano and Padilla, 1996; Solberg and Villarreal, 1997; Dennis et al. 2005). Therefore, perceiving strong family support may be particularly important for students in STEM as they prepare for courses that require a high level of studying and preparation.

Limitations and Future Directions

Despite the strengths and promising findings from the evaluation of three SA cohorts, we note some limitations. First, the 1-week SA program includes a variety of program modules, making it difficult to determine the exact content that impacts psychosocial gains and retention outcomes. Current research efforts seek to identify what program elements yield the strongest impact on student success in order to refine and expand these aspects of the program for future cohorts. To identify the most impactful parts of the program, we took a mixed-methods approach that includes qualitative interviews and additional surveys for the 2015 cohort before their graduation in 2019. Conducting focus group interviews provides nuanced insights into the SA experience that were previously overlooked with survey instruments. While the interviews can identify parts of program students believe to be the most beneficial to their careers, the additional surveys can track their sense of belonging and science identity development. The final goal of the mixed-methods approach is to identify changes to their initial career plans (e.g., changes from medicine) and evaluate their level of cocurricular engagement.

Sampling issues and university construction on campus residences resulted in a smaller 2017 cohort. In addition to having a small cohort, institutional policies prevented us from collecting an MCG. Therefore, we were unable to compare 2017 SA students with a comparable group of STEM students who did not participate in the program. However, we plan to select an MCG for future cohorts to longitudinally track the academic performance and retention of SA students compared with non-SA students. Third, the SA pre- and posttest design does not allow us to draw causal conclusions, because we cannot employ random assignment. Even with the ability to create a, MCG through propensity scoring, we do not have access to every type of student metric that would eliminate all individual differences between the groups. As stated in the Results and Discussion, self-selection biases prevent us from drawing causal conclusions.

CONCLUSIONS

Overall, the findings from three cohorts of SA students (2015, 2016, and 2017) suggest three conclusions. First, even though the SA is only 1 week in duration, it increases STEM retention compared with average rates of STEM retention at a large public university (except among pre-med students, who often change their majors to a non–STEM field while still taking their STEM core courses). Second, students demonstrate much stronger identities as scientists after completion of the program. Third, these positive STEM outcomes appear to be driven by an increase in feelings of belonging to the university and STEM disciplines. Therefore, the brevity of the program need not come at the expense of students’ ability to create long-term connections with faculty members, graduate students, and STEM peers that facilitate success as STEM students throughout the course of their studies.

ACKNOWLEDGMENTS

REFERENCES