ASCB logo LSE Logo

Evaluating Psychosocial Mechanisms Underlying STEM Persistence in Undergraduates: Evidence of Impact from a Six-Day Pre–College Engagement STEM Academy Program

    Published Online:https://doi.org/10.1187/cbe.16-10-0294

    Abstract

    The persistence of undergraduate students in science, technology, engineering, and mathematics (STEM) disciplines is a national issue based on STEM workforce projections. We implemented a weeklong pre–college engagement STEM Academy (SA) program aimed at addressing several areas related to STEM retention. We validated an instrument that was developed based on existing, validated measures and examined several psychosocial constructs related to STEM (science identity, self-efficacy, sense of belonging to the university and to STEM, career expectancies, and intention to leave STEM majors) before and after the program. We also compared students in the SA program with a matched comparison group of first-year students. Results show that SA students significantly increased in science identity and sense of belonging to STEM and to the university, all predictive of increased STEM retention and a primary aim of the program. Relative to the matched comparison group, SA students began their first semester with higher STEM self-efficacy, sense of belonging, and science identity, positive career expectancies, and lower intention to leave STEM. The SA cohort showed 98% first-year retention and 92% STEM major retention. The SA program serves as a model of a scalable, first-level, cocurricular engagement experience to enhance psychosocial factors that impact undergraduate persistence in STEM.

    INTRODUCTION

    The retention of undergraduates who enter science, technology, engineering, and mathematics (STEM) degree programs has been identified as an area of national concern. There is a predicted need for qualified individuals to fill nearly 1 million new STEM jobs over the next 10–20 years (Carnevale et al., 2011; Hossain and Robinson, 2012), sparking national calls to action in advocating reform in STEM education (see, e.g., American Association for the Advancement of Science [AAAS], 2011; President’s Council of Advisors on Science and Technology [PCAST], 2012). For the United States to remain internationally competitive in terms of education and science and technology advancements, it is necessary to increase focus on attracting a diverse body of talent to STEM fields and ensuring that students persist in STEM from the point of initial interest, into college, and subsequently into STEM careers (National Research Council [NRC], 2012). In the current study, we present preliminary findings of a novel cost-effective engagement program for incoming freshman students before the start of their first semester. The initial findings suggest promising results in terms of persistence at the university and in STEM disciplines, as well as in related psychosocial constructs (e.g., sense of belonging, science identity). We also suggest that adopting an interdisciplinary approach to examining persistence that includes psychosocial constructs can benefit understanding of STEM education.

    Efforts to Increase STEM Retention at Universities

    Many universities have developed or adopted programs to improve STEM retention, especially for students who come from underrepresented populations (e.g., Kuh, 2008; Labov et al., 2009; Estrada et al., 2011, 2016; Hernandez et al., 2013; Light and Micari, 2013). These programs vary in their structures, and include short 1-week summer boot camps (e.g., the Biology Intensive Orientation for Students [BIOS] program; Wischusen and Wischusen, 2007; Wheeler and Wischusen, 2014); intensive summer bridge programs that combine courses, mentoring, and research and continue into the students’ first year of study (e.g., the Meyerhoff Scholarship Program; Pender et al., 2010; Stolle-McAllister et al., 2011); cocurricular support and mentorship to students throughout their college careers (e.g., Biology Scholars Program; Matsui et al., 2003), or course-based undergraduate research experiences (CUREs [Auchincloss et al., 2014]; e.g., the Freshman Research Initiative program [Rodenbusch et al., 2016]; or the SEA-PHAGES program [Caruso et al., 2009; Harrison et al., 2011; Hanauer et al., 2016]). Many of these programs emphasize early engagement in undergraduate research, promoting hands-on, inquiry-based experiences that aim to increase “thinking like a scientist,” exposure to the application of course content, and connection with peers as a part of a “community” of scholars (Seymour et al., 2004; Laursen et al., 2010; Estrada et al., 2011; Egan et al., 2013; Hernandez et al., 2013).

    Although many of these programs show gains in STEM persistence, they are often tailored to an institution’s own STEM population needs and may require a significant investment of fiscal and faculty resources to implement. We sought to design a scalable, cost-effective cocurricular program (the STEM Academy, or SA) that did not require curricular changes or immediate undergraduate research but incorporated several practices shown to be effective in enhancing student success, retention, or engagement in long-term classroom or research settings (see Table 1 below). The SA took place across 1 week before the start of students’ first semester. Individual SA program elements (described in more detail in the Supplemental Material) were created based on research findings showing that: hands-on activities predict student motivation and interest in science areas (VanMeter-Adams et al., 2014); mentorship experiences enhance interest and achievement (Pfund et al., 2006; Feldman et al., 2013; Daniels et al., 2016); undergraduate research impacts persistence and engagement (Seymour et al., 2004; Kuh, 2008; Laursen et al., 2010; Linn et al., 2015); reflective writing has benefits across multiple disciplines (Wald and Reis, 2010); and, perhaps most importantly, a sense of belonging/community is critical for integration and success in the university setting (Tinto, 1993; Anderman and Freeman, 2004).

    TABLE 1. Selected high-impact practices in the SA program

    MentorshipStudents are mentored by graduate students and peers throughout all program modules with unstructured time built in for peer socialization and engagement.
    CollaborationStudents work in teams to solve puzzles, complete math problems, discuss action plans, reflect on their experiences, and strategize about careers.
    WritingStudents keep a journal and complete reflective writing exercises to specific prompts each day.
    Hands-on experiencesStudents engage in activities, including analyzing a standard curve assay, in a laboratory.
    Career planningStudents engage and network with professionals across allied health, medicine, business, engineering, and education in several unstructured sessions and participate in career-planning/building workshops.
    ResearchStudents attend professional development workshops and develop an action plan to engage in undergraduate research, tour research labs, and network with research faculty and graduate students.
    Math competenciesStudents complete exercises reflective of what they will experience in their first math courses, including regression analysis of their standard curve assay and precalculus problems.

    We also aimed to extend research on STEM persistence by examining individual-level psychosocial factors that may drive the decision to stay or leave the STEM discipline (Cromley et al., 2016; Estrada et al., 2011; Hernandez et al., 2013; Hanauer et al., 2016). Research in this area is warranted for several reasons. First, research in psychology has long recognized the importance of these and other factors that, directly or indirectly, contribute to achievement and adjustment. Second, pioneering work by Seymour and Hewitt (1997) shows that a significant proportion of students who leave STEM have a high level academic success before leaving the major and that factors other than academic talent are likely to contribute to retention issues. In the Seymour and Hewitt (1997) study, many of the students interviewed indicated that they felt unwanted in the STEM discipline and lacked a sense of belonging or connection to the major and university. Third, recent reports emphasize that educational research on psychosocial constructs is a critically important area to explore, with the potential to enhance understanding of the mechanisms that impact students’ decisions to remain in or leave STEM disciplines (NRC, 2012; Cromley et al., 2016; Estrada et al., 2016). Finally, participation in non–course based activities (e.g., undergraduate research) is related to increased confidence, science identity, and networking (Grunert and Bodner, 2011; Harrison et al., 2011; Ovink and Veazey, 2011). Thus, it is reasonable to expect the SA, as a noncurricular engagement program, to show gains in these important psychosocial measures that may in turn have significant impacts in STEM retention with the added benefit of minimal resource investment relative to some experiences, for example, mentored undergraduate research.

    Psychosocial Factors and STEM Retention

    Of the rich and relevant theoretical frameworks in psychology, this program and evaluation were primarily built from the complementary social cognitive theory and self-determination theory. In response to a call to increase educational research on psychosocial constructs in student experiences (NRC, 2012; Trujillo and Tanner, 2014), we aimed to examine several interrelated factors tied to STEM achievement and retention (self-efficacy, sense of belonging, science identity, and career expectancies in the context of STEM, as well as intention to leave STEM; for a similar approach, see Hanauer and colleagues, 2016).

    Social cognitive theory (Bandura, 1997) emphasizes the role of perceptions of competence and ability and outcome expectations in achievement of personal goals (Bandura, 1982, 1986) as well as outcome expectations, or anticipated consequences of one’s actions (Bandura, 1986). High self-efficacy is related to positive general academic measures (e.g., grade point average [GPA], persistence, and achievement), regardless of prior academic success (Multon et al., 1991; Usher and Pajares, 2008; van der Hoeven Kraft et al., 2011; Komarraju and Nadler, 2013; for a review, see Zimmerman, 2000). Self-efficacy beliefs specific to STEM drive achievement and retention in STEM (Chemers et al., 2001; Estrada et al., 2011; Hernandez et al., 2013; Perez et al., 2014). However, it is important that self-efficacy reflects, at least in part, one’s true ability, as overconfidence may negatively impact performance or undermine perceived effort needed to achieve a task (e.g., Jensen and Moore, 2008). Positive expected outcomes (e.g., obtainment and desirability of employment in STEM) in turn foster self-efficacy, and when specifically examined in terms of science and math, are related to interest and course selection (Lent et al., 1993).

    Because self-efficacy facilitates regulation of one’s own learning and goals, increasing self-efficacy may enhance persistence, especially in the face of challenging STEM course work.

    A sense of belonging, reflecting a feeling of acceptance by or membership in a particular community or group, also helps foster self-efficacy (Bandura, 1997) and is tied to positive academic experiences. From a self-determination theory perspective, belonging (along with competence and autonomy) facilitates intrinsic motivation, which underlies positive educational outcomes. Students fare better when they experience positive social interactions and peer relationships (Fass and Tubman, 2002; Dennis et al., 2005). Other influential models target social integration and belonging at institutions as key to student persistence and success (see Tinto, 1993; Anderman and Freeman, 2004; Strayhorn, 2012). Research on belonging specific to STEM fields finds that a sense of belonging in math is predictive of intention to remain in math (Good et al., 2012), and belonging in physics predicts exam grades and perceived value of the subject (Lewis et al., 2016). Further, belonging may be especially important for women and for ethnic or racial minorities (Hausmann et al., 2007; Good et al., 2012), who are historically underrepresented in STEM disciplines (National Science Foundation, 2006; American Association of University Women, 2010).

    Finally, belonging is closely tied to group identity. Commitment to STEM or science identity is important for understanding motivations and decisions regarding academic careers (Brickhouse et al., 2000) and is linked to greater rates of persistence and lower intentions to leave STEM (Chemers et al., 2011; Perez et al., 2014; Robnett et al., 2015), as well as to career choices (e.g., students who report a strong physics identity are more likely to pursue physics as a career; Hazari et al., 2010). Science identity may be increased by undergraduate research experiences (Aschbacher et al., 2010; Estrada et al., 2011; Hernandez et al., 2013) and is thus likely to also change based on other STEM-based experiences outside of course work. In this study, we expected the interrelated self-efficacy, positive career expectancies, belonging, and science identity to increase based on participation in a 1-week SA engagement program.

    Present Study

    In this study, we introduce the SA program and present data from its pilot cohort year (2015). Specifically, we 1) establish the validity of our survey measures; 2) examine changes in self-efficacy, belonging, career expectancies, science identity, and intention to leave STEM from beginning to end of the program; 3) compare these constructs with a matched comparison group of students beginning their first semester; and 4) determine the first-year persistence rates of SA and matched comparison students.

    METHODS

    SA Program Development

    The SA program is a 1-week intensive-engagement program for incoming STEM students occurring just before the start of their first semester. The SA program consists of 19 different modules that are designed to expose the students to elements of several of the high-impact practices known to increase retention and build community (Kuh, 2008). These include but are not limited to mentorship, collaboration, writing, hands-on exercises, career planning, undergraduate research, and targeting math competencies. Table 1 summarizes these practices and highlights SA student experiences with each. Notably, the majority of SA program activities are delivered in a small-group setting that requires collaboration and the completion of action plans. Thus, even though the SA program occurs over a short period, its structure is designed to build a community of scholars. A detailed SA program guide containing personnel needs, anticipated costs, student applications, management timelines, and program modules can be found in the Supplemental Material. In addition, photos and video can be accessed through the SA website (www.usf.edu/admissions/freshman/stem-academy).

    Participants

    All participants in this project were students at the University of South Florida (USF). USF is located in a metropolitan setting and is classified by Carnegie as very high research. For the purposes of this project, two separate samples were recruited. The first were first-time in college (FTIC) students who participated in the SA program. The second included a non-SA group of FTIC STEM major students. In the following sections, we describe both of the samples. Note, from the larger non-SA group sample (see Non-SA Group), we selected a matched comparison group for analysis of psychosocial measures and retention (procedures for matching are detailed in the respective sections). Thus, the term “non-SA group” refers to the larger sample of non-SA first-year students, whereas the term “matched comparison group” refers to the smaller group selected for comparison. All procedures were approved by the university’s institutional review board (IRB).

    SA Participants.

    All admitted FTIC students who selected STEM College of Arts and Sciences majors (Biology and Chemistry Department majors, Physics, Math, Statistics, and Geosciences) and had a quantitative math Scholastic Aptitude Test (SAT) score < 650 or ACT < 29 were invited to apply to the SA program through a secure online application. Approximately 1200 students were invited to apply, 172 completed applications, and 140 were provisionally accepted (20 students never completed the additional communication requirement and were placed on waitlist). Admission was primarily based on the sophistication of the personal statement, the student’s understanding of the scope of the program, and entry into precalculus as a first math course, as the program was not intended to recruit a population that generally enters into higher-level calculus courses. The SA application and details on the review and admission process are detailed in the Supplemental Material. The first SA cohort was capped at 120 and ultimately included 115 students. Five students never attended the university after accepting admittance to the SA program. Of the 115 students who did enroll, six completed the program but were removed from the study because they entered into non-STEM majors (nursing and public health). Thus, the final N of the SA was 109. Of the SA students, 99 agreed to complete the psychosocial survey measures at T1 (pre survey), and these students also completed the T2 survey (postprogram). There were also an additional eight students who did not complete the pre survey, but who opted to complete the post survey. Thus, only 99 students were included in the pre–post survey analysis, but 107 were included in the comparisons with our matched comparison group (detailed in Procedures for Creating a Matched Comparison Group). Retention data for the entire SA group are presented.

    Non-SA Group.

    Students enrolled in introductory chemistry courses were recruited to complete the same survey as the SA participants for extra credit in the course. Chemistry was selected because it is a requisite course for most STEM majors at the university. A total of 1411 students completed the survey, including 720 FTIC, nonengineering STEM majors (non-FTIC and non-STEM majors were omitted from the analysis). This sample included 273 (37.9%) females; 445 (61.8%) males (two participants [0.3%] elected not to indicate gender); 334 (46.4%) white, Caucasian; 63 (8.8%) Black/African American; 138 (19.2%) Hispanic; and 135 (18.8%) Asian/Pacific Islander students (6.8% were classified as other or elected not to answer).

    Procedures for Creating a Matched Comparison Group.

    To select a matched group via propensity scoring, we used an R plug-in for the Matchit program in SPSS (Ho et al., 2007; Thoemmes, 2012). In this approach, we used a number of covariates that are related to STEM persistence in a logistic regression model to create a propensity score reflecting the probability for one to select into the treatment (i.e., program) condition. For a review and example of methodology underlying and utility of propensity scoring in education programs, see Lane and colleagues (2012) or Rodenbusch and colleagues (2016).

    In our model, we used demographic and achievement data available to us and relevant to STEM persistence, which resulted in inclusion of 10 covariates reflecting aptitude (combined SAT score or ACT equivalent), high school achievement (GPA and number of Advanced Placement [AP] credits earned), semester entered college (Summer or Fall; binary coded), race (four binary-coded variables for white/Caucasian, African American/Black, Hispanic, and Asian/Pacific Islander), gender (male or female; binary coded), and eligibility for the Federal Work Study (FWS) program (yes/no; binary coded). We used a 1:1 nearest-neighbor without replacement matching procedure within a 0.25 caliper (see Rosenbaum and Rubin, 1984; Stuart, 2010; Thoemmes, 2012). This resulted in a matched group of 109 students from the larger non-SA group. (See Supplementary Material Figure S1 for the distributions of propensity scores across matched and unmatched samples; distributions between the matched samples show greater similarity than those in the unmatched samples.) To evaluate balance between the matched comparison group and SA samples, we first examined whether the continuous-score covariates used differed significantly between groups. There were no mean-level differences in the continuous covariate variables of SAT score, number of AP hours, or high school GPA (Fs = 0.13, 0.06, and 0.22, respectively, all nonsignificant). Second, we examined whether there were significant differences between the SA and the matched comparison groups in the distributions of the categorical variables (race, gender, FWS, entry semester). Chi-square analysis indicated that there were no significant differences when comparing program participation with proportion of females/males, white/Caucasian students, Black/African-American students, Hispanic students, Asian/Pacific Islander students, Summer/Fall entry semester, and FWS eligibility (χ2s = 2.08, 0.16, 1.79, 0.24, 1.10, 0.03, 0.00, respectively; ps all nonsignificant).

    Demographics of STEM Majors, SA Participants, and Matched Comparison Group.

    Table 2 shows the demographic (gender and race) and intended major information for the SA cohort, nonengineering STEM FTIC, and matched comparison group. It is noteworthy that the SA cohort had a higher percentage of females than the overall nonengineering STEM population, but was similar in terms of race/ethnicity. Further, the SA group had an average combined SAT score that was 42 points below the nonengineering STEM population. The lower SAT scores were expected, given that the program was targeted to incoming students who were mostly likely going to start their college career in precalculus and had a quantitative SAT score <650. The matched comparison group was nearly identical to the SA group in all demographics, with an average SAT score midway between the SA and full STEM FTIC group. Most students in the SA and matched comparison group were enrolled in biomedical science and biology majors, 83% and 89%, respectively, and this is consistent with the full nonengineering STEM FTIC entering these majors (87.5%). It is noted that both cohorts also contained a small number of students who were enrolled in engineering majors. This was not intended but was due to four admitted SA students who changed their majors to engineering after attending the SA program and having the opportunity to engage with engineering faculty.

    TABLE 2. Demographic information of total STEM FTIC, SA participants, and matched comparison groupa

    TotalMaleFemaleBlackBlack maleBlack femaleHispanicHispanic maleHispanic femaleAsian/Pacific Isl.Asian/Pacific Isl. maleAsian/Pacific Isl. femaleAve Q + M SAT
    STEM FTIC (not including engineering)14075398681363898316128188184851001196
    Percent of total38.361.79.727.972.122.540.559.513.146.254.3
    STEM Academy FTIC1093376111102371614591154
    Percent of total30.369.710.19.190.921.130.469.612.835.764.3
    Matched comparison group FTIC109357491826620176111184
    Percent of total32.167.98.311.188.923.923.176.915.635.364.7

    SA majors: biomedical sciences (62); biology majors (29); chemistry (2); physics (4); health sciences (6); environmental sciences (2); engineering (4). Matched comparison group majors: biomedical sciences (85); biology majors (12); chemistry (8); physics (2); engineering (2). FTIC, first-time in college.

    Measures

    All psychosocial measures included self-report scales rated on a Likert rating system (1–7). Individual anchors of each scale are described in their respective sections. Individual items for each scale may be found in Table 3.

    TABLE 3. Confirmatory factor analysis: factor loadings of the scale items of all measuresa

    MeasureItemFactor loading
    Academic self-efficacy for STEMI feel confident in my ability to learn the material in my STEM courses.0.82
    I am capable of learning the material in my STEM courses.0.86
    I am able to achieve my goals in my STEM courses.0.92
    I feel able to meet the challenge of performing well in my STEM courses.0.90
    Science-task self-efficacyGenerate a research question to answer0.69
    Use scientific literature to guide research0.73
    Create explanations for the results of the study0.83
    Develop theories by integrating results from multiple studies0.82
    Relate results and explanations of a research project to the work of others0.85
    Use scientific language and terminology0.71
    Science identityIn general, my interest in science important part of my self-image.0.76
    My interest in science is an important reflection of who I am.0.77
    I feel like I belong in the field of science.0.87
    I have a strong sense of belonging to the community of scientists.0.90
    I am a scientist.0.67
    Expectancy for STEM careerI would enjoy a career in STEM.0.94
    I have good feelings about a career in STEM.0.91
    Having a STEM career would be interesting.0.94
    I would like to have a career in STEM.0.93
    Belonging to universityI can get what I need at this university.0.63
    This university helps me fulfill my needs.0.69
    I feel like a member of this university.0.83
    I belong at this university.0.82
    I have a say about what goes on at this university.0.70
    People at this university are good at influencing each another.0.67
    I feel connected to this university.0.90
    I have a good bond with others at this university.0.77
    Belonging to STEM
     MembershipI feel that I belong to the STEM community.0.88
    I consider myself a member of the STEM community.0.95
    I feel like I am part of the STEM community.0.96
    I feel a connection with the STEM community.0.90
     AcceptanceI feel like an outsider. (−)0.76
    I feel accepted.0.88
    I feel respected.0.80
    I feel disregarded. (−)0.69
    I feel valued.0.84
    I feel neglected. (−)0.71
    I feel appreciated.0.83
    I feel excluded. (−)0.72
    I feel like I fit in.0.82
    I feel insignificant. (−)0.72
     AffectI feel at ease.0.82
    I feel anxious. (−)0.59
    I feel comfortable.0.86
    I feel tense. (−)0.66
    I feel nervous. (−)0.53
    I feel content.0.78
    I feel calm.0.72
    I feel inadequate. (−)0.67
     Desire to fadeI wish I could fade into the background and not be noticed. (−)0.84
    I try to say as little as possible. (−)0.77
    I enjoy being an active participant. (−)0.62
    I wish I were invisible. (−)0.80
    Intention to leave STEMI am likely to switch to a major that is NOT in a STEM field.0.63
    I am likely to remain in my STEM major through to graduation or completion of my program of study. (−)0.82
    I intend to leave my science major or science related track.0.64
    I do not intend to leave my STEM major before I graduate or complete my program of study. (−)0.71
    I am not going to remain in a major or track in the STEM fields.0.60
    I am going to remain in a major or track in the STEM fields. (−)0.78
    I have considered switching majors to one that is NOT in a STEM field.0.60

    aReverse-scored items are indicated by parenthetical minus signs. Model fit: χ2(1655) = 4356.87; CFI = 0.928; RMSEA = 0.046. All ps < 0.001.

    Self-Efficacy.

    Self-efficacy was evaluated in terms of academic course work and STEM-related tasks. Four items based on the self-determination theory literature (Williams and Deci, 1996; Ryan and Deci, 2000) assessed self-efficacy for course work (e.g., “I feel confident in my ability to learn the material in my STEM courses”; “I am able to achieve my goals in my STEM courses”; seven-point scale from “not at all true” to “very true”). Non-STEM versions of these items have been used reliably in many studies across varying contexts. Self-efficacy for science-related tasks was examined using a six-item scale developed by Chemers and colleagues (2011). For this scale, we prompted students to think about a project that they were involved in or may get involved in, and indicate the extent to which they were confident they could complete several tasks (e.g., “use scientific literature to guide research”; “use scientific language and terminology”; seven-point scale from “not at all confident” to “very confident”). This measure has been used reliably in past research on undergraduates.

    Science Identity.

    Identity as a scientist was examined using five items developed by Chemers and colleagues (2011; e.g., “My interest in science is an important reflection of who I am”; “In general, my interest in science is an important part of my self-image”; seven-point scale from “strongly disagree” to “strongly agree”). This measure has been used reliably in past research.

    Expectancy for Career in STEM.

    Positive expectancies of self in a career in STEM were examined using four items from a measure by Stake and Mares (2001). Items were partially modified to be specific to “STEM” rather than “science.” Participants read the prompt “Please think about yourself and rate how true the following statements are” and then indicated the degree to which they agreed with each statement (“not at all true” to “very true”; e.g., “I would enjoy a career in STEM”). The original science context of this measure has been used reliably.

    Sense of Belonging to University.

    Participants completed the eight-item sense of community scale, adapted from Peterson et al. (2008), to reflect the university context. That is, participants read the prompt “Please think about yourself as a member of this university, and rate your level of agreement with each statement. If you are new to the university, please use your initial feelings,” and then rated their level of agreement with statements such as “I belong at this university” and “I can get what I need at this university” (seven-point scale from “strongly disagree” to “strongly agree”).

    Sense of Belonging to STEM.

    Belongingness to STEM was measured using an adapted version of a 30-item scale developed by Good and colleagues (2012). The original scale was modified only in that the original context of math settings/courses was edited to reflect “STEM” settings/courses. This scale includes several subscales (all measured on a seven-point scale from “strongly disagree” to “strongly agree”) assessing belongingness in terms of membership (e.g., “I consider myself a member of the STEM community”), acceptance (e.g., “Regarding the STEM community, I feel respected”), affect (e.g., “Regarding the STEM community, I feel anxious”), desire to fade (e.g., “Regarding the STEM community, I try to say as little as possible”), and trust in instructors (e.g., “I trust my instructors to be committed to helping me learn”). Given that this project entails a precollege program, we did not presently analyze the trust in instructors subscale, as these items assume that students have begun their course work. Note, “desire to fade” items were reverse scored in order to be included in the composite sense of belonging scale (i.e., low scores indicated greater sense of belonging); that is, higher scores indicated lower desire to fade (and higher belonging).

    Participants first read the prompt: “Today we have some questions we would like you to answer about your experience in the STEM community at this university. When we mention the STEM community, we are referring to the broad group of people involved in these fields, including the students in your STEM courses. Given this broad definition of the STEM community, please respond to the following statements based on how you feel about your membership in it. There are no right or wrong answers to any of these statements; we are interested in your honest reactions and opinions. Please read each statement, and indicate the number that reflects your degree of agreement.” Then, before the membership subscale items, participants read “When I am in a STEM setting at (university)…,” and before the other subscale items “Regarding the STEM community, please indicate your degree of agreement with the following…,” and then rated their level of agreement with each statement (“strongly disagree” to “strongly agree”; seven-point scale).

    Intention to Leave STEM.

    Participants rated their level of agreement with seven items created by Perez et al. (2014; e.g., “I am likely to switch to a major that is NOT in the STEM fields”). Item responses ranged from “strongly disagree” to “strongly agree” and were both positively and negatively worded.

    Survey Data Collection

    The survey was administered through Qualtrics. SA students were introduced to the survey through email communication, during their USF orientation sessions (June and July), and through postings to the SA Canvas organization. SA students completed the pre survey 4–8 weeks before the start of the SA. Post surveys were completed on the last day of the SA program before the start of the semester. The non-SA group was introduced to the survey 2 weeks into the semester and could complete it over a 2-week window.

    STEM Retention

    All students entering the university as FTIC in Summer/Fall 2015 were evaluated for declared major at matriculation. STEM majors included biomedical sciences, cellular and molecular biology (Biology Department), chemistry/biochemistry, environmental biology (Biology Department), environmental microbiology (Biology Department), environmental sciences, geology/geosciences, integrated animal biology (Biology Department), Marine Biology (Biology Department), Mathematics, Microbiology (Biology Department), physics, statistics, all engineering majors, and management information systems. The 2015 cohort was revaluated in October 2016 for enrollment and declared major. Students who were not enrolled in Fall 2016 were scored as “loss from the university.” Students changing out of one of the listed STEM majors to a non-STEM major were scored as “loss to non-STEM major.” Students who changed major and also did not re-enroll were only scored as “loss from the university.”

    RESULTS

    Psychometric Properties of the Psychosocial Measures

    We completed a three-part validation of the measures used for our instrument. Development and selection of measures was driven by the theoretical foundation described earlier and followed a careful review of the literature. The constructs focused on in this study reflect the primary aims of the SA program. To assess the psychometric properties of our measures, we conducted a three-part analysis in which we 1) examined the factorial validity of the measures using confirmatory factor analysis (CFA), 2) examined the internal consistency (reliability) of the respective measures, and 3) examined whether the scale variables evidence criterion validity via significant associations with measures we expect them to correlate with, including intention to leave STEM. Given that the measures used in the present study were based on preestablished, validated, and reliable published measures, we expected our approach to sufficiently address our aims (for a similar strategy used to validate a new survey measure of networking in undergraduate research experiences, see Hanauer and Hatfull, 2015). To have the statistical power (i.e., with a large enough sample) to conduct a factor analysis, we examined psychometric properties of the measures using data combined from the initial SA (pre survey) and entire non-SA group samples. Note, comparisons between SA and non-SA students on the measures and on retention were conducted using only the matched comparison group (see Results).

    CFAs.

    We used Mplus, version 7.4 (Muthén and Muthén, 1998), to conduct CFA on the survey variables. Model fit was evaluated using comparative fit index (CFI; acceptable model fit = 0.90 or above; good model fit = 0.95 or above), and root-mean-square error of approximation (RMSEA; good model fit =.05 or below, with 0.07 as an upper limit of acceptability; see Hu and Bentler, 1999). We also report the chi-squared test of significance; however, due to the impact of sample size on chi-squared tests, the CFI and RMSEA criteria are preferable to evaluate model fit.

    A model in which the items from each measure were set to load on their respective latent factor was estimated. This model included 10 latent factors based on the expected structure of the data (factors include academic self-efficacy, task-specific self-efficacy, science identity, expectancies for STEM career, belonging to university, intention to leave STEM, and the belonging to STEM subscales of membership, acceptance, affect, and desire to fade). Modification indices in Mplus suggest that model fit would be significantly improved by adding paths depicting several correlated residuals between items. These simply reflect that a portion of the error terms between specific items are related, and are frequently added to CFA models of self-report survey data. Notably, all modification indices suggested adding correlated residuals between items within the same scales, but not across scales. A summary of these paths is as follows: paths were added between science identity items 1 and 2; university belonging items 1 and 2, 3 and 4; STEM belonging acceptance subscale items 5 and 7, 6 and 8, 4 and 6, and 4 and 8; and STEM belonging affect subscale items 2 and 4, 2 and 5, and 4 and 5. This model fit the data well. See Table 3 for factor loadings and model fit.

    A second CFA was conducted to examine whether the sense of belonging subscales (membership, acceptance, affect, and desire to fade) loaded onto a higher second-order factor of sense of belonging (in line with Good et al., 2012). In this model, all individual items again loaded onto factors reflecting their respective subscales with factor loadings of 0.59 or higher. Further, the four subscales loaded onto the higher-order factor with high factor loadings (membership = 0.78; acceptance = 0.97; affect = 0.87; desire to fade = 0.63). Finally, this model also fit the data acceptably, χ2(288) = 1429.41; CFI = 0.932; RMSEA = 0.07. Thus, the data suggest that it is appropriate to use either the four subscales or a composite score for sense of belonging..

    Reliability of the Survey Measures.

    Cronbach’s alpha was used to evaluate the internal consistency (reliability) of the survey measures. We first assessed alpha in the overall sample (combining across SA and all non-SA group participants). We also tested the reliability of survey measures in a matched comparison group (the procedures for selecting the matched group are outlined in the Methods section). All measures were sufficiently reliable, with alphas higher than the widely used 0.70 cutoff criteria. See Table 4 for alpha coefficients for the combined sample and for alphas for the SA and matched comparison group separately.

    TABLE 4. Cronbach’s alpha of measures

    Cronbach's alpha
    MeasureTotal sampleSA (pre)SA (post)Non-SA group (all)Comparison group (matched)
    Academic self-efficacy0.930.910.940.920.94
    Science-task self-efficacy0.890.910.840.890.91
    Science identity0.900.830.830.900.91
    Expectancy for STEM career0.960.950.950.960.97
    Belonging to university0.910.890.890.910.91
    Belonging to STEM0.960.960.930.960.96
     Membership0.960.950.970.960.96
     Acceptance0.940.960.920.940.94
     Affect0.900.850.890.900.91
     Desire to fade0.840.870.790.840.86
    Intention to leave STEM0.850.780.850.850.81

    Criterion Validity: Associations among the Variables.

    Table 5 shows the bivariate correlations among the survey variables. For the purposes of examining the psychometric properties of the survey measures, and because associations were not expected to differ significantly between the SA and control group, the correlations are presented as combined across the samples. Overall, the correlations were consistent with expectations and support the criterion validity of the scales. In particular, intention to leave STEM majors was significantly associated with all psychosocial variables in the expected directions. Intention to leave STEM was negatively related to academic and task-specific self-efficacy, science identity, positive expectancies for a career in STEM, and belonging to STEM (overall and all subscales) and the university community.

    TABLE 5. Bivariate correlations among the survey variables

    Measuresa1234566a6b6c6d7
    1. SE-Acad
    2. SE-task0.54
    3. Sci ident0.480.47
    4. Expect car0.430.360.50
    5. Belong univ0.330.330.370.31
    6. Belong STEM0.500.470.460.400.61
     6a. Mem0.490.430.590.470.550.77
     6b. Accept0.390.400.360.310.600.930.68
     6c. Affect0.440.390.370.290.450.870.570.73
     6d. Fade0.360.380.300.310.410.690.430.570.51
    7. Leave−0.33−0.25−0.39−0.50−0.18−0.34−0.39−0.27−0.24−0.31

    a1. Academic self-efficacy in STEM; 2. STEM task-related self-efficacy; 3. science identity; 4. positive expectancy for STEM career; 5. belonging to the university; 6. belonging to STEM community; 6a. belonging to STEM community (membership subscale); 6b. belonging to STEM community (acceptance subscale); 6c. belonging to STEM community (affect subscale); 6d. belonging to STEM community (desire to fade subscale; reverse-scored); 7. intention to leave STEM major.

    All correlations significant at the p < 0.01 level.

    Changes from Pre- to Post-SA

    To determine whether the variables of interest changed across the duration of the SA program, we assessed mean-level changes in the survey variables from pre- to post-SA by conducting paired t tests. Only the 99 students who participated in both the pre and post surveys were included in this portion of the analyses. The results are shown in Table 6. SA students exhibited increased belonging to the university, belonging to STEM (overall and across all subscales), and science identity across the duration of 1 week. Interestingly, self-efficacy for science tasks significantly decreased from beginning to end of the program. There were no differences in self-efficacy for academic courses, expectancies for STEM careers, or intention to leave STEM.

    TABLE 6. Mean-level differences in the study variables before and after STEM academy program

    PrePost
    MeasuresaMSDMSDtp
    1. SE-Acad6.330.676.250.751.07ns
    2. SE-task5.870.905.690.742.19<0.05
    3. Sci ident5.920.826.080.78−2.16<0.05
    4. Expect car6.600.636.590.610.13ns
    5. Belong univ5.910.786.220.68−4.12<0.001
    6. Belong STEM5.500.855.840.71−4.65<0.001
     6a. Mem5.681.006.230.86−5.24<0.001
     6b. Accept5.660.996.170.77−5.44<0.001
     6c. Affect4.980.975.281.16−3.01<0.01
     6d. Fade5.901.086.101.02−2.11<0.05
    7. Leave1.941.331.931.040.09ns

    a1. Academic self-efficacy in STEM; 2. STEM task-related self-efficacy; 3. science identity; 4. positive expectancy for STEM career; 5. belonging to the university; 6. belonging to STEM community; 6a. belonging to STEM community (membership subscale); 6b. belonging to STEM community (acceptance subscale); 6c. belonging to STEM community (affect subscale); 6d. belonging to STEM community (desire to fade subscale; reverse-scored); 7. intention to leave STEM major.

    Because there was no change in academic self-efficacy, and there was a negative change in self-efficacy for science tasks, we tested whether changes from pre to post differed depending on whether the participants entered with an already high level of academic self-efficacy. Because the sample size was low, we were unlikely to have sufficient statistical power to conduct true moderation tests; therefore, we simply classified participants as high, average, or low in academic self-efficacy by mean-centering the variable and then separating based on ±1 SD. Using this approach, we found that both measures of self-efficacy decreased only for those who scored high in academic self-efficacy in the preprogram survey (t = 4.21, p < 0.01; t = 2.31, p < 0.05).

    Differences between SA Participants and Matched Comparison Group

    Because students in the SA program showed increases in important measures such as belonging to the university, belonging to STEM (overall and across all subscales), and science identity, it was pertinent to determine whether the gains were also evident when the SA students were compared with a cohort of students who had not participated in the SA program.

    To examine whether SA students differed from the matched comparison group at the start of their first year, we conducted one-way analysis of variance (ANOVA) to test for differences in the study variables between the matched comparison group (who completed the identity survey within the first 4 weeks of entering the university) and the post scores of the SA participants (who completed the survey several days before entering their first classes). Thus, this comparison should approximate the levels of the constructs that students had at the beginning of their first semester in college. Note: SA participants’ average scores on the post survey differ somewhat from the previous analysis of changes from pre- to postprogram due to the additional eight students who participated in the post survey (but not the pre survey). When compared with the matched comparison group, SA students scored significantly higher in academic self-efficacy, science identity, positive expectancies for STEM career, belonging to university, and belonging to STEM (overall and in each subscale), and significantly lower in intention to leave STEM (Table 7).

    TABLE 7. Mean-level differences between STEM academy and matched comparison group at start of first semester

    STEM AcademyMatched comparison group
    MeasuresaMSDMSDFp
    1. SE-Acad6.220.855.870.968.04<0.01
    2. SE-task5.670.765.570.960.81ns
    3. Sci ident6.090.805.481.1420.92<0.001
    4. Expect car6.600.606.231.089.42<0.01
    5. Belong univ.6.200.685.760.8617.25<0.001
    6. Belong STEM5.830.765.070.9343.44<0.001
     6a. Mem6.240.895.091.2261.62<0.001
     6b. Accept6.150.825.221.0552.69<0.001
     6c. Affect5.271.204.651.1615.02<0.001
     6d. Fade6.071.025.571.1311.84<0.001
    7. Leave1.931.042.351.435.91<0.05

    a1. Academic self-efficacy in STEM; 2. STEM task-related self-efficacy; 3. science identity; 4. positive expectancy for STEM career; 5. belonging to the university; 6. belonging to STEM community; 6a. belonging to STEM community (membership subscale); 6b. belonging to STEM community (acceptance subscale); 6c. belonging to STEM community (affect subscale); 6d. belonging to STEM community (desire to fade subscale; reverse-scored); 7. intention to leave STEM major.

    Although we observed significant differences between the SA (post survey) and matched comparison group, we also noted that there were several unexpected differences between the matched comparison group and SA pre survey scores. Specifically, SA students scored higher in their pre scores of academic self-efficacy (F = 15.77, p < 0.001), self-efficacy for science tasks (F = 5.19, p < 0.05), science identity (F = 9.96, p < 0.01), positive expectancies for STEM career (F = 7.02, p < 0.01), and belonging in STEM (F = 11.68, p < 0.01) and lower in intention to leave STEM (F = 4.70, p < 0.05) than matched comparison group students. There was no difference in belonging to university. Thus, there are clearly unobserved characteristics in the SA group that were not captured by the covariates used in the propensity-scoring procedure but may have impacted selection into the program (even though variables such as high school GPA and number of math and STEM courses were used).

    Retention after the First Year

    The final analysis was to evaluate STEM retention of the SA and matched comparison group in the context of the full FTIC STEM cohort. Two factors contribute to STEM retention: 1) loss of a STEM student from the university, and 2) students who persist at the university but change from a STEM to non-STEM major. Table 8 shows the results for the SA, matched comparison group, and full 2015 STEM FTIC evaluated for enrollment in Fall 2016 (first-year retention after completion of the Fall 2016 drop/add period) and current major. In addition, results are presented for the specific STEM majors within each cohort. The SA lost 1.8% of the cohort from the university compared with 4.6% for the matched comparison group and 9.5% from the 2015 STEM FTIC (not including the 109 SA students). Loss from STEM due to transfer to non-STEM majors was also reduced in the SA (6.4%) compared with the matched comparison group (8.3%) and the total FTIC (8.0%). Thus, total STEM retention for the SA was 91.7% compared with 87.2% for the matched control and 82.5% for the full STEM FTIC. Increased retention in the SA cohort in comparison with the matched comparison group is also reflected in the biomedical science (BMS) major, which has the highest number of incoming STEM students to the university. Thus, when the SA cohort is included in the total STEM FTIC, overall STEM retention at the university increased 0.5%. It is noteworthy that a portion of the SA programming discusses how students can obtain STEM competencies while enrolled in a non-STEM majors. To assess the SA retention results in more detail, we evaluated the transcripts from students in the SA and matched comparison group who left the STEM majors but were still enrolled at the university. Of the seven students in the SA who switched to non-STEM majors, five were still taking core STEM courses compared with only one of nine in the matched comparison group. Thus, 96% of the SA cohort was still involved in taking STEM core courses compared with 89% of the matched comparison group.

    Table 8. Retention data for STEM Academy, matched comparison group, and FTIC STEM cohorta

    aSTEM FTIC (w/o SA) = all entering STEM majors without the 109 SA students. BMS = biomedical science majors; BIO = Department of Biology majors; other STEM majors = chemistry, physics, math, and geosciences majors. The last row: total STEM FTIC is equal to the 109 SA cohort + 2015 total STEM FTIC (w/o SA) cohort. Total heads lost from STEM = heads lost to non-STEM major changes + heads lost from the university. STEM retention = total heads lost from STEM/total heads entering in 2015.

    DISCUSSION

    In this study, we have introduced the early engagement SA program and examined psychosocial constructs significant to STEM persistence. The results suggest that the SA may be a promising cocurricular initiative for increasing measures of STEM retention at the undergraduate level that may be scaled to different universities and serve as a “first-pass” program to more intensive semester-long CURE initiatives. Further, results support the notion that considering individual-level and social factors of STEM persistence heightens understanding of a program’s effectiveness and may give educators a better idea of the areas that should be targets of reform efforts. In the following sections, we discuss our results first in terms of the efficacy of the program (survey results and first-year retention) and then consider our analysis of psychosocial constructs, as well as practical considerations, limitations, and future directions.

    Efficacy of the SA Program: Psychosocial Constructs

    As a whole, the first cohort of the SA program showed promising results in terms of psychosocial underpinnings of STEM success. Now, we consider results from the pre–post changes and comparisons with the matched comparison group separately for each construct.

    SA participants increased in their sense of belonging to the university and to the STEM community across the duration of 1 week. This was true in terms of overall sense of belonging to STEM, as well as in several subscales: participants reported greater feelings of acceptance and membership, more positive emotions in the STEM context, and less desire to fade at the end of the program relative to the beginning. Also, relative to a matched comparison group, participants began their first semester in college with a greater sense of belonging to the university and to STEM (again, across all subscales). Thus, the data suggest preliminary support for the capacity of the SA program to build community and peer connections, identified as fundamental reasons for persisting in science disciplines both during college and into careers. This is especially true for women and underrepresented minorities, who often experience both direct and subtle hostility and bias from their majority counterparts (Seymour and Hewitt, 1997; Estrada et al., 2011; Hernandez et al., 2013).

    Considerable psychological research supports the need to heighten sense of belonging in STEM disciplines. For instance, work in goal congruity theory (see Diekman et al., 2015) suggests that individuals (most often women) who hold communal values are more likely to leave STEM, as they perceive these fields to be discrepant with their values. As feelings of belonging correspond with communal values, targeting this construct is highly important. Thus, the increase in belonging in SA students is perhaps the single most encouraging finding from this study. However, in addition to the increased belonging in STEM disciplines, SA students also showed increase in belonging to the university; thus, students in the SA appear primed to participate in other engagement initiatives and use resources (regardless of ultimate major), to have more positive overall university experiences.

    Participants also increased in science identity from beginning to end of the SA. Broadly speaking, identity and self-concepts guide motivation and behavior in a variety of contexts. Thus, science identity likely impacts the degree to which one is willing to pursue and persist in STEM disciplines, even in the face of potential challenges or hardships. Previous research has found that commitment to a science identity is predictive of persistence at various levels of STEM education (e.g., Aschbacher et al., 2010; Chemers et al., 2011; Perez et al., 2014; Robnett et al., 2015) and that science identity may be increased via undergraduate experiences in research (Charney et al., 2007). In this study, immersing students in settings that foster peer support and provide hands-on engagement opportunities and rigorous training in how to engage in undergraduate research in subsequent semesters likely facilitated greater identification with the values of a scientist. However, future research should disentangle which portions of the program may have driven the increases found.

    Existing programs that include undergraduate research experiences/CUREs or hands-on activities (e.g., Charney et al., 2007; VanMeter-Adams et al., 2014; Wheeler and Wischusen, 2014) find meaningful effects on psychological and motivational constructs. However, our results suggest that similar goals may be achieved even before the students gain valuable research experience at the university, and this is beneficial for students who will enter into a longer-term program. Indeed, fostering feelings of belonging by engaging students in a peer group and with supportive mentors and staff likely helps prepare the students for the challenging experiences they will face (including undergraduate research) within the context of the STEM curriculum.

    It is important to note that some of the results were more challenging to interpret but still shed light on students’ experiences in the SA program. First, academic self-efficacy did not increase from beginning to end of the program, but participants in the SA scored higher than those in the matched comparison group. Further, self-efficacy for science tasks decreased across the duration of the program, and SA participants did not differ from the matched comparison group in this variable. The decrease in science-task self-efficacy and lack of change in academic self-efficacy were unexpected, given that an aim of the program was to build students’ self-confidence for STEM-related course work and tasks. However, this may simply reflect an imposed sense of realism regarding the rigor necessary in STEM contexts, as students sometimes lack understanding of the effort required in a university setting (Ozga and Sukhnandan, 1998). Our exploratory analysis supports this idea: self-efficacy decreased only for those students who entered with very high levels. While perceived competence bolsters achievement (van der Hoeven Kraft et al., 2011), an inflated sense of competence can undermine effort exerted to complete tasks (Usher and Pajares, 2008). Relatedly, researchers also found that the BIOS program (a “biology boot camp” of curricular exposure for first-year students), may decrease participants’ self-efficacy (considered evidence of positive impact; Wischusen and Wischusen, 2007; Wheeler and Wischusen, 2014).

    Even with this explanation, the importance of modifying self-efficacy should not be downplayed, because a large body of evidence supports its importance in fostering self-regulated learning, academic achievement, and overall well-being (Zimmerman, 2000). As we presently consider self-efficacy as especially important to success in challenging STEM fields, some modifications to SA content may lead to more favorable outcomes. For instance, given some elements, like lab settings where students are required to solve several precalculus problems or workshops designed to correct misperceptions regarding medical school admissions, it may be possible to achieve the learning goals of these exercises in a more positive and empowering manner. The analysis of the 2016 and upcoming cohorts will hopefully provide additional insight into this area.

    Career expectancies in STEM did not change from beginning to end of the program, but SA students reported more positive expectancies than the matched comparison group. The end session of the SA program highlights aspects of a variety of STEM and health careers (e.g., a “career speed-dating” session in which students interacted with professionals in various disciplines). However, the majority of the SA focused on experiences at the college level. Thus, it is not entirely surprising that expectancies of future careers did not seem to be affected, as these are more distal in students’ thoughts about their trajectories. Regarding intention to leave STEM majors, results were similar: SA participants did not change from beginning to end of the program, but scored more favorably than their matched comparison group counterparts. This likely reflects a floor effect in which students entering the SA began with such low levels that there was little room for change (although any variation should be captured by the analyses used). Importantly, intention to remain or leave STEM majors will ultimately be evidenced in actual persistence rates and longitudinal follow-ups later in students’ college experience. However, the analysis of first-year retention showed that the SA students were retained at higher levels both to the university and in STEM relative to the general STEM population and matched comparison group.

    In summary, from the results presented, we conclude that, relative to the matched comparison group, SA students started their first year with a more favorable mind-set that should foster success in STEM and further engagement with the university community. This is presented with the caveat that evaluation of the SA scores on the pre survey also suggest that the SA students may have begun their semester more favorably even if they had not participated in the SA. It will be of interest to see whether this trend continues when the 2016 cohort of 222 students is evaluated and the total number of participants increases. Nonetheless, the SA did produce measurable changes to key attributes associated with STEM retention, and the difference in SA participants relative to the matched comparison group was amplified.

    First-Year Retention and the SA Model

    It is reported that only 40–50% of undergraduates who begin in a STEM degree program graduate with a STEM degree (e.g., AAAS, 2011; PCAST, 2012). Loss from STEM includes loss from the university and also students who start in STEM but change majors and graduate from non–STEM degree programs. The time frame of actual STEM loss is not readily available in the literature, but it is clear that the first year is critical when targeting overall persistence to the university (Cuseo, 2003; Hossain and Robinson, 2012). First-year retention is a key metric for many universities, although most do not publicly report STEM retention specifically. Outcomes of retention and graduation with a STEM degree, as the ultimate goals of the SA program, cannot be evaluated for several years. Yet evaluation of first-year retention of the cohort reveals evidence of program effectiveness in retaining students. During their first year, SA students were retained in STEM majors and left the university at a rate much lower relative to the matched comparison group or total STEM FTIC (91.7%, 87.2%, and 82.5%, respectively). Although the SA cohort was only 109 students, the impact of the population on university retention was 0.5%. This may appear to be a small percentage increase, but it can represent a significant number of students at a large public university. In addition, 100% of the SA students from underrepresented groups were retained and 94% remained in STEM disciplines (full demographic analysis of the SA will be the focus of future work). We consider these results to be very encouraging moving forward, and as the program increases in size, and we anticipate obtaining follow-up information from current students regarding their academic progression.

    Cost and Scalability

    We believe that the current study provides preliminary support for the SA model as an effective, entry-level engagement program to increase STEM persistence at the university. Because financial resources for education reform are often limited, the SA program costs were manageable, at approximately $390 per student including early move-in fees, or $260 if fees for housing and meals were deferred. The other SA costs went toward graduate student mentor assistantships ($3000 per mentor), peer mentor stipends ($450 per mentor), student supplies (e.g., SA-branded lab coat, lab supplies, resource notebook, and other expenses), and an opening reception (keynote speaker, meal). These costs were fully subsidized by the HHMI grant; SA students did not occur any costs to participate in the program. Notably, these costs will differ depending on the university, what program elements are included, and what staffing (and associated salaries) are required for implementation of the program (for the program described here, there were no costs associated with faculty or the SA director). These general values are provided simply to offer context to the scalability of the program and for comparison with other longer-term curricular interventions.

    Integrating Psychology into Understanding STEM Persistence

    We consider the interdisciplinary nature of this study to be a strength. Just as teaching in the hard sciences is informed by educational research, educational research in STEM contexts can and should be informed by psychological theory, studies, and methods. As outlined by Seymour and Hewitt (1997), students do not persist due to a variety of factors outside of competence and talent; however, these factors have only recently been considered in the STEM education literature. This study extends existing research on STEM education by directly assessing several relevant psychosocial correlates of academic and STEM-specific success. Our program and evaluation were framed in terms of prominent theoretical frameworks reviewed in the introduction. Accordingly, this study provides a model in terms of theory and method for future programs to consider adopting when attempting to understand precisely how the program affects students. In this study, we used a survey instrument that included partially modified preexisting and validated scales. This approach ensures the validity and reliability of constructs measured (as opposed to using piecemeal, single items or assuming effects on students). In fact, our analysis demonstrated reliability of the scales, construct validity (via a CFA), and criterion validity (via correlations among the study variables). Each of the psychosocial constructs was correlated with intention to leave STEM in the expected directions. One implication of this result is that educators with limited access to confidential student data may benefit by conducting time-efficient survey research in which students report their intentions (assuming that there is little motivation to inaccurately report intent to remain or leave majors). Finally, it is important to note that, although we present the constructs as correlates of STEM success, the developmental interplay of the constructs is more complex in nature and should be evaluated with longitudinal research. For instance, a sense of belonging may directly increase science identity, which in turn may drive motivation and persistence. Incorporating sophisticated longitudinal methodology from the psychological sciences allows for more detailed understanding of directional associations of multiple constructs, and as we obtain additional data from other cohorts, we will begin to assess such relationships (e.g., Robnett et al., 2015).

    Limitations and Future Directions

    Despite the strengths and promising findings in our study, there were some limitations. First, a randomized control trial was not possible in the present context; thus, we were not able to establish true causal effects of the program. Although the study used a matched comparison group, allowing for stronger claims or inferences to be made, it does not fully eliminate the self-selection bias of the students in the program. However, this issue may lead to areas of future inquiry. For instance, it will be interesting to determine what traits drive students to self-select into such programs. As we controlled for high school achievement (GPA, number of AP courses, and SAT scores), and showed that the SA students’ average SAT score was lower than the general STEM student population, academic competence alone does not explain the self-selection. Rather, there are likely other personality or motivational characteristics that should be evaluated (and potentially targeted). These unmeasured characteristics may also partially explain the differences found between the matched comparison group and preprogram levels of the psychosocial constructs. Additionally, future programs could consider attempting to include a covariate that taps into prior participation in extracurricular activities (during high school), as students with previous experience may have been more inclined to apply to the program. Another potential issue with the matched comparison group is that these participants completed the survey once their semester had already begun (vs. 3 days before the start of the semester in the case of the SA students). Experiencing some college may have affected findings. It may be possible to address this in future cohorts of the program by attempting to solicit responses from a comparison group before the start of the semester.

    Second, the 1-week SA program entailed many components, making it difficult to disentangle which portion(s) may have affected the psychosocial and retention outcomes. For instance, the mentorship and group work likely influenced sense of belonging, whereas the hands-on lab experiences may have been more related to self-efficacy. Future research could conduct more controlled, small experiments to evaluate each component separately. Further, more detailed comparisons in future studies with larger sample sizes may allow for greater understanding of differential program impact based on social identity characteristics of student gender and ethnicity (e.g., Eddy et al., 2015).

    In summary, this report reports the implementation of the SA program and our findings through 1 year with the first cohort; there are several anticipated works to come in the future, such as longitudinal follow-up of the first cohort of SA students including STEM persistence and graduation rates. In addition, we have collected survey data from nearly 200 students in the 2016 SA program and more than 1000 comparison group responses to determine whether the results reported here are also observed in this larger group. As our data sets grow, we anticipate using modeling techniques (e.g., longitudinal multigroup structural equation modeling) to examine the interplay of program participation, psychosocial factors, and retention outcomes across students’ time in college.

    ACKNOWLEDGMENTS

    This project was supported in part by an undergraduate science education grant from the Howard Hughes Medical Institute #52008123 (RSP Director). The project is fully approved by USF IRB #Pro00022157 (recertified 2016). We thank the following collaborators for their assistance with this project: USF Office of Decision Support, USF Admissions, USF Residential Life, and University Community Marketing; Robert Potter for review and critical comments; David Rabson and Cynthia Brown Hernandez for assistance with data mining; and all SA graduate students, peer mentors, faculty, and staff who contributed to the various program elements and were committed to helping change the lives of those involved in the program.

    REFERENCES

  • American Association for the Advancement of Science (2011). Vision and Change in Undergraduate Biology Education a Call to Action, Washington, DC. Google Scholar
  • American Association of University Women (2010). Why So Few? Women in Science, Technology, Engineering, and Mathematics, Washington, DC. Google Scholar
  • Anderman LH, Freeman T (2004, Ed. ML MaehrPR Pintrich, Students’ sense of belonging in school In: Advances in Motivation and Achievement, vol. 13, Motivating Students, Improving Schools: The Legacy of Carol Midgley, Greenwich, CT: Elsevier, 27-63. Google Scholar
  • Aschbacher P, Li E, Roth EJ (2010). Is science me? High school students’ identities, participation, and aspiration in science, engineering, and medicine. J Res Sci Teach 47, 564-582. Google Scholar
  • Auchincloss LC, Laursen SL, Branchaw JL, Eagan K, Graham M, Hanauer DI, Lawrie G, McLinn CM, Pelaez N, Rowland S, et al. (2014). Assessment of course-based undergraduate research experiences: a meeting report. CBE Life Sci Educ 13, 29-40. LinkGoogle Scholar
  • Bandura A (1982). Self-efficacy mechanism in human agency. Am Psychol 37, 122-147. Google Scholar
  • Bandura A (1986). The explanatory and predictive scope of self-efficacy theory. J Soc Clin Psychol 4, 359-373. Google Scholar
  • Bandura A (1997). Self-Efficacy: The Exercise of Control, New York: Freeman. Google Scholar
  • Brickhouse NW, Lowery P, Schultz K (2000). What kind of a girl does science? The construction of school science identities. J Res Sci Teach 37, 441-458. Google Scholar
  • Carnevale AP, Smith N, Melton M (2011). STEM, Washington, DC: Georgetown Center on Education and Workforce. Google Scholar
  • Caruso SM, Sandoz J, Kelsey J (2009). Non-STEM undergraduates become enthusiastic phage-hunters. CBE Life Sci Educ 8, 278-282. LinkGoogle Scholar
  • Charney J, Hmelo-Silver CE, Sofer W, Neigeborn L, Coletta S, Nemeroff M (2007). Cognitive apprenticeship in science through immersion in laboratory practices. Int J Sci Educ 29, 195-213. Google Scholar
  • Chemers MM, Hu L, Garcia BF (2001). Academic self-efficacy and first year college student performance and adjustment. J Educ Psychol 93, 55-64. Google Scholar
  • Chemers MM, Zurbriggen EL, Syed M, Goza BK, Bearman S (2011). The role of efficacy and identity in science career commitment among underrepresented minority students. J Soc Iss 67, 469-491. Google Scholar
  • Cromley JG, Perez T, Kaplan A (2016). Undergraduate STEM achievement and retention: cognitive, motivational, and institutional factors and solutions. Policy Insights Behav Brain Sci 3, 4-11. Google Scholar
  • Cuseo JB (2003, Ed. G Kramer, Comprehensive academic support for students during the first year of college In: Student Academic Services: An Integrated Approach, San Francisco, CA: Jossey-Bass, 271-310. Google Scholar
  • Daniels H, Grineski SE, Collins TW, Morales DX, Morera O, Echegoyen L (2016). Factors influencing student gains from undergraduate research experiences at a Hispanic-serving institution. CBE Life Sci Educ 15, ar30. LinkGoogle Scholar
  • Dennis JM, Phinney JS, Chuateco LI (2005). The role of motivation, parental support, and peer support in the academic success of ethnic minority first-generation college students. J Coll Stud Dev 46, 223-236. Google Scholar
  • Diekman AB, Weisgram ES, Belanger AL (2015). New routes to recruiting and retaining women in STEM: policy implications of a communal goal congruity perspective. Soc Iss Pol Rev 9, 52-88. Google Scholar
  • Eddy SL, Brownell SE, Thummaphan P, Lan MC, Wenderoth MP (2015). Caution, student experience may vary: social identities impact a student’s experience in peer discussion. CBE Life Sci Educ 14, ar45. LinkGoogle Scholar
  • Egan MK, Hurtado S, Chang MJ, Garcia GA, Herrera FA, Gabibay JC (2013). Making a difference in science education: the impact of undergraduate research programs. Am Educ Res J 50, 683-713. MedlineGoogle Scholar
  • Estrada M, Myra Burnett M, Campbell AG, Campbell PB, Denetclaw WF, Gutiérrez CG, Hurtado S, John GH, Matsui J, McGee R, et al. (2016). Improving underrepresented minority student persistence in STEM. CBE Life Sci Educ 15, es5. LinkGoogle Scholar
  • Estrada M, Woodcock A, Hernadez P, Scultz WP (2011). Toward a model of social influence that explains minority student integration into the science community. J Educ Pyschol 11, 1-17. Google Scholar
  • Fass ME, Tubman JG (2002). The influence of parental and peer attachment on college students academic achievement. Psychol Sch 39, 561-573. Google Scholar
  • Feldman A, Divoll KA, Rogan-Klyve A (2013). Becoming researchers: the participation of undergraduate and graduate students in scientific research groups. Sci Educ 97, 218-243. Google Scholar
  • Good C, Rattan A, Dweck CS (2012). Why do women opt out? Sense of belonging and women’s representation in mathematics. J Soc Pers Psychol 102, 700-171. MedlineGoogle Scholar
  • Grunert ML, Bodner GM (2011). Finding fulfillment: women’s self-efficacy beliefs and career choices in chemistry. Chem Educ Res Pract 12, 420-426. Google Scholar
  • Hanauer DI, Graham MJ, Hatful GF (2016). A measure of college student persistence in the sciences (PITS). CBE Life Sci Educ 15, ar54. LinkGoogle Scholar
  • Hanauer DI, Hatfull G (2015). Measuring networking as an outcome variable in undergraduate research experiences. CBE Life Sci Educ 14, ar38. LinkGoogle Scholar
  • Harrison M, Dunbar D, Ratmansky L, Boyd K, Lopatto D (2011). Classroom-based science research at the introductory level: changes in career choices and attitude. CBE Life Sci Educ 10, 279-286. LinkGoogle Scholar
  • Hausmann LRM, Schofield JW, Woods RL (2007). Sense of belonging as a predictor of intentions to persist among African American and white first-year college students. Res High Educ 48, 803-839. Google Scholar
  • Hazari Z, Sonnert G, Sadler PM (2010). Connecting high school physics experiences, outcome expectations, physics identity, and physics career choice: a gender study. J Res Sci Teach 47, 978-1003. Google Scholar
  • Hernandez PR, Schultz PW, Estrada M, Woodcock A, Chance RC (2013). Sustaining optimal motivation: a longitudinal analysis of interventions to broaden participation of underrepresented students in STEM. J Educ Pyschol 105, 89-107. Google Scholar
  • Ho D, Imai K, King G, Stuart E (2007). Matching as nonparametric preprocessing for reducing model dependence in parametric causal inference. Polit Anal 15, 199-236. Google Scholar
  • Hossain MM, Robinson MG (2012). How to motivate US students to pursue STEM (science, technology, engineering and mathematics) careers. US-China Educ Rev A 4, 442-451. Google Scholar
  • Hu L, Bentler PM (1999). Cutoff criteria in fix indexes in covariance structure analysis: conventional criteria versus new alternatives. Struct Equation Modeling 6, (1), 1-55. Google Scholar
  • Jensen PA, Moore R (2008). Students’ behaviors, grades, and perceptions in an introductory biology course. Am Biol Teach 70, 483-487. Google Scholar
  • Komarraju M, Nadler D (2013). Self-efficacy and academic achievement: why do implicit beliefs, goals, and effort regulation matter. Learn Indiv Differ 25, 67-72. Google Scholar
  • Kuh GD (2008). High Impact Practices: What They Are, Who Has Access to Them, and Why They Matter, Washington, DC: Association of American Colleges and Universities. Google Scholar
  • Labov J, Singer S, George M, Schweingruber H, Hilton M (2009). Effective practices in undergraduate STEM education, part I: examining the evidence. CBE Life Sci Educ 8, 157-161. LinkGoogle Scholar
  • Lane FC, To YM, Shelley K, Henson RK (2012). An illustrative example of propensity score matching with education research. Car Tech Educ Res 37, 187-212. Google Scholar
  • Laursen S, Hunter A-B, Seymour E, Thiry H, Melton G (2010). Undergraduate Research in the Sciences: Engaging Students in Real Science, San Francisco, CA: Wiley. Google Scholar
  • Lent RW, Lopez FG, Bieschke KJ (1993). Predicting mathematics-related choice and success behaviors: test of an expanded social-cognitive model. J Vocat Behav 42, 223-236. Google Scholar
  • Lewis KL, Stout JG, Pollock SJ, Finkelstein ND, Ito TA (2016). Fitting in or opting out: a review of key social-psychological factors influencing a sense of belonging for women in physics. Phys Rev Spec Top Phys Educ Res 12, 020110. Google Scholar
  • Light G, Micari M (2013). Making Scientists, Cambridge, MA: Harvard University Press. Google Scholar
  • Linn MC, Palmer E, Baranger A, Gerard E, Elisa Stone E (2015). Undergraduate research experiences: impacts and opportunities. Science 347, 1261757. MedlineGoogle Scholar
  • Matsui J, Liu R, Kane CM (2003). Evaluating a science diversity program at UC Berkeley: more questions than answers. Cell Biol Educ 2, 117-121. LinkGoogle Scholar
  • Multon KD, Brown SD, Lent RW (1991). Relation of self-efficacy beliefs to academic outcomes: A meta analytic investigation. J Couns Psychol 18, 30-38. Google Scholar
  • Muthén LK, Muthén BO (1998). Mplus User’s Guide, Los Angeles. Google Scholar
  • National Research Council (2012). Discipline-Based Education Research: Understanding and Improving Learning in Undergraduate Science And Engineering, Washington, DC: National Academies Press. Google Scholar
  • National Science Foundation (2006). Science and Engineering Indicators, 2006. Google Scholar
  • Ovink SM, Veazey BD (2011). More than “getting us through”: a case study in cultural capital enrichment of underrepresented minority undergraduates. Res High Educ 52, 370-394. MedlineGoogle Scholar
  • Ozga J, Sukhnandan L (1998). Undergraduate non-completion: developing an explanatory model. High Educ Q 52, 316-333. Google Scholar
  • Pender M, Marcotte D, Sto. Domingo M, Maton K (2010). The STEM pipeline: the role of summer research experience in minority students’ graduate aspirations. Educ Policy Anal Arch 18, (30), 1-39. MedlineGoogle Scholar
  • Perez T, Cromley JG, Kaplan A (2014). The role of identity development, values, and costs in college STEM retention. J Educ Psychol 106, 315-329. Google Scholar
  • Peterson NA, Speer PW, McMillan D (2008). Validation of a brief sense of community scale: confirmation of the principal theory of sense of community. J Commun Psychol 36, 61-73. Google Scholar
  • Pfund C, Pribbenow CM, Branchaw J, Lauffer SM, Handelsman J (2006). The merits of training mentors. Science 311, 473. MedlineGoogle Scholar
  • President’s Council of Advisors on Science and Technology (2012). Engage to Excel: Producing One Million Additional College Graduates with Degrees in Science, Technology, Engineering and Mathematics, Washington, DC: U.S. Government Office of Science and Technology. Google Scholar
  • Robnett RD, Chemers MM, Zurbriggen EL (2015). Longitudinal associations among undergraduates’ research experience, self-efficacy, and identity. J Res Sci Teach 52, 847-867. Google Scholar
  • Rodenbusch SE, Hernandez PR, Simmons SL, Dolan EL (2016). Early engagement in course-based research increases graduation rates and completion of science, engineering, and mathematics degrees. CBE Life Sci Educ 15, ar20. LinkGoogle Scholar
  • Rosenbaum PR, Rubin DB (1984). Reducing bias in observational studies using subclassification on the propensity score. J Am Stat Assoc 79, 516-524. Google Scholar
  • Ryan RM, Deci EL (2000). Intrinsic and extrinsic motivations: classic definitions and new directions. Contemp Educ Psychol 25, 54-67. MedlineGoogle Scholar
  • Seymour E, Hewitt NM (1997). Talking about Leaving: Why Undergraduates Leave the Sciences, Boulder, CO: Westview. Google Scholar
  • Seymour E, Hunter A-B, Laursen SL, DeAntoni T (2004). Establishing the benefits of research experiences for undergraduates in the three-year study. Sci Educ 88, 493-534. Google Scholar
  • Stake JE, Mares KR (2001). Science enrichment programs for gifted high school girls and boys: predictors of program impact on science confidence and motivation. J Res Sci Teach 38, 1065-1088. Google Scholar
  • Stolle-McAllister K, Sto. Domingo MR, Carrillo A (2011). The Meyerhoff way: How the Meyerhoff Scholarship Program helps Black students succeed in the sciences. J Sci Educ Technol 20, 5-16. MedlineGoogle Scholar
  • Strayhorn TL (2012). College Students’ Sense of Belonging: A Key to Educational Success for all Students, New York: Routledge. Google Scholar
  • Stuart EA (2010). Matching methods for causal inference: a review and a look forward. Stat Sci 25, 1-21. MedlineGoogle Scholar
  • Thoemmes FJ (2012). Propensity score matching in SPSS In: http://arxiv.org/ftp/arxiv/papers/1201/1201.6385.pdf (accessed 1 August 2016). Google Scholar
  • Tinto V (1993). Leaving College: Rethinking the Causes and Cures of Student Attrition, 2nd ed. Chicago, IL: University of Chicago Press. Google Scholar
  • Trujillo G, Tanner KD (2014). Considering the role of affect in learning: monitoring students’ self-efficacy, sense of belonging, and science identity. CBE Life Sci Educ 13, 6-15. LinkGoogle Scholar
  • Usher EL, Pajares F (2008). Sources of self-efficacy in school: critical review of the literature and future directions. Rev Educ Res 78, 751-796. Google Scholar
  • van der Hoeven Kraft KJ, Srogi LA, Husman J, Semken S, Fuhrman M (2011). Engaging students to learn through the affective domain: a new framework for teaching in the geosciences. J Geosci Educ 59, 71-84. Google Scholar
  • VanMeter-Adams A, Frankenfeld CL, Bases J, Espina V, Liotta LA (2014). Students who demonstrate strong talent and interest in STEM are initially attracted to STEM through extracurricular experiences. CBE Life Sci Educ 13, 687-697. LinkGoogle Scholar
  • Wald HS, Reis SP (2010). Beyond the margins: reflective writing and development of reflective capacity in medical education. J Gen Intern Med 25, 746-749. MedlineGoogle Scholar
  • Wheeler ER, Wischusen SM (2014). Developing self-regulation and self-efficacy: a cognitive mechanism behind the success of biology boot camps. Elec J Sci Educ 18, 1-16. Google Scholar
  • Williams GC, Deci EL (1996). Internationalization of biopsychosocial values by medical students: A test of self-determination theory. J Pers Soc Psychol 70, 767-779. MedlineGoogle Scholar
  • Wischusen SM, Wischusen EW (2007). Biology intensive orientation for students (BIOS): A biology “boot camp.”. CBE Life Sci Educ 6, 172-178. LinkGoogle Scholar
  • Zimmerman BJ (2000). Self-efficacy: an essential motive to learn. Contemp Educ Psychol 25, 82-91. MedlineGoogle Scholar