Teaching Undergraduates to Communicate Science, Cultivate Mentoring Relationships, and Navigate Science Culture

INTRODUCTION

Over the last four decades, there has been a national effort to increase the representation of women, racial and ethnic minorities, and, more recently, first-generation college, economically disadvantaged, and disabled individuals in science, technology, engineering, and mathematics (STEM) careers (James and Singer, 2016; Schultz et al., 2011). For the most part, these efforts have taken the form of summer and academic-year programs for undergraduates that pique interest in pursuing STEM careers and provide research experience and career development opportunities. To date, individual programs have demonstrated success in sparking interest and moving students into graduate programs (Maton et al., 2012).

While the number of trainees from historically underrepresented (UR) groups entering the STEM career pipeline has increased, relatively few persist and thrive in STEM careers (Clauset et al., 2015; Valantine and Collins, 2015; Whittaker et al., 2015; Gibbs et al., 2016; Li and Koedel, 2017). UR trainees exit science career pathways at various stages due to an array of factors: lack of academic preparation, insufficient financial resources, an absence of mentors and role models, ineffective mentor and peer support, a paucity of professional networking opportunities, low personal or societal expectations, discouragement, unwelcoming environments, and the absence of an appreciation for diversity (Summers and Hrabowski, 2006; Gibbs and Griffin, 2013; Gibbs et al., 2016; Seymour and Hewitt, 2018). An increasing body of research suggests that focusing on academic aspects of UR students’ development alone may be insufficient to address the disparate rate at which UR students depart STEM education, training, and careers relative to their counterparts from overrepresented (OR) groups (Estrada et al., 2011, 2018; Hernandez et al., 2013; Findley-Van Nostrand and Pollenz, 2017). Instead, increasing the persistence of UR students in STEM careers will require focus on both the institutional barriers to their advancement and a more holistic view of the needs of UR students confronting educational and research environments that do not promote inclusion (President’s Council of Advisors on Science and Technology, 2012; Estrada et al., 2016). The literature suggests that skills such as communicating science and navigating research-mentoring relationships, as well as the ability to address the interaction of individual social identities with the culture of research environments, critically contribute to persistence in STEM fields.

Effective written and oral communication skills are key components of a successful career in STEM and serve as critical gatekeepers in determining advancement, for example, in applications and candidate interviews for graduate admissions. In fact, communication has been listed as a core competency for undergraduate science education (National Research Council, 2003; AAAS, 2011). However, this important skill is infrequently taught explicitly alongside requisite content knowledge and technical skills, especially at the undergraduate level (Coil et al., 2010; Mercer-Mapstone and Kuchel, 2015a,b, 2016). Those who have pioneered science communication education at the undergraduate level have found a concomitant enhancement of content knowledge and skills, including increased confidence and understanding of the importance of research projects, improved critical-thinking skills, and ability to present science to a variety of audiences both effectively and accurately (Quitadamo and Kurtz, 2007; Brownell et al., 2013; Cirino et al., 2017). Most importantly for UR researchers, a strong foundation in science communication skills significantly promotes persistence in STEM careers (Cameron et al., 2015).

Effective communication in relationships with peers, colleagues, and mentors is essential to career success. In particular, research-mentoring relationships are an integral part of cultivating successful STEM careers (Hurtado et al., 2009; Eagan et al., 2013; Prunuske et al., 2013; Linn et al., 2015; McGee, 2016; Pfund et al., 2015, 2016). Mentors foster the acquisition of knowledge, development of skills, and socialization into the culture of science. Moreover, they provide critical feedback, which is the basis of all growth and learning. While the historic focus in research-mentoring relationships has been on the actions of mentors, it is essential that mentees appreciate the critical importance of mentorship, and their own responsibility to influence the effectiveness of mentoring, by actively managing mentoring relationships (Zerzan et al., 2009; Lee et al., 2015; McGee et al., 2015; Montgomery, 2017). Recent studies show that mentoring relationships are instrumental in promoting the persistence of UR undergraduate students embarking on STEM careers (Maton and Hrabowski, 2004; Wilson et al., 2011; Graham et al., 2013; Estrada et al., 2016, 2018). UR students, in particular, may have barriers to optimizing the value of their mentoring relationships due to cultural differences and insufficient awareness about socially acceptable science mentor engagement (Byars-Winston et al., 2016, 2018; McGee, 2016; Pfund et al., 2016).

While a strong ability to communicate one’s science and maintain effective mentoring relationships contributes to persistence in STEM careers, these factors alone are not sufficient. Aspiring scientists must feel that they belong and will be able to succeed as scientists. Perceived belonging and therefore persistence of individuals in any culture hinges on the individual’s and the group’s perception of that individual (Ibrahim and Heuer, 2016). Individuals who are not perceived as belonging experience self- and group-imposed barriers to persistence. Science training and research culture is not different (Thoman et al., 2017). UR undergraduate science majors have a lower sense of belonging in their majors compared with their OR counterparts. This perception is associated with decreased retention in science majors (Rainey et al., 2018). Belonging requires that one understand and embody a unique set of social norms and values. Compared with OR students, UR students may not be aware of these expectations and/or science culture–related social norms, and their values may be at odds with those learned in the cultural groups of their birth (Chemers et al., 2011). A diminished or absent sense of belonging to the scientific community makes UR students prone to the adverse effects of psychosocial phenomena (such as unconscious bias, stereotype threat, and impostor syndrome) that collectively act as barriers to STEM persistence and advancement (Steele and Aronson, 1995; Hunter et al., 2007; Chang et al., 2011; Estrada et al., 2011; Woodcock et al., 2012). Raised awareness of these psychosocial phenomena and instruction on how to cope with them promote persistence in STEM (Woodcock et al., 2016; Ben-Zeev et al., 2017; Findley-Van Nostrand and Pollenz, 2017; Williams et al., 2017).

These three aspects of science culture—scientific communication, mentoring relationships, and individual social identities—are highly connected and interdependent. For example, a strong foundation in scientific communication accelerates the development of science identity and intention to pursue research careers (Cameron et al., 2015). Moreover, most scientific communication skills are learned informally through the mentored research relationship. In turn, relationships, especially those with mentors, both directly and indirectly strengthen science identity and belonging, thus promoting STEM career persistence (Hernandez et al., 2017; Rainey et al., 2018). This learning process is confounded by at least two factors. First, young investigators often have not developed the communication skills necessary to interact effectively with mentors (Cameron et al., 2013). And second, mentors do not always have the requisite skills to fulfill their role as communication instructor or cultural guide (Thiry and Laursen, 2011; Cameron et al., 2013; Hernandez et al., 2013). Trachtenberg and colleagues underscore the important relationship among scientific communication, mentorship, and identity by suggesting that identity-based differences in spoken English language can both adversely impact trainees’ self-perceived ability to communicate and create barriers to receiving adequate mentorship (Trachtenberg et al. 2018). The interrelated nature of communication skills, mentoring relationships, and awareness of the interaction between researchers’ identities with the culture of science suggests potential synergy from focusing on the development of all three deliberately and simultaneously.

Although critical for persistence in higher education and to transition successfully from classroom-based learning to research, these three forms of capital are rarely described in formal curricula. Undergraduate education is a critical stage for developing scientists. It is often when students have their first laboratory experience and begin to develop a science identity. We have taken a novel approach that combines evidence-based instruction in all three topic areas into one course. Our goal is to make explicit the unwritten rules that govern research interactions, as previous research has shown these explicit results in career advancement for undergraduates. Further, through classroom activities and discussion, we emphasize the interrelated nature of the new skills and awareness acquired in the course. We explicitly addressed the fact that, for members of UR groups, each element is necessary but insufficient to ensure STEM advancement, particularly because they can be judged differently from their OR peers. Here, we describe the course and assess student learning outcomes using both quantitative and qualitative measures. To assess student learning outcomes, we asked whether participation in the course strengthened students’ ability to communicate their research, comfort navigating mentoring relationships, and familiarity with relevant psychosocial phenomena. This course represents a unique opportunity for young scientists to strengthen these skills in a context that illuminates their interconnectedness and importance to career success. Ample literature suggests that establishing a foundation in these three topic areas early will prepare students to succeed in graduate school and promote their persistence in STEM careers.

METHODS

This study was reviewed and approved by the University of Massachusetts (UMass) Boston Institutional Review Board (Study 2017098).

Course Structure and Roles

The Communicating in Science for Undergraduates course was offered in three consecutive semesters (Spring 2016, Fall 2016, and Spring 2017) at UMass Boston, which is recognized as a minority-serving institution by the U.S. Department of Education. The course was taught by four instructors each semester. The instructors and teaching/research assistants were all active researchers who offered broad representation of racial and other social identities and research disciplines; these deliberate differences greatly contributed to class discussions about identities in science. The research team served as course instructors and also developed and revised course content and designed and tested the pre and post surveys and the assignment evaluation rubrics. This design/teaching team incorporated a course teaching assistant for the second and third semesters who assisted with assignment review, rubric development, and in-class discussions. In the first two semesters, all four instructors participated in each class session. To better approximate implementation of a typical undergraduate course, in the final semester just two instructors were present during each class session.

Students

Course instructors recruited students to the course by distributing flyers through on-campus undergraduate research training programs as well as by posting them throughout academic buildings on campus. Principal investigators with active research programs were asked to encourage their undergraduate researchers to enroll as part of the student recruitment effort. Mentors were not recruited directly but were invited to participate once their students enrolled in the course. The first offering of the course closely coincided with a research mentor training workshop in Boston organized by the instructors and their colleagues. This 8-hour workshop was facilitated by staff of the Mentor Training Core of the National Research Mentoring Network. All of the original instructors took part in that workshop, as did eight of the students’ research mentors. For the second offering of the course, one of the student’s research mentors attended an orientation at the start of the semester that described the structure of the course, the major assignments, expectations for the students and their mentors, and the forms of support the instructors could provide. In contrast to the variation in the mentors’ training, all students received the same instruction regarding mentorship (based on Lee et al., 2015). Throughout all offerings of the course, we used the definition of research mentorship of Pfund et al. (2016).

A total of 33 students actively engaged in research projects completed the course (Table 1). Seventy percent of students were underrepresented in STEM careers, and 61% were women. Except for one sophomore, all students were juniors (33%) and seniors (64%) with a variety of majors: biology (48%), biochemistry (18%), chemistry (18%), engineering (6%), psychology (6%), and women and gender studies (3%). As a course requirement, all students were actively engaged in a mentored research project while taking the course. Students’ research projects were directly supervised by technicians (6%), graduate students (58%), and/or faculty (36%) mentors. Their projects spanned a variety of disciplines, including basic science (73%), population science (15%), computational science (6%), and engineering (6%). For nearly half of the students (48%), these projects were their first research experiences, and 58% had been working on their projects for five or fewer months when they started the course. The majority of students maintained overall grade point averages of 3.6 or above (61%) and participated in research training programs (58%) such as the National Institutes of Health Initiative for Maximizing Student Development (IMSD) and the Department of Education Ronald E. McNair Scholars Program. Most students planned to go on to graduate school (55%), postbaccalaureate programs (15%), jobs (15%), or professional school (12%) after graduation. One student was undecided about postgraduation plans.

The Course

Course content was based on curricula developed for the Broad Summer Research Program (BSRP) at the Broad Institute of MIT and Harvard that combined learning objectives in three main topic areas: scientific communication, mentoring relationships, and social identities as they relate to science culture (Table 2 and Supplemental Figure 1). BSRP is a 9-week-long program for undergraduates from UR groups in STEM. BSRP typically included only eight to 12 students per year, and they were relatively uniform in their race/ethnicity, age, and major areas of study. We therefore sought to develop a formalized curriculum that would be effective with a larger number of students who reflected the greater diversity of the UMass Boston population. In addition, the greater amount of available time allowed us to incorporate more assigned reading and more in-class time for practicing skills for communicating and giving feedback under the guidance of instructors. Finally, we sought to take advantage of a larger instructional staff by incorporating more small-group work and discussion. Although the curriculum (course syllabus in Supplemental Figure 2) ascribes class activities and assignments (Supplemental Table 1) to one of the three main topic areas (Supplemental Table 2), instruction deliberately blended topics from each area to emphasize their inherent synergy. For example, fostering a deep value for seeking and receiving feedback while developing skills to give effective feedback was not only essential in improving students’ ability to communicate their science, but also inculcated in them an understanding of the fundamental aspects of mentoring relationships and the culture of science.

To maximize student mastery of course learning objectives, instructors employed an inverted classroom model (Lage and Platt, 2000). Preclass assignments introduced and familiarized students with topics. Classroom time was devoted primarily to active learning in the form of primed discussions and small-group/partner exercises that reinforced the concepts introduced through pre-work. In addition, the course incorporated the students’ research mentors at key points in the semester to improve mentors’ ability to help students navigate the culture of science and to assist the students in achieving excellence in communicating their research.

Pre- and Postcourse Surveys

Course instructors developed pre- and postcourse questionnaires (shown in their entirety as Supplemental Figures 3 and 4) to assess student learning. While the questionnaires addressed all three topic areas covered by the course, because improvement in students’ scientific communication skills was measured directly from student work, the questionnaires primarily asked about shifts in students’ attitudes about scientific communication and their learning concerning research mentorship and social identity in science. Due to the unique combination of interrelated course topics, we developed new questionnaires rather than relying on previously published surveys. Questionnaires were developed in three stages, in which instructors: 1) listed potential questions specific to each topic area; 2) completed the survey, along with teaching/research assistants, to ensure the instrument was comprehensive and clear; and 3) administered the preliminary questionnaires to the first cohort of 14 students. Based on the answers from the first semester, the questionnaires were modified to improve clarity, reduce redundancy, and ascertain actual learning in addition to students’ perceived knowledge.

These final questionnaires were administered to students in the two subsequent cohorts, and results from those 18 students are reported here. During class on the first and last days, students received individualized links deployed from SurveyMonkey and completed the questionnaires. Ninety-five percent of students completed both pre- and postcourse questionnaires. One student did not complete the postcourse questionnaire and was excluded.

The questionnaires were a combination of short-answer and five-point Likert scale responses. The precourse questionnaire has 25 items, 21 of which assess students’ prior knowledge, experience, and understanding of how each course topic area relates to STEM research, while four items collected demographic information. Information about educational level, major, research, and postgraduation plans were collected before acceptance into the course. Seven items asked about scientific communication experience, knowledge, and skills, and eight items asked about mentoring relationships. Six items asked about students’ familiarity with psychosocial concepts and their ability to define terms. The psychosocial phenomena covered in the course include unconscious bias, stereotype threat, and impostor syndrome. If students answered that they were familiar with a psychosocial phenomenon, they were asked to define it. The postcourse questionnaire had 31 items, including 21 items that mirrored the pre survey, and 10 items to assess course satisfaction. Demographic questions were not included on the post survey.

For the analysis of Likert scale items, answers from pre- and postcourse questionnaires were collated, matched, and converted to numerical values. Paired t tests were used to determine whether there was a statistically significant difference in student responses before and after the course.

Scientific Communication Assignments

Four major assignments involving traditional science communication skills were evaluated: 1) a verbal introduction of oneself and one’s research; 2) a written abstract; 3) a formal presentation slide deck; and 4) a formal 8-minute oral presentation.

For the introduction, often referred to as an “elevator pitch,” students were instructed to introduce themselves for 30–60 seconds as they would if meeting a guest speaker and to include the following: name, year in school, laboratory in which they were an undergraduate researcher, the laboratory’s general area of research, the student’s specific research question, and why their research matters. Students delivered their introductions once per week for most weeks and received immediate feedback from instructors and classmates each time. In the third semester, introductions were video-recorded on three separate days. When revising their introductions, students were asked to consider peer and instructor feedback as well as aspects of their introductions that they would like to keep, improve, or omit.

Before writing abstracts, students were assigned reading material on how to write an abstract and participated in various exercises on writing abstracts, such as identifying parts of an abstract and editing example abstracts. Students then wrote abstracts about their research targeted to a general scientific audience. After receiving feedback from their mentors, students submitted these abstracts, which were considered the first draft abstract. Students received feedback from an instructor or teaching assistant and classmates on this first abstract and a second draft. After these drafts and additional in-class exercises on writing abstracts, students submitted their final abstracts at the end of the semester.

To prepare for the oral presentation, students worked throughout the semester in a progressive manner. Students began by giving oral descriptions of their research, using markers and a whiteboard to provide illustrations and list keywords or phrases, and eventually wrote an outline for an 8-minute talk for an audience with a general scientific background. Each major section of the presentation (introduction, methods, results, and conclusions) was addressed separately and sequentially, with students first receiving instruction, then preparing draft slides, and finally practicing their oral delivery of each section of the presentation. For example, after reading material on how to make effective slides for a presentation, students were asked to create two to three draft slides for their introductions. In the early stages, students shared slides during class while orally summarizing the key points they wanted to convey with each slide, that is, students were initially encouraged to not “give their talk” but instead to describe in the fewest words the essential concepts associated with each slide. This provided a round of feedback just on the slides, specifically focusing on how effectively the slides supported these points. Students thus received multiple rounds of feedback from classmates and instructors on the draft slides from the earlier sections of the talk before preparing and submitting a slide deck comprising all sections, which was considered the first version of their slides. The slides students submitted for their final presentations were considered their final slide decks.

Students practiced their talks two to three times during class and received oral and written feedback on both their slides and presentation from peers and instructors. Each practice presentation in the second and third semesters was video-recorded; the first full-length practice presentation was considered the first presentation. In addition to in-class practice, students practiced their presentations outside class with mentors and peers. The final presentation on the last day of class was recorded for the second and third semesters and considered the final presentation.

“Take-Away” Concepts

Every week, students were asked to list three things they learned, or their “take-aways,” from class that week. In the first two semesters, responses were written on index cards and submitted anonymously; in the third semester, students submitted these online. To analyze the students’ self-reported learning, we identified provisional themes and subthemes for the student responses. Student responses across the three semesters were then categorized into these provisional themes and subthemes using Dedoose mixed-methods software. We identified patterns in student responses across the three semesters, including the frequency with which each theme was referenced, and then the themes were revisited and expanded or pruned. This yielded the frequency of responses by theme, represented in Figure 3, which is represented graphically in Supplemental Figure 1.

Data Analysis

To assess each of the four major assignments, we developed rubrics that were based on materials provided to students and were consistent with instructors’ expectations. The oral presentation rubric was modified from rubrics in Sevian and Gonsalves (2008) to reflect the material taught in the course and expectations communicated by the instructors. Before using these rubrics to generate final assessments, we revised them by applying them in multiple rounds to actual course work to improve accuracy and minimize ambiguity. Students’ slide decks and oral presentations were assessed using two separate rubrics for each assignment: one for content and another for style. Rubrics were structured to assign a score of 1, 2, 3, or 4 for each component, which corresponded to beginning, developing, proficient, and mastery levels, respectively. In using the rubrics to assess student work, we differentially weighted the sections to reflect their emphasis within course instruction (Supplemental Tables 3–8).

All assignments were scored over 3 days in the summer following the third semester of the course by 18 graduate students recruited from various departments at UMass Boston (Biology, Chemistry, Physics, Psychology, Nursing, and School for the Environment), using a custom-made Web application that randomly assigned student work to an appropriate reviewer and ensured that each piece of work was assigned to three independent reviewers and that only valid scores were entered. To minimize reviewer bias, student assignments were anonymized, identifying information was removed where possible, and review sets were assigned randomly with two exceptions: 1) reviewers were not assigned to score any items by students whose research was from the same laboratory; and 2) first and final assignments from the same student were not scored by the same reviewer. Each reviewer was assigned between six and 20 abstracts, 10–12 slide decks, nine to 12 oral presentation videos, and/or three to four introduction videos, and means were calculated from the scores of different reviewers. Concordance among reviewers was moderate (0.61) to high (0.81) across assignments.

All student abstracts were assessed on the first day of scoring, all slides the second day, and all oral presentations and introductions the third. At the beginning of each day, reviewers participated in a calibration session. After reviewers were given initial instructions by a course instructor, they scored two to three example assignments using the rubrics and were encouraged to ask questions. Examples used for calibration did not include any assignments that were part of the research set. Scores for the example sets were then collected and discussed so that reviewers could reach a consensus on what the expectations were for each category in the rubric. Variation in the number of students scored for each assignment reflects the number of students for which there were first and final assignments available and the fact that recordings were not introduced until the second semester (for presentations) and third semester (for introductions), respectively.

RESULTS

Students Displayed Statistically Significant Improvement in Their Ability to Communicate Science

Students used their mentored research projects as the basis for four major assignments that, through iterations with rising expectations, developed students’ abilities to present the purpose and significance of their research to broad audiences. Together, the assignments foster the development of skills needed for spontaneous scientific conversation, writing, and planned speeches that students will use repeatedly throughout their careers. The acquisition of these skills is essential for career success and has been shown to solidify intention to continue in STEM careers (Cameron et al., 2015).

To assess student performance on oral, written, and graphical communication assignments, graduate students from multiple disciplines were recruited and trained to use rubrics to evaluate student work. For each of the four major assignments, the scores obtained from applying each rubric averaged across all students, or mean rubric score, are reported in Figure 1. The magnitude of difference, d, was also calculated. Overall, students’ abstracts (p ≤ 0.01, d = 0.49), slide decks (p ≤ 0.001, d = 0.63), and presentations (p ≤ 0.001, d = 1.03) showed statistically significant improvement with moderate to high effect sizes, from first to final drafts. While introductions showed a trend of improvement, the difference between first and final versions was not statistically significant. There was, however, a high effect size (d = 0.86) for introductions. The lack of statistical significance likely reflects the low sample number.

While overall improvement in ability to communicate science is an important finding, we looked more closely at the components that drove the improvement signal and found that students improved most in areas that are most relevant to effective communication. For example, our guidelines for writing abstracts contained sections on requisite content and length as well as the importance of conveying the significance and implications of one’s work to a broad audience. While overall abstract length did not change between the first and final drafts, reviewers scored the final abstracts statistically higher in conveying the significance (p ≤ 0.033) and implications (p ≤ 0.027) of student research at an appropriate level for an audience (p ≤ 0.007) of scientists from different disciplines.

The course also emphasized that graphical forms of communication help convey the motivation and significance of one’s work. Students’ slides demonstrated improvement in their ability to convey the motivation for (p ≤ 0.042), and significance of (p ≤ 0.015), their projects and to summarize (p ≤ 0.005) their work. Instruction also highlighted the importance of having both an appropriate number of slides and a highly readable format (e.g., the amount and size of text). Students’ slide decks improved significantly for both slide number and text readability (p ≤ 0.022 and 0.009, respectively).

Students’ oral presentations improved in multiple areas, including their description of the underlying background (p ≤ 0.026), motivation (p ≤ 0.008), and methodology (p ≤ 0.018). In addition, we measured statistically significant improvement in student’s poise (p ≤ 0.002); pace (p ≤ 0.038); voice (p ≤ 0.034); transitions (e.g., between slides, sections, or major ideas; p ≤ 0.022); balance (p ≤ 0.011); and scaffolding, the overall building of the story (p ≤ 0.013; Sevian and Gonsalves, 2008). In addition to seeing improvements in aspects of oral presentation that were emphasized in the course, we noted improvements in topics like balance and scaffolding that were not explicitly emphasized during class. We found no significant correlations in student performance on any assignment with gender, major, or underrepresentation in STEM (unpublished data).

Postcourse surveys revealed additional information about students’ valuation and perceptions of the process of scientific communication that were not captured by our assessment of student work. For example, through participation in the course, students gained confidence in their ability to prepare an oral presentation and a sense of responsibility to tailor the talk to their audience:

Moreover, students recognized that building scientific communication skills is an iterative process they should continue to develop outside the course:

Women, First-Generation College, and Latinx Students Experienced a Significant Increase in Comfort with Navigating Mentoring Relationships

Effective mentoring relationships catalyze career development and advancement. Mentees can have great influence over the success of their research mentoring relationships if they navigate those relationships deliberately (Lee et al., 2015; Montgomery, 2017). The course explicitly outlined student responsibilities in, and taught skills for, maximizing the effectiveness of mentoring relationships. Specifically, students were taught and prompted to use strategies to align mentor/mentee expectations, communicate effectively, give and receive feedback, and navigate difficult conversations (Branchaw et al., 2019). Skills were reinforced through in-class activities and conversations with their mentors, with the structure and content defined by course assignments. When possible, mentors attended and participated in specific class sessions to discuss their experiences navigating challenges in science communication or culture and provide feedback on student presentations.

Before and after the course, students were asked to assess their comfort levels with communicating their goals to their mentors, navigating difficult conversations with their mentors, and receiving critical feedback from their mentors. In general, students began the semester at a notably high level of comfort with 67%, 44%, and 94% of students reporting comfortable or very comfortable levels of communication in each category, respectively (Figure 2). At the end of the semester, students reported gains in comfort in all three categories, moving to 89% and 55% of students being comfortable or very comfortable with discussing their goals and having difficult conversations. Though the already high absolute number of students who reported being comfortable or very comfortable with receiving feedback from their mentors remained the same, more students report being very comfortable than comfortable after the course. Relative to the other two measures of student attitudes toward their mentoring relationships, students reported lower levels of comfort having difficult conversations with their mentors and less improvement in this category. This difference likely reflects the fact that only in the third semester did the course include a module on how to constructively approach difficult conversations.

While overall there were no statistically significant differences between pre and postcourse survey comfort levels in students discussing their goals, navigating difficult conversations, or receiving feedback from their mentors, bivariate analysis revealed a statistically significant increase in comfort levels among subgroups of students (Supplemental Table 8). For example, women and first-generation college students showed more improvement in their self-reported ability to navigate difficult conversations than men and non–first generation college students, respectively (p = 0.046). Latinx students showed statistically significant improvement in comfort level with receiving feedback (p = 0.017). These results suggest that the course material may have the desired effect of improving women’s and Latinx’s ability to navigate mentoring relationships and warrants further investigation with a larger number of students.

In their postcourse surveys, students acknowledged stronger research mentor relationships and attributed the improvement to what they learned in the course:

Students Became More Aware of Ways in which Social Identities Interact with the Culture of Science

To foster a sense of belonging in science, students explored aspects of their personal identities and how their identities contribute to their experiences in science through readings, mentor interviews, written reflections, and small-group discussions. In addition, students were introduced to psychosocial phenomena that may influence their sense of belonging and drive to pursue a STEM career. Knowledge and specific instruction on how to combat these psychosocial phenomena has been shown to inoculate novice scientists against their adverse effects (Woodcock et al., 2016; Ben-Zeev et al., 2017; Findley-Van Nostrand and Pollenz, 2017; Williams et al., 2017).

In pre- and postcourse surveys, students were asked to report their level of familiarity with each of three psychosocial phenomena: imposter syndrome, stereotype threat, and implicit bias (Figure 3A–C). Initially, the majority of students were unfamiliar with these ideas, with only three of 18 students reporting familiarity with one or more of the concepts. Throughout the course, students learned to define these psychosocial phenomena and how to recognize them and combat their effects. In the postcourse survey, students demonstrated a clear, statistically significant shift in understanding, with 17 of 18 students reporting being familiar or very familiar with the topics.

To confirm concept familiarity, students who reported moderate or greater familiarity with a concept were asked to define it (Figure 3D). On the precourse survey, only one student was able to accurately define one of the concepts, suggesting perceived, but not actual, familiarity with concepts. The observed postcourse knowledge shift was further corroborated by students’ increased ability to accurately define imposter syndrome, stereotype threat, and implicit bias: 67%, 59%, and 83% of students, respectively. Postcourse surveys also demonstrated students’ ability to connect the relevance of these concepts to their relationships with their peers and mentors, their self-perceptions, and their ability to overcome the impact of these factors in the future:

Student-Reported Learning Outcomes Reflect Balance among Course Components

To determine what students learned throughout the course, we asked students each week to list three things that they took away from class. The responses, take-aways, from all three semesters were compiled (n = 655), and common themes were extracted. The 30 most frequently captured themes among student responses are listed in Table 3. The category with the most responses, “making use of/giving feedback,” was cited 78 times, or 12% of all responses. We emphasized feedback during most classes (directly and indirectly) as an essential part of the culture of science, including learning to communicate and work with mentors. Students were introduced to the importance of seeking and incorporating feedback through discussion of the growth mindset (Claro et al., 2016; Dweck, 2017). They were then taught a framework for providing critical feedback, and their skills were honed through exercises. For example, students practiced giving constructive feedback to peers on an abstract provided by their instructors that contained specific flaws. Students then received feedback from their peers on their effectiveness in providing that feedback. Throughout the course, students provided oral and written feedback to their peers during and after each practice presentation as well as after practicing their introductions. The overall effect was a high comfort level with both giving and receiving constructive feedback.

All three main components of the course (Table 2) are represented within the five most frequent responses among the student-reported learning concepts (Table 3). To better understand the extent of alignment between course objectives and student learning, we compared the frequency of concepts mentioned by students in their take-aways with the amount of time devoted to that topic in lesson plans. Because of the iterative changes in course content, we analyzed this separately for each semester. As an example, we present the amount of time, in hours, devoted to each of the major topic areas in the third semester (Figure 4A). We heavily emphasized science communication throughout the semester; 40% of class time was devoted to that topic, with an additional 22% of class time being spent practicing and delivering the students’ final presentations. Not surprisingly, the most frequently mentioned topic of student-reported learning was science communication, with 130 of the 229 comments from the third semester falling under that topic (Figure 4B). Interestingly, although discussion of mentoring relationships received the second-most emphasis in class (15% of class time), it received the lowest number of comments in student-reported learning (20 comments). Conversely, although only 10% of class time focused on science culture and social identities, 75 of the comments, 33%, focused on this topic. To provide higher resolution on how the amount of time devoted to each major topic and self-reported student learning varied week by week, we present these for the third semester (Figure 4C and D). The nonlinear relationship between class time allocated and student learning is readily apparent in weeks 9 and 12, when no significant in-class time was devoted to discussing science culture and identities, but students included these concepts in their take-aways. The differences between the frequency with which students mentioned mentoring relationships compared with mentioning cultural topics may be due to students already having some knowledge of mentoring relationships, so they report a smaller impact on learning this topic. Similarly, they may have had little knowledge of science culture, and the information that was provided had a big impact.

Looking at them holistically, the take-aways illustrate students’ collective recognition of the value of the following:

1. Preparation and delivery of presentations:

   “I learned about ways to reduce anxiety which include breathing, maintaining a confident posture and eye contact, smiling, and above all, lots and lots of practice!”

   “Practice is key.”

   “Practice does make perfect.”

2. Students’ contributions to mentoring relationships:

   “The mentee plays a big role in the mentee/mentor relationship.”

   “A key component to feeling confident about being a mentee is realizing that the relationship is symbiotic. This provides a frame to contribute to your mentor’s experience either through contributing to their work or promoting learning in terms of unknown knowledge. Recognizing the nature of the relationship makes me feel less guilty for seeking help as I now understand that it is a cycle that science development thrives on.”

   “There’s a lot of similarities between a mentor and mentee, but also differences that can be minimized by having an effective communication.”

   “I learned how important it is to prepare for a difficult conversation in order for it to go successfully. I learned what questions you should ask yourself before going into a certain conversation and that it is important to not have a narrow perspective.”

3. Personal identities in science and the reality of a shared experience:

   “My identity is an asset in a lab.”

   “Talking about experiences makes things better.”

   “Everyone deals with problems like impostor syndrome, stereotype threat, and solo status. It is wrong to think that you’re the only one in the situation.”

4. Giving and receiving feedback:

   “How hard giving feedback actually is.”

   “Getting feedback on the slides was great.”

   “I really appreciate all of the feedback.”

   “Critical feedback is helpful.”

DISCUSSION

To address persistent underrepresentation of certain groups in scientific research, we developed a novel undergraduate course that integrates instruction of skills crucial within the research community and that makes explicit cultural norms that operate in a research environment. We focused on communication skills, such as succinctly describing one’s research in a compelling manner and providing critical feedback. We also taught methods to make the most of mentoring relationships, because of the importance of mentoring in academic and career advancement. We equipped students with knowledge about psychosocial phenomena that impact persistence in STEM and strategies to identify and combat them. These skills and concepts are rarely formally taught to undergraduates, and even more rarely described in a single framework; by doing so, we seek to promote persistence in STEM careers of members of historically UR groups.

To assess the course’ effectiveness, we evaluated student learning in the three topic areas. Students made statistically significant gains in their ability to write abstracts, generate slide decks, and create and deliver oral research presentations, as well as in other areas covered in the course, like the ability to convey the significance of one’s research. It is important to note that the gains we demonstrated in students’ communication abilities were likely underestimates of their actual learning, because, instead of being true baselines, the initial assessments were made after students received considerable course instruction and coaching. Overall, students demonstrated a trend toward improved mentor engagement, but not a statistically significant one. However, UR students did display statistically significant improvements in mentor comfort level, suggesting that course instruction may have a differential effect on these groups. Alternatively, the lack of significant change among OR students may reflect overestimates of their comfort at the outset of the course. Collecting retrospective data on the post survey may clarify this issue, as would evaluation of a larger number of students.

The course had a significant impact on students’ understanding of the existence of psychosocial phenomena that have been shown to impact persistence in STEM. Students began each semester unaware of these phenomena; by the end of the semester, they had a statistically significantly improved familiarity with each phenomenon, and more students were able to accurately define them. Future studies are needed to determine whether learning about strategies to combat the effects of psychosocial phenomena resulted in the actual use of these strategies. Finally, in the student’s self-reported take-aways, they demonstrated a value for course content.

Over the three semesters, instructors made multiple modifications to improve the course. For example, evidence suggests that changes to instruction in abstract writing improved student learning (unpublished data). Other notable changes included increased use of video recordings (particularly in the third semester) and sharing these with the students to promote self-evaluation and further learning. In addition, topics related to identity and scientific culture progressively received greater emphasis to achieve a better balance among the three topic areas.

While we have demonstrated learning tied to the course content, we believe our metrics underestimate what was learned. A more accurate approach would use true baseline assignments, those made before any instruction. Alternatively, a larger study could compare students’ abilities with those of students who did not take the course, though matching these carefully would be challenging. More importantly, we believe that our short-term metrics do not reflect the more impactful outcomes of the class. These long-term outcomes are both harder to measure and will require more time to appear. For example, we predict students who complete this class will show more successful interviews for graduate school admission and, having been admitted, will be better able to navigate their relationships with research mentors and lab members and will thus more quickly acclimate to, and progress in, graduate school than if had they not taken the class. We are especially eager to see whether the course contributes to improved persistence of UR students in STEM and, if so, what elements were most responsible. Although such long-term studies are beyond the scope of the initial work, we look forward to seeing them undertaken.

Although we believe this class for undergraduates offers substantial benefit to the mentors of course participants, we found uneven participation of, and support from, the research mentors. In the first semester, nearly all mentors attended a course orientation session and actively participated in at least two classes. In addition, in the first semester, half of the mentors attended a daylong research mentor training workshop offered by the National Research Mentoring Network. In subsequent semesters, fewer mentors participated in class, with only two attending the course orientation and few receiving formal mentor training. Moreover, the response rate of mentors to our surveys was low, so we were unable to capture the amount of time they spent with their mentees on course assignments or their thoughts about the course. Although we made stronger efforts to communicate in advance of the course and prepare mentors, we cannot fully explain the difference in mentor engagement between semesters. In future iterations of the class, we will increase the expectations for mentor engagement to allow the mentors to better reinforce all course content and strengthen their own mentoring skills.

The most important, unexamined aspect of the course is the extent to which OR students would recognize the relevance of course content designed to foster UR student persistence. To be clear, all students initially expressed skepticism about the relevance of identity and culture to their futures in science. The course deliberately introduced a broad and nuanced interpretation of identity (Loden and Rosener, 1991), and through readings and subsequent discussions, students shared personal experiences that highlighted how academic opportunities and encouragement can be linked to aspects of identity. For this reason, it was valuable that the class included students from groups that are both OR and UR in STEM fields. Interestingly, at this stage of their lives, many of the students, from a variety of backgrounds, had not yet given much thought to the aspects of their identity that are most salient in, for example, school versus home. One clear benefit of holding structured discussions including both UR and OR students was the recognition that all students can be subject to the negative consequences of implicit bias, impostor syndrome, and stereotype threat. In addition, the OR students in this class, who we believe are now also more likely to persist in STEM, may be more motivated to support UR students and foster a culture of equity and inclusion in their environments. Finally, our instructors deliberately included active researchers from both UR and OR groups. We suspect that this had important implications in our discussions about identity within the scientific culture, for example, in giving “formal” recognition of the presence of inequities in science as well as solutions that can promote inclusion and persistence of members of UR groups. In the future, we would like to test these and other hypothesized impacts of the course on OR students.

Because of the fundamental importance to all STEM students of the topics covered, in particular, their contribution to the persistence of UR students, and the limited representation of these topics in undergraduate curricula (Cirino et al., 2017), we look forward to continuing to improve the course, expanding its scope to include more writing, and disseminating additional materials to allow others to implement this course in a variety of contexts.

ACKNOWLEDGMENTS

REFERENCES