Validation and Application of the Survey of Teaching Beliefs and Practices for Undergraduates (STEP-U): Identifying Factors Associated with Valuing Important Workplace Skills among Biology Students

INTRODUCTION

Current science, technology, engineering, and mathematics (STEM) graduates are often perceived as being ill-prepared for the workplace (Arum and Roksa, 2011; President’s Council of Advisors for Science and Technology [PCAST], 2012; Hart Research Associates, 2015). In particular, surveys of STEM employers indicate that recent college graduates do not meet employer expectations for working collaboratively, thinking critically, and writing scientifically (Hart Research Associates, 2015). This has motivated national efforts to redesign the undergraduate STEM curriculum to allow students to better develop these skills, for example, by adopting evidence-based teaching approaches that increase student engagement in learning (American Association for the Advancement of Science [AAAS], 2011, 2015).

These curricular changes have been accompanied by increased efforts to understand the impact of course and curriculum redesign. The science education research literature has focused primarily on the teaching approaches faculty members report using in the classroom and their beliefs about teaching (Martin et al., 2000; Marbach-Ad et al., 2014; Smith et al., 2014) and the impact of changed teaching practices on student engagement, conceptual learning, and academic performance (McShannon and Hynes, 2005; Umbach and Wawrzynski, 2005; Freeman et al., 2014). There is a relative lack of empirical data on student values and perceptions regarding their educational experiences. Understanding the student perspective is critical, since values are thought to influence actions and behaviors across several contexts (Prosser et al., 1994; Trigwell et al., 1999). Thus, students who have an educational experience that assists them in recognizing the value of skills important in the workplace may be more likely to use those skills. For example, students who value the ability to work in groups may be more likely to engage in collaboration in the workplace.

Here, we introduce a novel assessment tool, the Survey of Teaching Beliefs and Practices for Undergraduates (STEP-U), which is designed to measure biology students’ values and experiences using a long-term perspective by asking students to reflect holistically on their undergraduate degree programs. This instrument aims to measure the extent to which students value skills that have been identified as important by both biology educators (AAAS, 2011) and employers (Hart Research Associates, 2015). The STEP-U uses the conceptual framework of Mayer (2002), which recognizes two categories of learning: retention (memorization and rote learning) and transfer (knowledge application and conceptual understanding). In addition to assessing student attitudes toward retention and transfer learning skills, the STEP-U measures the degree to which students report having experienced specific teaching approaches that purport to develop these skills. Students’ demographic characteristics have been shown to influence their perspectives about learning (Eddy and Hogan, 2014; Martinho et al., 2015), so we also examined values in the context of demographics (e.g., gender, underrepresented minority group status) and academic factors (e.g., grade point average [GPA], research experience, postgraduate plans). There is a growing consensus that instructors should move from teaching practices that emphasize retention toward practices that promote transfer. A better understanding of student values and how these values are influenced by what students experience in the classroom will provide feedback that is vital in designing curricula that better prepare students for their intended postgraduate careers.

The STEP-U survey is unique in that it provides an opportunity to link students’ classroom experiences (e.g., group work) with their attitudes toward related skills (e.g., working in groups). Many psychological theorists have posited that life experiences shape the individual and his/her values (Pajares, 1992; Bandura, 1997). Science educators have suggested that specific classroom experiences similarly influence students’ attitudes, both positive and negative, toward particular educational knowledge and skills in science classes (Armbruster et al., 2009; Finelli et al., 2014). For example, it has been widely reported that students dread working on group projects because of previous negative experiences with this mode of instruction (Michaelsen et al., 2004). It follows that the more students are exposed to group work in courses, the less they might value working in groups subsequently (Felder, 2007; Finelli et al., 2014; Shekhar et al., 2015). Conversely, it is possible that experiencing certain teaching approaches in the classroom might engender an increased appreciation of the corresponding skills, particularly if assessed near the end of undergraduate degree programs. Evidence-based teaching approaches often require more effort by students (Doyle, 2008; Weimer, 2013), so students may not appreciate the benefits until well after the course has ended. Studies examining student perspectives toward active learning during or immediately following single courses often document high student resistance toward these teaching practices (Henderson and Dancy, 2007; Seidel and Tanner, 2013; Bourrie et al., 2014). However, our experience suggests that student perspectives toward active learning may change over time, and as such, a longer-term assessment approach may be warranted.

In this paper, we concentrated on a subset of items from the STEP-U. We examined the relationship between five specific skills and their corresponding teaching practices in a large sample of biological sciences graduates from a research-intensive university. Specific skills included 1) scientific writing, 2) application of science to everyday life, 3) appreciation of the interdisciplinary nature of science, 4) problem solving, and 5) group work. These skills and teaching approaches are closely aligned with both employer values (Hart Research Associates, 2015) and the recommendations of national science organizations (AAAS, 2011; Association of American Universities [AAU], 2011; PCAST, 2012), providing a meaningful context for investigating student perceptions of their educational experiences.

We considered two research questions:

• RQ1: To what extent do students value transfer versus retention skills, and do student subpopulations differ in the degree to which they value these skills?

• RQ2: To what extent do the values attributed to specific skills relate to student experience with corresponding teaching methods and student demographics?

METHODS

Context of the Study

Our university is a research-intensive university on the East Coast of the United States and enrolls 27,000 undergraduates annually. Students of color comprise 40% of our undergraduate population. The biological sciences major within the College of Computer, Mathematical, and Natural Sciences is collaboratively sponsored by three departments: Biology, Entomology, and Cell Biology and Molecular Genetics. The biological sciences major enrolls 1700 undergraduate students, and more than 90 faculty members teach courses in the program. Students within the biological sciences major choose from among five areas of specialization (cell biology and genetics, ecology and evolution, general biology, microbiology, and physiology and neurobiology), but all students complete a sequence of four courses that provide a common foundation for the major. Following these foundation courses, students pursuing different specialization areas take different course paths. Students can have substantially different classroom experiences even within the foundation sequence. We have a very large number of instructors teaching these courses, and each instructor is free to use whatever teaching practices he or she prefers, ranging from lecture only to highly student-centered, evidence-based teaching practices. With very few exceptions, teaching methods are not standardized or coordinated across the curriculum.

Research Instruments

The STEP-U.

The STEP-U is distinguishable from other surveys that assess student beliefs, attitudes, and practices. For example, the Colorado Learning Attitudes about Science Survey (CLASS-Bio; Semsar et al., 2011) assesses learning attitudes and perceptions about biology, and the degree to which these perceptions resemble those of experts or novices. The Maryland Biology Expectations Survey (MBEX) (Hall et al., 2011) measures student expectations about learning in biology, including expectations about whether biology education involves learning facts versus principles, and isolated versus connected concepts.

The STEP-U is unique because 1) it assesses the values attributed to specific skills relevant to the workplace, 2) these skills are grouped into empirically derived categories, 3) it simultaneously assesses student experience with teaching practices thought to reinforce those skills, and 4) it can be completed by respondents in only 5 minutes, making it easy to combine with other student assessments. Most importantly, the STEP-U allows quantification of how student experiences in the classroom (i.e., group work, problem solving, scientific writing) are associated with the degree to which students value specific educational and professional skills. In essence, while existing surveys explore the attitudes that students hold toward disciplinary learning in the college classroom, the STEP-U allows one to investigate students’ holistic educational experiences and values as they pertain to longer-term educational and professional endeavors.

Toward the end of their final semesters, students in our college complete an online exit survey that asks general questions about their undergraduate education experiences. STEP-U survey items were incorporated into this survey beginning in 2011. STEP-U items constituted 40% of the exit survey and followed questions about co-curricular experiences (e.g., undergraduate research, internships) and postgraduate plans. Students were asked to “Rate the following skills (e.g., memorization, conceptual understanding) in terms of importance to you in your undergraduate education” (where 1 = not important, 2 = slightly important, 3 = fairly important, 4 = important, and 5 = very important; see the Supplemental Material, questions 1–14). Students were also asked about how often they were exposed to different teaching practices in their major courses. Specifically, students were asked, “In the undergraduate courses for your major, how often did instructors use these methods?” (where 1 = none of my courses, 2 = a few of my courses, 3 = some of my courses, 4 = most of my courses, 5 = almost all of my courses; see the Supplemental Material, questions 15–33).

The STEP-U survey is based on the STEP survey that was developed and validated for faculty members in a series of pilot studies (Marbach-Ad et al., 2012, 2013, 2014). In these studies, items for faculty members were created and face validated through iterative processes. Items were also face validated for students. Faculty members from psychology, education, and biology evaluated each item for clarity and meaning. A pilot study was conducted in 2007 (Marbach-Ad et al., 2012), in which item responses were analyzed and revisions to the wording of survey items were made. We also used response process validation procedures through interviews with 11 faculty members (Marbach-Ad et al., 2013). The survey was readministered in 2011 to a larger population that included graduate students, undergraduate students, and faculty members (Marbach-Ad et al., 2014).

Here, we build upon the face validation for the STEP-U using two well-established validity approaches: 1) identification of the internal structure of the test and 2) examination of response process validity (for an explanation of these approaches to establishing validity, see Reeves and Marbach-Ad, 2016). We conducted factor analyses to identify the internal structure of the test items assessing the values students attributed to different skills. We also established internal consistency reliability of the survey through computation of Cronbach’s alpha coefficients (Cronbach, 1951). Internal consistency reliability, calculated as an average of item intercorrelations, is a commonly utilized index of the degree to which test items consistently measure the same psychological construct.

Individual Interviews.

In general, one criticism raised in previous literature (Williams et al., 2015) is that many surveys in science education use a substantial amount of jargon specific to science education research, raising the possibility that respondents may have difficulty understanding the intended meaning of survey items. To address respondents’ understanding of items, we included in the STEP-U validation process an investigation of the degree to which student respondents understood survey terminology. In this regard, we coupled our quantitative approach with a qualitative validation approach. A semistructured interview protocol was used to establish response process validity of the STEP-U survey, defined as “the fit between the construct and the detailed nature of the performance or response actually engaged in by test takers” (American Educational Research Association, American Psychological Association, National Council on Measurement in Education [AERA et al.], 2014, p. 15; see also Reeves and Marbach-Ad, 2016).

In the interviews, students first completed a pencil-and-paper version of the value items of the survey (see the Supplemental Material, items 1–14). Immediately afterward, students were interviewed by a science education researcher and were asked to explain their responses. Generally, students were asked four questions for each survey item: 1) “What do you think about or understand when you read this item?,” 2) “Why did you respond the way that you did?,” 3) “Can you provide an example of this teaching practice?,” 4) “How often did you experience the teaching practice in your undergraduate education?” We did not directly ask students about items 15–33 (teaching practice items). However, questions 3 and 4 helped us to judge students’ understanding of and familiarity with teaching practices corresponding to the value items (see the Supplemental Material, items 15–33) and to further assess their understanding of the value items. Through “think-alouds” (Reeves and Marbach-Ad, 2016), we gauged respondents’ rationalizations and explanations for their written responses. This provided qualitative data on whether or not students understood the intended meaning of each item (see Validity Based on Response Processes under Results). It also provided detailed examples of student experiences that we used to contextualize quantitative data (see RQ2).

Participants

We surveyed biology seniors from 2011 through 2015.¹ The sample was composed of 1389 students who provided complete responses to the survey items. The survey response rate was high (roughly 56% of all graduating biology students). The sample was representative of the biology student population at our university with regard to demographic variables (i.e., gender, cumulative GPA, underrepresented minority [URM] status, research experience and postgraduation plans). URM status included students self-identifying as African-American, Hispanic, Pacific Islander, or Native American. Research experience was defined as participating in either an on- or off-campus research experience during undergraduate study, based on student self-report.

Students were asked to select one of five options for their postgraduation plans: 1) enroll in a degree-granting program in the next year following graduation; 2) enter the workforce and do not anticipate future enrollment in degree-granting program; 3) enter the workforce or pursue a postbaccalaureate training program with the goal of future enrollment in a degree-granting program; 4) I’m not sure what I am going to do in the future, but I plan to stay in a field related to my degree; or 5) I’m not sure what I am going to do in the future, but I am fairly certain it will be in a field unrelated to my degree. Preliminary analyses showed that survey responses consistently fell into two well-defined clusters, and so we coded postgraduate plans as a dichotomous variable composed of options 1 and 3 (intending to enroll in a degree-granting program in the immediate or near future) and options 2, 4, and 5 (entering the workforce or uncertain of future plans, but not intending to enroll in a degree-granting program).

Table 1 illustrates demographic characteristics of the sample by graduation year.

In addition to the survey sample, five biology students were interviewed to provide qualitative data. Characteristics and pseudonyms of these students are provided in Table 2. The interviewed students were representative of biological sciences graduates. Around 90% of graduating students in their senior year aspire to pursue graduate studies or professional degrees, with a large majority hoping to enter the health professions (e.g., medicine, dentistry, physician’s assistant, pharmacy). About 60% of graduates are female. Enrollment in the biological sciences degree program is limited to those who have successfully completed gateway science courses, so GPAs are fairly high on average.

All participants provided informed consent to participate. The study was approved by the Institutional Review Board (IRB protocol 375954-5).

Data Analysis

Following validity analyses (see Results), we analyzed the degree to which students value transfer versus retention skills using a paired-samples t test (see RQ1). We also examined whether or not student subpopulations differ on their values of retention and transfer skills using multiple linear regression models (MLR; see RQ1). Next, we examined the relationship between values of unique skills and corresponding teaching methods used in class through a series of MLR models (RQ2). We explored five specific skills queried in the STEP-U survey: 1) scientific writing, 2) application of science to everyday life, 3) understanding the interdisciplinary nature of science, 4) problem solving, and 5) group work. Each of these skills corresponds to a related teaching practice (e.g., use of group work in class, use of scientific writing assignments).

Given that student characteristics influence their perspectives about learning (Eddy and Hogan, 2014; Martinho et al., 2015), we entered teaching practice as a predictor in each regression model along with the following student characteristics: gender, URM status (dichotomous, yes or no), cumulative GPA at graduation, research experience (dichotomous, yes or no), postgraduate plans (dichotomous, defined as students who were planning to pursue an advanced degree in the immediate or near future vs. students who planned to enter the workforce or students who were undecided about their postgraduate plans). The dependent variable in each model was the value students attributed to the skill. Initial analyses showed that, across skills and teaching practices, there was no effect of graduation year (i.e., the extent to which students valued particular skills and their experiences with specified teaching practices did not increase or decrease significantly over time). Therefore, graduation year was not included in regression models. All statistical analyses were conducted in SPSS version 22.

For qualitative analysis of the interview data, a graduate student from the College of Education and a science education faculty member analyzed the responses separately. For each interview transcription, reviewers assessed three components for each STEP-U item (see Individual Interviews under Research Instruments earlier in this section). First, reviewers assessed whether or not the student understood the meaning of the item as conceptualized by researchers. If the meaning students attributed to the item did not match the meaning intended by survey developers, the discrepant meaning was recorded. Second, reviewers examined the degree to which the student valued the skill, looking at consistencies and discrepancies between their written and verbal responses, and they looked for major themes in students’ explanations for their ratings. Third, reviewers examined the degree to which students experienced related teaching practices in class, along with major themes emerging from their responses. Following this process, reviewers together discussed which items were misunderstood by students, major themes related to why students value or devalue skills, and major themes related to students’ experience with various teaching practices in the classroom. Interpretation of the data was discussed until they came to agreement.

RESULTS

Validity of the STEP-U Survey

Exploratory factor analyses were conducted to identify the internal structure of the test items assessing the values students attributed to different skills. Validity evidence based on internal structure of tests concerns “the degree to which the relationships among test items and test components conform to the construct on which the proposed test score interpretations are based” (AERA et al., 2014, p. 16; see also Reeves and Marbach-Ad, 2016). We performed exploratory principal axis factoring with oblique rotation in accordance with recommended best practices in the social sciences (Worthington and Whittaker, 2006). Oblique rotation methods were selected because we anticipated that the factors might be correlated. The Kaiser-Meyer-Olkin measure of sampling adequacy was adequate (0.893), and Bartlett’s test of sphericity was significant (p < 0.001). Results of parallel analyses (Horn, 1965; Patil et al., 2008) indicated that a two-factor solution was statistically appropriate. We also examined one- and three-factor solutions, which yielded relatively unsatisfactory solutions as compared with the two-factor solution, due to the presence of several cross-loadings, low loadings, and poor conceptual interpretability. The two factors, which were correlated in the small to moderate range (r = 0.26), explained 51.76% of the variance in the item data (eigenvalues = 5.6 and 1.6, respectively).

Items loading onto factor 1 appeared to reflect skills pertaining to meaningful learning (e.g., conceptual understanding), whereas items loading onto factor 2 seemed to reflect skills related to lower-level skills (e.g., memorization). These factors are conceptually consistent with Mayer’s (2002) way of thinking about skills in association with Bloom’s taxonomy (1984). Mayer (2002) differentiated between two major categories of learning, retention, which encompasses memorization and remembering information, and transfer, which refers to deeper learning processes such as applying material to novel situations and analyzing data. The two emergent factors were interpreted to represent these two psychological constructs. Factor loadings, item means, and SDs are illustrated in Table 3.

Internal Consistency Reliability

To calculate internal consistency reliability of subscales, we computed Cronbach’s alpha coefficients for the retention and transfer subscales. Internal consistency for the retention factor was 0.69, and for the transfer factor 0.89. These results indicate acceptable levels of internal consistency reliability for each of the subscales (see Streiner, 2003).

Validity Based on Response Processes

Validity evidence based upon response processes was examined qualitatively. Interview data suggested that students understood the meaning of 12 of 14 items. For example, biology students were aware that the dynamic nature of science referred to the understanding that scientific discoveries change over time. Students also showed understanding of apply quantitative reasoning, giving examples of how they valued the ability to use mathematical and statistical knowledge to solve biological problems. There were two items that caused confusion for two of four interviewed students: 1) develop understanding of the interdisciplinary nature of science, and 2) develop information literacy. When asked about the interdisciplinary nature of science, one student discussed the diversity of topics within biology (e.g., genetics and virology), while another student had difficulty articulating thoughts about the meaning of the interdisciplinary nature of science. When asked about information literacy, two of four students reported that they did not understand the question. We revised these two items to enhance clarity. Quantitative data analyses have not yet been conducted on the revised items, as these revisions will be included in future administrations of the STEP-U. For the interdisciplinary item, the item now reads: develop understanding of the interdisciplinary nature of science (e.g., how biology relates to chemistry, how physics relates to biology). We also modified the corresponding teaching item (see the Supplemental Material). The develop information literacy item now reads: develop information literacy (e.g., being able to understand articles about science).

RQ1. To What Extent Do Students Value Transfer versus Retention Skills, and Do Student Subpopulations Differ in the Degree to Which They Value These Skills?

Subscale scores for retention and transfer were created by computing mean scores for each student on each subscale. Overall, students valued transfer skills (M = 4.13, SD = 0.57) more than retention skills (M = 3.54, SD = 0.91), as indicated by a paired-samples t test (t(831) = 17.75, p < 0.0001).

MLR analyses were conducted to examine whether student subpopulations differed in the degree to which they valued transfer and retention skills. URM status, gender, and GPA were entered as predictors in two separate regression models, with the value attributed to transfer and retention, respectively, as the dependent variable. GPA, but not gender or URM status, was significantly associated with the extent to which retention skills were valued. Students with lower GPAs placed a higher value on retention skills (b = −0.16, t(1115) = −2.47, p < 0.05). Student subpopulations did not differ significantly in the extent to which they valued transfer skills.

RQ2: To What Extent Do the Values Attributed to Specific Skills Relate to Student Experience with Corresponding Teaching Methods and Student Demographics?

Figure 1 shows the percentage of students who placed a high value (i.e., rated as important or very important) on each of the STEP-U skills. Students placed a high value on conceptual understanding (92% rated as important or very important) and problem solving (89%). Substantially fewer students placed a high value on group work (45%) and remembering formulas, structures, and procedures (46%).

Figure 2 shows the percentage of students who rated specific teaching practices as occurring in some, most, or almost all of their undergraduate courses. Almost all of the students surveyed (93%) reported that they experienced extensive lecturing in some, most, or almost all of their courses. One of the least commonly used teaching practice of those examined here was group work, with only 57% of students reporting that they experienced group work in some, most or almost all of their courses. There were two teaching practices (debates and games) with smaller percentages.

To assess the relationship between classroom experiences and values, we focused on five key skills and corresponding teaching practices: 1) scientific writing, 2) application of science to everyday life, 3) appreciation of the interdisciplinary nature of science, 4) problem solving, and 5) group work. Figure 3 shows the general relationship between extent of experience with each of these teaching practices and the average value attributed to the corresponding skill. Greater exposure to a specific teaching practice was consistently associated with placing a higher value on the corresponding skill.

We used MLR models to examine the relationship between classroom experiences, values, and student demographics. In each model, the extent to which students reported experiencing a particular teaching practice was entered as an independent predictor of the value students attributed to corresponding skills, along with the following demographic variables: GPA, gender, URM status, postgraduate plans, and participation in undergraduate research (Table 4).

Scientific Writing.

Sixty-eight percent of students reported that they experienced scientific writing in some, most, or almost all of their courses in the biological sciences major (Figure 2). This finding was corroborated by qualitative data, as interviewed students indicated that they were exposed to scientific writing mainly through laboratory reports in biology or in professional writing classes (Annie and Jeremy [pseudonyms]; see Table 2). Only 66% percent of students surveyed placed a high value on scientific writing, which is low in comparison with other surveyed skills (Figure 1). Jeremy explained that scientific writing is an essential skill to have as a scientist: “I think since we are in the science field, I think it’s essential to be exposed and learn how to write and learn how to understand. Each week we get exposed to scientific writing in [course name]. I think it ties in what we learn in class to research that’s being done, to help us further enhance our learning. So I think it’s pretty important.”

MLR results (F(51104) = 12.15, p < 0.001, R² = 0.06) indicated that previous experience with writing assignments positively predicted the value students attributed to writing skills (b = 0.21, t(1104) = 6.78, p < 0.001). Research experience positively predicted the value students attributed to scientific writing skills (b = 0.24, t(1104) = 3.77, p < 0.001), such that students who had participated in undergraduate research valued scientific writing to a greater extent than those who had not participated in research. In the interviews, Michelle explained why scientific writing might be of greater interest to those seeking research-oriented careers, “I think it depends on the field, I think that if you’re going into research, it’s more applicable to what you’re going to be doing. If you’re going into medicine or some other health related field, I don’t think writing is as important.” Travis, an aspiring medical student, explained that he highly values scientific writing because he enjoys being able to “express things objectively, removing biases from your work, and being able to write concisely and succinctly.” He worked in a research laboratory on our campus and acknowledged that the most meaningful writing experience he received was through this experience, during which he collaborated in writing peer-reviewed papers.

Postgraduate plans were also associated with valuing scientific writing. Specifically, those students who were planning to pursue an advanced degree placed a higher value on scientific writing than students who planned to enter the workforce and those who were undecided about their postgraduate plans (p < 0.01). GPA was inversely related to the value attributed to scientific writing (p < 0.01). The remaining predictors (gender and URM status) were not significant.

Relating Science to Everyday Life.

A high percentage of students (84%) rated understanding how science applies to everyday life as an important or very important skill. A similarly high percentage (81%) reported that their instructors related course material to the real world during some, most, or almost all of their courses. Michelle explained how real-life examples were used in her classes and why she valued this skill: “The classes that I’m in now, like they have anatomy, they’ll relate things that we learn to things that we actually see, which is pretty cool. My endocrinology class, the exams are very diagnostic, the instructor will say here’s a patient, here’s the symptoms, how do the symptoms relate to what disease.” She felt that this skill was important to her and those intending to enter the health professions. Jeremy also discussed why he rated relating science to everyday life as important, “because [you] learn why certain things happen, let’s say for example the biology of cancer. I want to understand why cancer happens.… Or epidemiology, like certain countries that are experiencing various epidemics, diseases. How can we control that in everyday life?”

Results from the MLR model (F(61105) = 13.24, p < 0.001, R² = 0.07) indicated that the more frequently students were exposed to examples relating course material to the real world in their courses, the more highly they valued understanding how science applies to everyday life (b = 0.19, t(1105) = 7.40, p < 0.001). For example, students who reported that they were taught with real-life examples in almost all of their courses valued these skills at a very high level (M = 4.7, see Figure 3). Jeremy noted that his classes for biological sciences majors focused more on teaching with everyday life examples than other classes he had taken. He gave an example for such a class, “our virology class really ties in our understanding, for example, why we get vaccinated, and that ties in with current events as well.”

Female students placed a higher value on understanding how science applies to everyday life than did males (b = 0.17, t(1105) = 3.37, p < 0.01). Postgraduate plans were also associated with valuing understanding how science applies to everyday life. Specifically, those students who were planning to pursue an advanced degree valued this to a greater extent than students who planned to enter the workforce and those who were undecided about their postgraduate plans (p < 0.001). The remaining predictors (URM status and research experience) were not significant.

Understanding the Interdisciplinary Nature of Science.

A large percentage of students (80%) rated developing an understanding of the interdisciplinary nature of science as an important or very important skill. A similarly high percentage (75%) reported that their instructors taught with an interdisciplinary approach during some, most, or almost all of their courses. Regarding predictors of the extent to which undergraduates valued understanding the interdisciplinary nature of science, MLR results (F(6840) = 9.5, p < 0.001, R² = 0.06) indicated that the more frequently instructors taught with an interdisciplinary approach, the more highly students valued understanding the interdisciplinary nature of science (b = 0.23, t(840) = 6.65, p < 0.001). Students who reported that they were taught with an interdisciplinary approach in almost all of their courses placed a very high value on interdisciplinary understanding (M = 4.7; Figure 3).

In the interviews, Annie discussed why she thought that understanding the interdisciplinary nature of science was very important, “I realize how important it is. Subjects are all interconnected, and I see that’s what they’re doing with [the redesigned] physics [course]. They’re trying to connect it with biological and chemical principles.” She stated that in high school and middle school the disciplines (e.g., biology, chemistry) were taught separately, and she described a shift in her thinking when she entered the university: “I noticed that I took chemistry my first semester, I didn’t take any biology courses, and I thought chemistry is separated from biology until my TA said, no they’re all connected, and then I started to realize that as the semesters went on.” Michelle also rated understanding the interdisciplinary nature of science as important, and she felt that instructors of the courses she had taken taught with this approach, stating, “Physics was very biology based, for example.”

The MLR also showed that there was a relationship between research experiences and valuing interdisciplinary understanding, with those students who had engaged in undergraduate research valuing interdisciplinary understanding more than those who had not engaged in research (b = 0.12, t(840) = 1.93, p = 0.05). In addition, students who were planning to pursue an advanced degree valued interdisciplinary understanding to a greater extent than students who planned to enter the workforce and students who were undecided about their postgraduate plans (p < 0.01). The remaining predictors (GPA, gender, and URM status) were not significant. The interpretation of these results is somewhat complicated by the fact that two interviewed students interpreted this survey item differently from the others and from the survey developers, thinking that interdisciplinary connections referred to the relationship between biology subdisciplines (e.g., virology and genetics).

Problem Solving.

There was a high consensus among students (i.e., 89%) that problem solving is an important or very important skill. This skill was the second highest rated, following conceptual understanding. Sixty-eight percent of students reported that their instructors used inquiry-based learning (e.g., problem-based learning, case studies) during some, most, or almost all of their courses. MLR results (F(6840) = 3.1, p < 0.001, R² = 0.02) indicated that the more frequently instructors taught using inquiry-based approaches, the more highly students valued acquiring problem-solving skills (b = 0.09, t(840) = 2.97, p < 0.01; Figure 3). Students who were planning to pursue an advanced degree valued problem solving to a greater extent than students who planned to enter the workforce and those who were undecided about their postgraduate plans (p = 0.05). The remaining predictors (GPA, gender, URM status, and research experience) were not significant.

Three of five students interviewed rated problem-solving skills as very important. Ava discussed how a single course, mammalian physiology (an upper-level course), was particularly influential in fostering an appreciation of problem-solving skills. She also emphasized the value of these skills both for the workplace and for college exams, “this one [problem solving] I thought was really important, just because it goes into things that we really need after we graduate college, and I think it’s something that’s used a lot during the exams too.” Jeremy mentioned his professors’ use of clickers in lecture classes:

Michelle, who rated problem solving as an important skill, noted that she was asked to do problem-solving exercises “more in upper level than lower level courses … if they present you with a situation, you’ll have to take what you’ve learned and apply it to the situation.”

Group Work.

Only 57% of students reported that they experienced group work in some, most, or almost all of their courses in the biological sciences major, placing group work among the least commonly used teaching approaches (Figure 2). Relatively few students (45%) placed a high value on acquiring group-work skills as compared with other skills (Figure 1). The MLR model (F(6, 1106) = 10.01, p < 0.001, R² = 0.05) indicated that students who experienced group-work activities more frequently during their undergraduate education valued group-work skills to a greater degree than those who experienced group work less frequently (b = 0.28, t(1106) = 7.1, p < 0.001; Figure 3). The remaining predictors (GPA, gender, URM, postgraduate plans, and participation in undergraduate research) had no significant relationship with the value students attributed to group work.

Our qualitative data provide insight into student perceptions of this teaching practice. Annie discussed how positive group-work experiences impacted her: “I enjoyed when I was paired with someone else, because there was more than one perspective, and if I was unsure about some direction I could ask my partner or partners.” Jeremy commented, “I believe it’s a good skill to be able to work with different people, to be able to talk, be able to communicate in certain aspects.” Travis acknowledged the importance of this skill for the workplace, and felt that it would be easier to work in groups in the workplace: “Overall in college, group projects, … it’s such a difficult thing when everyone’s schedules are so difficult. In a career, you all work for the same job, you have the same hours and the same responsibilities.”

The three interviewees with the highest GPAs added additional reasons as to why group work could be problematic. Michelle commented, “If I work in groups too early, it’s a little distracting. And I think it kind of hurts me, understanding the material.” Ava explained, “I think in group work … ideally, everyone will be carrying out the work equally, but we all know that’s not how it actually happens … [Group work] is an important component, but not to be overly emphasized in a curriculum.” Travis explained the variable group-work experience he encountered in his studies, “I’ve had ones where people did absolutely no work, and I had to do everything. There were groups where people were willing to do the work, but were completely incompetent. I’ve had groups where people were on top of their stuff, and it was great.”

DISCUSSION

We developed and validated an assessment tool, the STEP-U, which measures the value that students attribute to transfer and retention skills and their experience with teaching practices thought to reinforce those skills. This tool has multiple potential applications. It could be used to characterize student attitudes and experiences between cohorts, within subdisciplines of biology, or between majors. It could also be used to document long-term shifts in student perceptions associated with broad-scale curricular change.

We demonstrate the utility of the STEP-U using a large, multiyear data set. Our study differs from previous studies in that it links student values and experiences integrated over the entire undergraduate degree program, rather than focusing on a single course or year of study. We show that the extent to which students value particular skills is predicted by demographic and experiential factors.

We did not observe any changes in student experiences or values between 2011 and 2015. This result was disappointing, but not entirely unexpected. Previous course and curriculum redesign efforts in the biological sciences at our university have been relatively isolated, rather than coordinated through the curriculum. Only within the past 2 years has substantial funding become available to biological sciences faculty to de-emphasize lecture and incorporate more evidence-based teaching approaches into their courses. There are also new teaching facilities becoming available to support student interactivity and the integration of teaching technologies. As such, our current data serve as a baseline, and we expect to see shifts in student experiences and values in the coming years.

Encouragingly, biology students in our sample valued transfer (i.e., higher-order thinking) skills to a greater extent than retention skills. However, we observed a significant relationship between the value attributed to retention skills and student GPA, such that students with lower GPAs valued retention skills more than those with higher GPAs. This is consistent with previous research showing that lower-achieving students have study habits that rely predominantly on rote memorization (Biggs, 1987), which is a less effective learning strategy than focusing on knowledge application (Mayer, 2002).

Multiple studies, particularly those based on end-of-semester student course evaluations, report that students show resistance to active-learning instructional approaches (Felder, 2007; Finelli et al., 2014; Shekhar et al., 2015). On the basis of this, one might predict that students with greater exposure to active learning in their classes would be more resistant to active learning and consequently would attribute less value to skills associated with active learning. We found the opposite. For all skills examined, there was a positive relationship between valuing a skill and the frequency with which students experienced the related teaching practice. The more students experienced a particular teaching practice (e.g., group work in class), the more they valued the corresponding skill (e.g., ability to work in groups) at graduation. This suggests that repeated exposure to active-learning teaching methods may overcome initial student resistance. It also suggests that, to cultivate an appreciation for skills valued by employers, teaching approaches that reinforce those skills should be imbedded throughout the undergraduate curriculum.

Our results are correlational, however, and there are other competing explanations for the findings worth considering. First, students who appreciate a particular skill may be more likely to recall having experienced corresponding teaching practices. Future studies could address this possibility through the use of observational methodologies rather than relying on student self-report. Specifically, researchers could use classroom observations to characterize the degree to which various teaching practices are used in specific classes and subsequently relate these data to student values. Second, values and recollection of exposure to particular teaching practices could be correlated to a third, unmeasured variable. Future studies examining psychosocial variables and their relationships to student values would be illuminating in this regard.

In addition to the global relationship between teaching experience and student values, there were interesting findings with regard to student characteristics.

Female students valued applying science to everyday life more than male students did. These findings are in accord with Heffler (2001), who showed that females tend to prefer concrete, experiential modes of learning as compared with males, who prefer traditional learning modes. Because females are underrepresented in many STEM fields (National Science Foundation, National Center for Science and Engineering Statistics, 2015), the relationship between gender, learning styles, and teaching methods in science is an important topic for future research.

Students who reported participating in undergraduate research placed greater value on scientific writing and interdisciplinary understanding. Students who are involved in research may gain a deeper understanding of the interdisciplinary nature of science by participating in interdisciplinary research teams. Participation in research may also foster greater appreciation for scientific writing, because students can see firsthand how scientific studies are extensions of previously published work, and they may even get to help write the results of their research for publication. Faculty-mentored undergraduate research has been defined by the National Survey of Student Engagement report (NSSE, 2007) as a “high impact practice” and is associated with students feeling more prepared for their postgraduate endeavors. Our study adds to the growing list of benefits students can derive from research participation (NSSE, 2007).

Students who reported plans to pursue an advanced degree placed greater value on scientific writing, relating science to everyday life, interdisciplinary understanding, and problem solving. It is concerning that those who intend to enter the workforce sooner are less appreciative of the skills that are highly valued by employers (Hart Research Associates, 2015). This raises the possibility that our academic settings are optimized for students who plan to remain (at least for the time being) in academia, rather than for students who plan to enter the workforce directly upon graduation. It would be prudent to assess teaching methods and contexts used for instruction and instructor explanations of the relevancy of each. Medical examples and contexts are commonly used in biological sciences course work, while comparable examples from industrial and regulatory settings may be less prevalent. It could be that we are inadvertently biasing our instruction to those who intend to pursue graduate and professional degrees. This may subtly affect the attitudes of those who anticipate entering the workforce after completing a bachelor’s degree.

In an overall sense, students valued some skills substantially more than others. Group work was almost universally underappreciated by students. In contrast, problem-solving skills were valued by almost all students in our sample. In the following sections, we discuss group work and scientific writing skills in greater depth, since these skills were ranked as less important by students and are of high value to employers.

Group Work

Our results suggest that the ability to work in groups is one of the skills that students perceive as least valuable in their undergraduate education. Students in our interview sample recognized that group work can be problematic because of the possibility of unequal workload distribution and the perception that group work can interfere with taking ownership of one’s learning. The literature documents many reasons for student resistance to group work, often arising from previous negative experiences due to poorly designed group activities, such as inequitable distribution of workload or leadership, poor timing, and assigning overall group grades that fail to reward individual student efforts (Michaelsen et al., 2004; Weimer, 2013; Shekhar et al., 2015). In our data, some students voiced an understanding of the helpfulness of group work in fostering a collaborative learning environment. The appeal and effectiveness of group work can be enhanced using specific strategies that facilitate cohesiveness, the purposeful creation of permanent and heterogeneous student groups, and holding students accountable for their individual contributions within the group (Michaelsen et al., 2004).

Given the importance of group-work skills for the workplace and current employer perceptions that graduating students lack adequate collaborative skills (Hart Research Associates, 2015), these findings are highly relevant for teaching practice. Instructors may have the potential to cultivate student values regarding group work by using carefully planned group activities in their classrooms. Our data showed that those students who experienced more group work from their instructors eventually reported greater appreciation for group-work skills. Therefore, administrators and instructors should promote a climate of interactive, group work–oriented learning throughout undergraduate curricula.

Scientific Writing

Scientific writing is very important to employers, with 81% of employers (N = 400) reporting they would be more likely to consider a job candidate if the applicant had completed multiple university courses involving significant writing projects (Hart Research Associates, 2015). In our sample, we found that only 68% of students reported that they had scientific writing assignments incorporated into more than a few of their courses in the biological sciences major. This is consistent with findings from a previous study of biology faculty members, which showed that, although a high proportion of faculty valued scientific writing, only one-third reported that they incorporated scientific writing assignments into their classes (Marbach-Ad et al., 2014). In a related study, 21 biology faculty members who were interviewed about scientific writing instruction were divided in their opinions as to whether scientific writing is best taught in science courses or in courses offered by English professors (Marbach-Ad and Arviv-Elyashiv, 2005). Many felt that they were less well suited to teach this skill as compared with expert writing instructors. In addition, they pointed out that the large-enrollment classes that typify many courses within the biological sciences major lack sufficient human resources to give detailed, constructive feedback on student written assignments.

Recent studies examining teaching interventions that incorporate scientific writing instruction in undergraduate biology (e.g., Freeman et al., 2007; Quitadamo and Kurtz, 2007; Linton et al., 2014; Tonissen et al., 2014) have shown that such interventions effectively increase students’ writing confidence and competence. Students in our sample varied in their appreciation for scientific writing skills based on their prior research experience and postgraduate plans. However, scientific writing is an important skill for a variety of occupations, and our findings suggest that students would benefit from having multiple opportunities to refine this skill within the biological sciences curriculum, despite the difficulties associated with embedding writing across the curriculum.

FUTURE DIRECTIONS

The STEP-U is a novel assessment tool validated in a large sample of students majoring in the biological sciences at a large, research-intensive university setting. We demonstrate how this survey tool was used to gather information on student values and experiences that is now informing curricular and pedagogical reform at our institution. We believe the STEP-U has wide applicability to biology departments that seek to better understand how student values of skills needed for the workplace change over the course of undergraduate education and are influenced by educational reform initiatives. We recommend that the STEP-U be administered in conjunction with a qualitative approach (e.g., interviews, classroom observation) to provide a richer, more fully faceted characterization.

It would be valuable to replicate the present validation in biology departments at multiple institutions, as the present study was conducted at a single institution only. It would also be of value to compare biology departments across institutions of differing types to examine how differences in institutional missions and student characteristics impact student values. Given that biology encompasses a large array of subdisciplines (e.g., physiology, ecology, microbiology), it would also be of interest to probe student values and experiences within subdisciplines at those institutions where students can specialize in a particular subdiscipline.

We are currently validating the tool for use across STEM disciplines, both in terms of factor structure and relevance of items for students. For example, the item understand the dynamic nature of science may be less relevant to math majors. Once validated across STEM disciplines, the STEP-U could then be used to examine department-specific patterns in student values and educational experiences.

FOOTNOTES

ACKNOWLEDGMENTS

REFERENCES