Insight from Biology Program Learning Outcomes: Implications for Teaching, Learning, and Assessment
Abstract
Learning goals and objectives are a key part of instruction, informing curricular design, assessment, and learning. These goals and objectives are also applied at the programmatic level, with program learning outcomes (PLOs) providing insight into the skills that undergraduate biology programs intend for their students to master. PLOs are mandated by all major higher education accreditation agencies and play integral roles in programmatic assessment. Despite their importance, however, there have not been any prior attempts to characterize PLOs across undergraduate biology programs in the United States. Our study reveals that many programs may not be using PLOs to communicate learning goals with students. We also identify key themes across these PLOs and differences in skills listed between institution types. For example, some Vision & Change core competencies (e.g., interdisciplinary nature of science; connecting science to society; quantitative reasoning) are highlighted by a low percentage of programs, while others are shared more frequently between programs. Similarly, we find that biology programs at 4-year institutions likely emphasize PLOs relating to computational skills and research more than at 2-year institutions. We conclude by discussing implications for how to best use PLOs to support student learning, assessment, and curricular improvements.
INTRODUCTION
Learning goals and objectives, which convey the intended knowledge, skills, and processes that a student should be able to accomplish at the end of a given curricular unit, play a critical role across disciplines in both teaching and learning (Marzano, 2010). For instance, a commonly used approach for creating curricula is backward design, wherein instructors first identify the learning goals and objectives for students before designing and implementing lessons that promote student mastery of such learning goals and objectives (Reynolds and Kearns, 2017; Matsuda, 2020; Neiles and Arnett, 2021). Learning goals and objectives also play important roles for asssessment, with instructors encouraged to align items on assessment instruments (e.g., quizzes and tests) to learning goals and objectives in order to determine student progress and mastery of learning goals and objectives (Torrance, 2007; Combs et al., 2008; Towns, 2010).
In addition to informing instructional practices, learning goals and objectives can also play major roles in shaping student learning. For instance, the addition of learning goals and objectives to a simulated set of readings and activities in a biology class improved student learning (Sana et al., 2020), and learning goals and objectives have also been shown to influence student study strategies in biology and other disciplines (Osueke et al., 2018; Stanton et al., 2019; Barnard et al., 2021). This work has demonstrated that students find utility in using learning goals and objectives to shape their studying and likewise that such learning objectives can positively impact student learning (Simon and Taylor, 2009; Fata-Hartley, 2011; Minbiole, 2016; Osueke et al., 2018).
Given the importance of learning goals and objectives for both instructors and students, there has been a substantial amount of literature examining best practices on writing and implementing learning goals and objectives in the classroom (Simon, 2006; Marzano, 2010; Redelius et al., 2015; Chatterjee and Corral, 2017). Within biology, there have been several calls for instructors to make learning goals and objectives explicit and to align assessments with them (Allen and Tanner, 2006; Ewell et al., 2022).
However, nearly all the literature on learning goals and objectives is situated at the course level, which includes work examining course learning goals and outcomes or the impact of learning outcomes for a specific class activity. In contrast, there has been almost no work that has examined the use of learning outcomes at the program and degree level. These program learning outcomes (PLOs) convey goals, objectives, competencies, and outcomes for students enrolled in a given degree program, with course learning outcomes that should align with and support student development of these PLOs (Orr et al., 2022a,b). PLOs can be used to create and update curricular maps, design and implement assessment plans, and refine curriculum and co-curricular supports based on the results of assessment (Towns, 2010; Johnson and Envick, 2014; Denicoló, 2019). In addition, PLOs communicate the main goals of a degree to current and prospective students of the program, as well as instructors, staff, parents, and other stakeholders involved in the degree program (Praslova, 2010).
Given this gap in knowledge, we conducted an exploratory study to characterize undergraduate biology program PLOs across the United States. Our research addressed the following questions:
To what extent are PLOs publicly available for undergraduate biology programs in the United States?
What skills, competencies, and knowledge are undergraduate biology programs listing as goals in their PLOs, and are there differences by institution type (as defined by Carnegie Classifications)?
What is the cognitive level of the skills and competencies that undergraduate biology programs list in their PLOs, and are there differences by institution type?
Our work is therefore designed to provide a first characterization of the skills and competencies listed as PLOs across undergraduate biology programs in the United States. In addition, we situate our work in the context of Vision & Change, a national call for transforming undergraduate biology education (American Association for the Advancement of Science [AAAS], 2011). This report included a list of core concepts and competencies for biology programs, with multiple guides and tools developed based upon these principles (Brownell et al., 2014; Brancaccio-Taras et al., 2016; Couch et al., 2019; Clemmons et al., 2020). Our study thus provides insight into how well biology PLOs reflect the framework proposed in Vision & Change. In addition, we also examine whether there are differences in PLOs between institution types, given that past work has found different norms between institution types, differences in course modalities offered (e.g., proportion of online courses), and also differences in what criteria biology programs at different institution types use in hiring faculty (Fleet et al., 2006; Varty, 2016; Tibbetts et al., 2018).
Use of Terminology: Learning Goals, Objectives, and Outcomes
The terms “learning goals,” “objectives,” and “outcomes” are all used in the literature, though these terms are nuanced and can convey different meanings. For instance, learning goals tend to be broader, more general statements of a given purpose of a course or program, while learning objectives and outcomes present more specific, actionable skills or results (Hartel and Foegeding, 2004). However, there remains ambiguity on the definitions and nuances between these terms, and there is no consensus among the major higher education accreditation agencies in the United States concerning the terminology used (Mitchell and Manzo, 2018; see Table 5). We also acknowledge that there likely exists variation among institutions and biology programs in how they refer to and conceptualize these program learning goals, outcomes, and objectives. However, we choose to use the term “program learning outcome” or “PLO” in our work to refer to any set of goals, objectives, or competencies that an undergraduate biology degree program lists for its degree. We choose this term because of its use in past literature (Denicoló, 2019) and because the term “outcome” is the term most commonly referenced among the major accreditation bodies (Table 5). Similarly, we incorporate a broad, inclusive definition of this term, given our goals of characterizing what undergraduate biology programs are listing and communicating to students and faculty about different programs’ intended aims.
Positionality
Statements of positionality are increasingly common in education research in order to acknowledge the impact that the identities of the authors may play in the research process (Secules et al., 2021). We thus include a summary of our previous experiences and identities that may inform and shape our perspectives on this work. N.C. is an undergraduate biology student who currently serves as a supplemental instructor for introductory biology, where she has worked with course learning objectives, which has shaped her teaching strategies. J.L.H. is a pre-tenure faculty member in biology who was trained in evolutionary genetics but is now active in biology education research. J.L.H. has incorporated learning objectives into his courses and currently serves as chair of his university’s assessment committee, where he helps review PLOs and assessment reports across the university.
METHODS
Sampling of Colleges and Universities
We first conducted a stratified random sampling scheme of colleges and universities in the United States, using the Carnegie Classifications to ensure that diverse institutions were represented in our sample (McCormick and Zhao, 2005). For the initial creation of a codebook (see “Identification of Undergraduate Biology Programs”), we first generated a random list of 61 colleges and universities with undergraduate biology programs, chosen as an initial, manageable subset that would allow both authors to collaboratively examine and discuss these PLOs. In addition, we ensured even representation across the four basic Carnegie Classification categories (doctoral universities, master’s colleges and universities, baccalaureate colleges, and associate’s colleges). Given the potential variation within each of these categories, we further substratified to ensure that there was likewise roughly equal coverage across each of the Carnegie Classification subcategories (e.g., doctoral universities are subdivided into universities with very high research activity, high research activity, or doctoral/professional universities). The Carnegie Classification provides a separate category of baccalaureate/associate’s colleges, which we grouped with baccalaureate colleges, and we did not sample from any institutions listed under special focus or from Tribal Colleges and Universities, given the unique status and/or nontraditional focuses of the schools in these categories. Following the development of our initial codebook and our check for interrater reliability (described in the Analysis of key themes in PLO section below), we continued our stratified random sampling until we had sampled 305 colleges and universities with undergraduate biology programs. This number was chosen for several reasons: first, it represents more than 10% of all colleges and universities in the United States listed in the sampled Carnegie Classification categories (n = 2817), providing us a broad, representative snapshot of all colleges and universities for our exploratory study. In addition, we compared the themes that emerged from these PLOs from our initial sampling subset (n = 61) with our final sample (n = 305). This comparison determined that there were no additional themes emerging in the larger subset and similarly very little variance in the results from the smaller to larger sample size, suggesting that we had reached theoretical saturation (Guetterman, 2015).
Identification of Undergraduate Biology Programs
For each institution sampled, we searched for degree-granting undergraduate biology programs. This was done by initiating a Web search of the institution name plus certain keywords, such as “biology major,” “biology program,” and “biology degree.” We included any institution with a general biology degree program, which encompassed both bachelor’s and associate’s degrees. Some institutions had both degrees in a subdiscipline of biology (e.g., a degree in molecular and cellular biology) and a degree in general biology. To ensure consistency in our comparisons of PLOs, we limited our comparison to PLOs of general biology programs. If an institution did not have a biology program, we excluded it from our data and randomly selected another institution from within the same Carnegie Classification to replace that institution, thus ensuring that our sample set of 305 institutions each has a general undergraduate biology program. Overall, only eight (2.6%) of the initially sampled institutions did not have general biology programs. Similarly, if an institution offered both a bachelor of science and a bachelor of arts in biology, we only included the PLOs from the bachelor of science degree.
Identification and Availability of PLOs
For each program in our data set, we recorded the type of degree awarded for the undergraduate biology program (i.e., bachelor’s or associate’s) and then searched for its biology PLOs. This was done by first visiting the home page of the biology department or program, if available, and searching for PLOs listed either on the home page or on associated pages linked to from the home page. If no PLOs were identified on these pages, we conducted a general Web search with the name of the institution and degree program followed by a keyword search for PLOs. Finally, if this still did not yield any results, we searched the institutions’ academic or course catalogues for PLOs, given that catalogues are a common place for academic policies, degree requirements, and descriptions (and thus PLOs) for programs (Vavolizza, 2010). We categorized the availability of PLOs based on whether PLOs were posted on the program or department home page, available only in the catalogue, or not publicly available at all.
Analysis of Key Themes in PLOs
We first read through and analyzed the PLOs for the initial subset of undergraduate biology programs. Both authors independently read and inductively determined key commonalities of skills, competencies, and goals between the PLOs and themes that emerged from the data, following the principles of grounded theory (Bingham and Witkowsky, 2021). The two authors then discussed and generated a consensus codebook. Given that several colleges and universities included PLOs that were modeled from the Vision & Change core competencies (AAAS, 2011), we expanded our codebook to include all six of the core competencies listed in Vision & Change in order to allow us to characterize alignment of biology PLOs with these core competencies. However, we did not include the Vision & Change core concepts, which convey specific topics, rather than skills. These core concepts thus naturally align less well with PLOs, and we noted that few programs listed specific biological concepts in their PLOs, limiting the ability for us to examine alignment with Vision & Change core concepts.
Next, a random subset of 25 universities was selected to check for interrater reliability. Sixty-nine PLOs were identified across 10 universities in this subset, with the remaining programs not having any publicly available PLOs. We verified interrater reliability with this subset of 69 PLOs, calculating Cohen’s kappa using the program ReCal2 (Freelon, 2013). Cohen’s kappa is a measure of interrater reliability (how closely two coders using a consensus codebook agree on the same code for a set of responses) that starts with the percent agreement (the percent of time that the two coders agree) and then adjusts for the percent chance that such agreement occurs through random chance (Cohen, 1960; Warrens, 2015). Cohen’s kappa was 0.78, indicating substantial agreement (Landis and Koch, 1977). In addition, we noted that colleges and universities often had more than one PLO with a similar theme, with the same code applied to more than one PLO at that institution. Given our research question of characterizing the skills and competencies listed in each institution’s PLOs, we merged all identified themes across each institution’s PLOs. Next, we checked interrater reliability by comparing the independently generated codes across PLOs per institution. We similarly identified substantial agreement with the interrater reliability (Cohen’s kappa = 0.70). Given these high levels of interrater reliability, one coder (N.C.) coded the remainder of the institutions’ PLOs. In addition, we conducted a post hoc check of interrater reliability with another 30 randomly selected PLOs, again identifying high interrater reliability (Cohen’s kappa = 0.70).
Identifying Cognitive Levels of PLOs
In addition to characterizing the themes of the PLOs, we also examined whether each PLO was written to present lower- (LOC) or higher-order cognitive skills (HOC). We draw upon Bloom’s taxonomy of cognitive domains as a framework; this taxonomy presents a hierarchical view of six types of thinking: knowledge, comprehension, application, analysis, synthesis, and evaluation (Bloom, 1956; Krathwohl, 2002; Adams, 2015). These levels are arranged to go from LOC to HOC skills: the first two levels (knowledge and comprehension) are generally recognized as LOC skills, while the last three levels (analysis, synthesis, and evaluation) are recognized as HOC skills (Zoller, 1993). The third level (application), in contrast, is often considered a “transition” between LOC and HOC skills (Crowe et al., 2008). Indeed, past work that has attempted to categorize cognitive skills in biology classes using Bloom’s taxonomy has needed to examine specific assessment questions in depth to determine whether questions that fall under the “application” level in Bloom’s taxonomy are assessing a LOC or HOC skill, depending on whether or not the question requires both conceptual and procedural knowledge or whether it is an “algorithmic” application question that does not require conceptual information (Crowe et al., 2008; Freeman and Parks, 2010; Arneson and Offerdahl, 2018).
Our goal was to distinguish PLOs written with LOC skills from those written with HOC skills; we did not have access to any additional information (e.g., specific course assessments) to determine how instructors or students were interpreting each PLO. As such, we only categorized whether each PLO was conveying a LOC or HOC skill, without further subdivision into the specific Bloom’s levels. However, we relied on Bloom’s taxonomy to guide our decision making. We first discussed the differences between LOC and HOC skills and the definitions provided for each Bloom’s taxonomy level (Bloom, 1956). Next, we read, discussed, and categorized by consensus a random subset of 30 PLOs, using Bloom’s taxonomy as a framework for deciding whether each PLO conveyed a LOC or HOC skill. During this discussion, we decided to categorize a PLO that conveyed an application-level objective as a LOC skill, given that we did not have any further information to distinguish whether instructors and students viewed that PLO as a LOC or HOC skill. However, we acknowledge that future work will need to examine more data to explore how instructors and students are interpreting these PLOs, given the possibility that such application-level PLOs may be referring to either LOC or HOC skills. Next, we independently coded another 89 PLOs, roughly three times the size of the initial subset. We identified substantial agreement (Cohen’s kappa = 0.73). Given the high level of interrater reliability, one coder (N.C.) proceeded to classify the cognitive levels of the remaining PLOs. We again conducted a post hoc check of interrater reliability using 30 randomly selected PLOs, finding high interrater reliability (Cohen’s kappa = 0.78). A list of sample PLOs and their classification (as LOC or HOC) is included in Supplemental Table 1.
RESULTS
Our random sampling of colleges and universities spanned different types of institutions (Table 1) as well as multiple degree types (Table 2). Most programs offered bachelor of science in biology degrees, with fewer than 5% of programs only offering a bachelor of arts but not a bachelor of science in biology. More than one-fourth of our sampled programs offered associate of science degrees in biology, with the remaining 11.1% of schools not listing a degree type or providing an alternative degree type, such as associate of arts degree in biology.
| Carnegie Classification | Percent of schools |
|---|---|
| Doctoral universities | 24.59% |
| Master’s colleges and universities | 25.90% |
| Baccalaureate colleges | 28.20% |
| Associate’s colleges | 21.31% |
| Degree type | Percent of schools |
|---|---|
| Bachelor of science | 58.7% |
| Bachelor of arts | 4.3% |
| Associate of science (including associate of science–transfer degrees) | 26.1% |
| Other (including associate of arts and associate of arts–transfer degrees) | 11.1% |
To What Extent Are PLOs Publicly Available for Undergraduate Biology Programs in the United States?
We identified that slightly more than half of undergraduate biology programs had publicly available PLOs, with approximately 45% of programs not listing any of their PLOs publicly (Table 3). There were no differences in availability of PLOs based on institute type or by degree type (Pearson’s chi-square test). In sum, we identified 904 individual PLOs across these institutions (an average of 5.7 PLOs per degree).
| Availability of PLOs | Percent of institutions |
|---|---|
| Publicly available on biology program home page | 26.6% |
| Publicly available, not on biology program home page, but in course catalogue | 25.9% |
| Not publicly available | 44.9% |
What Skills, Competencies, and Knowledge Are Undergraduate Biology Programs Listing as Goals in Their PLOs, and Are There Differences by Institution Type (as Defined by Carnegie Classifications)?
We identified 17 different themes from the PLOs (Table 4). Nearly all programs (90.0%) indicated that students should be able to learn specific biological principles and concepts, the most common theme, followed by 72.5% of schools including PLOs with students learning about the process of science (Table 4 and Figure 1). Similarly, 65.0% of institutions indicated that their degree goal included students learning how to communicate science, the only other theme that was shared across more than 50% of institutions. While all six Vision & Change core competencies were identified within the PLOs, their frequencies varied. For instance, while the process of science was included in most schools’ PLOs, only 3.8% explicitly listed modeling and simulation, the lowest frequency of the six Vision & Change core competencies. In addition, we note that some of the themes may tangentially overlap. For example, one of the themes that emerged was the importance of conducting research, highlighted by a fifth of institutions. Similarly, a fifth of institutions had PLOs describing the goal of working in a field-based setting. Fieldwork can be an integral component of research projects, and some of these PLOs discussed a goal of having students conduct research in a field-based setting. These PLOs were therefore coded for both themes. However, other PLOs only noted field-based technical skills without discussing research (e.g. “Students will practice proper usage of scientific instrumentation and will improve techniques both in the laboratory and in the natural environment”) and thus were only coded for fieldwork.
| Code name | Percent of institutions with code | Code description | Example(s) |
|---|---|---|---|
| Biology Concepts | 90.0% | Incorporates specific ideas, concepts, and biological principles for students to learn | “Graduates will demonstrate a level of biological content knowledge appropriate to their degree level.” |
| Scientific Process* | 72.5% | Explicitly mentions using observations, hypotheses, questions, and experiments, and/or analysis to address a scientific question; aligns with Vision & Change core competency of ability to apply the process of science | “Understand scientific techniques used to define biological principles, distinguish among these techniques to select one appropriate to answer the question posed, and be able to design an experiment including selecting controls, recording observations, and drawing conclusions based on experimental results” |
| Communication* | 65.0% | Mentions scientific communication or conveying science to others; includes both oral and written communication skills; aligns with Vision & Change core competency of ability to communicate and collaborate with other disciplines | “Communication: Biology graduates should communicate clearly and accurately about biological issues in both oral and written form. In particular, they should be able to argue cogently from evidence, write the findings of a simple biological study in the format of a scientific paper, and give an effective oral presentation on a biological issue.” |
| Lab | 48.1% | Discusses mastering knowledge or skills in the lab (including both teaching labs and independent research labs); may overlap with “research” code but applies to any discussion of lab work without necessarily being research-oriented (e.g., technical proficiency) | “Demonstrate the ability to engage in library, field and/or lab research” |
| Quantitative Reasoning* | 34.4% | Explicitly mentions using numerical, graphing, statistical, or other quantitative skill; aligns with Vision & Change core competency of ability to use quantitative reasoning | “Be able to analyze and answer biologically relevant problems through the successful application of quantitative and analytical methods” |
| Primary Literature | 27.5% | Describes students’ ability to read and interpret peer-reviewed literature | “Critically read the scientific literature in order to evaluate the scientific process” |
| Information Literacy | 26.6% | Discusses searching, gathering, and/or evaluating information (e.g., articles) | “Identify, retrieve, and properly interpret and apply published and electronic sources of biological and other scientific information”“Search and evaluate the scientific literature in order to identify acceptable sources” |
| Scientific Ethics | 21.3% | Specifically mentions use of integrity through scientific process | “Recognize and understand ethical principles of fundamental and applied science and practice in society” |
| Research | 20.6% | Explicitly mentions research or novel scientific questions, including gathering and organizing scientific data | “Integrate themes and principles learned in the classroom through conducting original research” |
| Field | 20.0% | Describes experiments or research conducted outside a laboratory setting (e.g., fieldwork) | “Demonstrate the ability to engage in library, field and/or lab research” |
| Society* | 18.1% | Discusses societal context for science; aligns with Vision & Change core competency of ability to understand the relationship between science and society | “Recognize and understand ethical principles of fundamental and applied science and practice in society”“Students should be able to relate contemporary societal and global issues to the physical and life sciences.” |
| Careers | 17.5% | Specifically describes preparing students for a future occupation or professional school | “Seniors will be prepared for success in biology-related fields.”“Have acquired the knowledge and skills necessary for success in graduate programs in biological sciences or in professional (medical, dental, veterinary, nursing) school” |
| Collaboration | 13.1% | Discusses working with others, in groups or in teams, as a key skill | “Students should be able to work with peers in a team setting.” |
| Interdisciplinary Nature of Science* | 12.5% | Involves students learning skills and concepts from non-biological fields and/or integrating with biology; aligns with Vision & Change core competency of ability to tap into the interdisciplinary nature of science | “A general competency in basic inorganic and organic chemistry as well as in introductory physics, mathematics and statistics”“Interdisciplinary Thinking: Integrate concepts from across various science disciplines and from other, non-science, ways of knowing” |
| History of Science | 6.9% | Mentions historical context or study of past developments in science | “Undergraduate students in Biology will describe major achievements, historical contexts, and current issues in Biology.” |
| Computational Skills | 5.6% | Describes students’ ability to use computer/technical skills applied in a biological setting | “Demonstrate computer and technology literacy, including the ability to access databases within the context of course research and project development”“They should be able to use computer software to produce a technical report that includes graphs, tables, and references.” |
| Modeling and Simulation* | 3.8% | Describes using “mathematical or computational tools [to] describe living systems” (AAAS, 2011); aligns with Vision & Change core competency of ability to use modeling and simulation | “Modeling: Recognize scientific models, use models to make inferences and solve problems, and build and evaluate your own models.”“Gather and organize scientific data, analyze it, evaluate its meaning and develop models for interpreting scientific phenomena” |
FIGURE 1. Frequency of each PLO theme across institutions.
In addition, we compared the themes to see whether they varied across both institution type (as defined by Carnegie Classification) and degree type (associate’s vs. bachelor’s degrees). We identified six themes that varied in frequency between either institution type or degree type (Pearson’s chi-square test, p < 0.05; Figure 2). For instance, a greater percentage of undergraduate bachelor’s programs indicated biology concepts and research skills in their PLOs as compared with associate’s programs (Figure 2). Similarly, no associate’s or baccalaureate colleges had computational skills in their PLOs, while this was included in approximately 20% of master’s and doctoral programs’ PLOs. Finally, a greater percentage of bachelor’s programs indicated fieldwork-based skills or scientific communication abilities in their PLOs as compared with associate’s programs.
FIGURE 2. Comparison of frequency of themes by (A) institution type and (B) degree type. Only themes that are significantly different by institution type are included here. Significance (Pearson's chi-square test): *p < 0.05; **p < 0.01; ***p < 0.001 level.
What Is the Cognitive Level of the Skills and Competencies That Undergraduate Biology Programs List in Their PLOs, and Are There Differences by Institution Type?
Approximately half (51.3%) of the PLOs examined were LOC skills, with the other 48.7% representing PLOs with HOC skills. When compared by institution type, doctoral universities showed a higher proportion of LOC PLOs compared with HOC PLOs (p < 0.05, Pearson’s chi-square test; Figure 3). There were no differences in frequency of PLOs with different cognitive levels based on degree type.
FIGURE 3. Comparison of PLOs by HOC versus LOC skills.
DISCUSSION
Our results provide the first characterization of PLOs across undergraduate biology programs in the United States, and this is the first study we are aware of that examines a broad spectrum of PLOs in any science, technology, engineering, or math (STEM) discipline. For instance, past work examining PLOs in STEM has been limited to an introspective look at how one undergraduate physics program used PLOs to guide its assessment process and curricular reform (Denicoló, 2019) and a case study that examined similar processes of writing PLOs, assessing student gains for each PLO, and making curricular changes in four undergraduate chemistry programs (Towns, 2010). By providing a comprehensive look at PLOs across a broad subset of biology programs, our work reveals the extent of PLOs being publicly available, providing insight into how PLOs may be used for teaching, learning, and assessment. Similarly, we also characterize what skills and competencies biology programs are including in their PLOs.
Importance of PLOs to Drive Curriculum Development and Assessment and Curricular Improvements
Our work identifies that nearly half of all undergraduate biology programs do not have PLOs listed publicly on any of their program webpages or in academic course catalogues. This is concerning for several reasons, and we call on biology programs to ensure that PLOs are thoughtfully written for the program and are widely available to prospective and current students, faculty, staff, and other stakeholders in the process.
First, we note the critical importance of PLOs in the creation of curricular maps for a biology degree. Curricular maps show the progression of learning for each PLO across different courses in a program and allow faculty to identify potential gaps in students’ learning when examining this progression of learning and comparing it with the required and elective courses offered by a program (Joyner, 2016; Metzler et al., 2017). Similarly, instructors should rely on the curricular map and PLOs to design and revise their courses, with course learning outcomes aligned with and supporting the PLOs. While it is possible that the PLOs may be disseminated internally within these programs, the lack of publicly posted PLOs calls into question the accessibility, availability, and use of the PLOs for these programs. It is possible that many of these programs may not be using or regularly examining PLOs and curricular maps for their programs.
PLOs are also critical for the assessment process, wherein programs collect data to examine student progression and mastery of each PLO (Towns, 2010; Metzler et al., 2017; Denicoló, 2019). These data can identify areas of improvement and inform curricular and co-curricular changes to strengthen student learning, retention, and graduation rates (Shupe, 2007). PLOs are central to this assessment process; without clearly identified PLOs, programs will likely struggle to incorporate an effective assessment plan, identify areas of strength and weaknesses within the program, and make appropriate changes. Similarly, interpreting multiple courses’ learning outcomes together will be more challenging in the absence of unifying PLOs for the degree, again hindering instructional changes and improvement.
We also note the importance of PLOs for conveying to current and prospective students, as well as prospective employers, what goals and skills each biology program teaches and incorporates in its curriculum. Given that past work has found that showing specific course-level learning objectives is useful for students and can improve student learning in biology (Armbruster et al., 2009; Brooks et al., 2014; Sana et al., 2020), we speculate that continued alignment of PLOs to course learning objectives, combined with regular reminders to students of their PLOs and conveying how the courses they are taking fit in with these PLOs, may have positive impacts on student learning, affect, and motivation, and may even influence students’ choice of courses within the curriculum. Similarly, past work has identified that PLOs play a key role in communicating to prospective students what an institution and program value, and what their standards are (Shupe, 2007). Thus, having publicly available and accessible PLOs can help prospective students make more informed decisions about their choice of college or university, or even their choice of major, and will help ensure that students are aware of the goals and learning outcomes of a program before enrolling. Finally, past work in other disciplines has identified that there are sometimes gaps between PLOs and skills listed by employers, suggesting that PLOs can play an important role for communicating recent graduates’ skills to potential employers (Rhew et al., 2019).
Situating the Role of PLOs within Institutional Accreditation
The critical role of PLOs is also acknowledged within the standards of each of the seven regional accreditation agencies for colleges and universities in the United States recognized by the Council for Higher Education Accreditation and the U.S. Department of Education (Council for Higher Education Accreditation). All institutions listed in the Carnegie Classifications must be accredited by one of these organizations, and each of the organizations lists PLOs or equivalent as a requirement for degree programs (Table 5).
| Accreditation agency | Requirement for PLOs or equivalenta |
|---|---|
| Accrediting Commission for Community and Junior Colleges (ACCJC) Western Association of Schools and Colleges (WASC) | “Student learning and student achievement: The institution defines standards for student achievement and assesses its performance against those standards. The institution publishes for each program the program’s expected student learning and program-specific achievement outcomes. Through regular and systematic assessment, it demonstrates that students who complete programs, no matter where or how they are offered, achieve the identified outcomes that the standards for student achievement are met” (Accrediting Commission for Community and Junior Colleges, 2014). |
| Higher Learning Commission (HLC) | “The institution has clearly articulated goals for its academic programs and has strategies for assessment in place” (Higher Learning Commission, n.d.). |
| Middle States Commission on Higher Education (MSCHE) | “Assessment of student learning and achievement demonstrates that the institution’s students have accomplished educational goals consistent with their program of study, degree level, the institution’s mission, and appropriate expectations for institutions of higher education. An accredited institution possesses and demonstrates the following attributes or activities: 1. Clearly stated educational goals at the institution and degree/program levels, which are interrelated with one another, with relevant educational experiences, and with the institution’s mission; 2. organized and systematic assessments, conducted by faculty and/or appropriate professionals, evaluating the extent of student achievement of institutional and degree/program goals. Institutions should define meaningful curricular goals with defensible standards for evaluating whether students are achieving those goals … and use assessment results for the improvement of educational effectiveness” (Middle States Commission on Higher Education, n.d.). |
| New England Commission of Higher Education (NECHE) | “Assessment of learning is based on verifiable statements of what students are expected to gain, achieve, demonstrate, or know by the time they complete their academic program. The institution publishes statements of its goals for students’ education and makes available to the public timely, readily accessible, accurate, and consistent aggregate information about student achievement and institutional performance” (NECHE, n.d.). |
| Northwest Commission on Colleges and Universities (NWCCU) | “Student learning: The institution identifies and publishes the expected learning outcomes for each of its degree, certificate, or credential programs. The institution engages in regular and ongoing assessment to validate student learning and, consistent with its mission, the institution establishes and assesses student learning outcomes (or core competencies)… across all associate and bachelor level programs” (Northwest Commission on Colleges and Universities, n.d.). |
| WASC Senior College and University Commission (WSCUC) | “The institution clearly defines and publishes educational objectives for each program, including expected student learning outcomes, and identifies how these objectives and outcomes will be addressed within the curriculum. Strategies for assessing students’ achievement of these educational objectives, including direct assessment of student learning, are also established” (WASC Senior University and College Commission, n.d.). |
| Southern Association of Colleges and Schools Commission on Colleges (SACSCOC) | “The institution identifies, evaluates, and publishes goals and outcomes for student achievement appropriate to the institution’s mission, the nature of the students it serves, and the kinds of programs offered. The institution uses multiple measures to document student success. The institution identifies expected outcomes, assesses the extent to which it achieves these outcomes, and provides evidence of seeking improvement based on analysis of the results in student learning outcomes for each of its educational programs” (Southern Association of Colleges and Schools Commission on Colleges, 2016). |
We note that each of these accreditation agencies requires PLOs as well as assessment of the PLOs. Additionally, most of the agencies state that programs should “publish” PLOs, implying that such PLOs must be publicly available and readily accessible to students, faculty, staff, and other stakeholders. It is possible that some of the biology programs we examined may not be in compliance with their accreditation agency’s standards and may either not have PLOs or may not be publishing their PLOs. While we did not align each institution with its accreditation agency, past work has found similar levels of noncompliance. For example, a survey of more than 800 provosts or chief academic officers at accredited institutions in the United States revealed that more than a fourth of the respondents’ respective institutions did not have any college- or university-wide learning outcomes (Kuh and Ewell, 2010). A similar survey was conducted of provosts or chief academic officers of member institutions of the American Association of Colleges and Universities. This survey showed that 28% of surveyed institutions were not conducting assessment of learning outcomes across the curriculum, with more than a third of surveyed institutions (35%) acknowledging that they did not have defined PLOs for each of their programs (Hart Research Associates, 2009).
Taken together, these data indicate that there are likely a significant number of undergraduate biology programs that may not have any PLOs, may not be posting them publicly, or may not be undertaking significant efforts at examining curricular maps and conducting assessment. We urge every undergraduate biology program to establish PLOs for the degree, if they do not already exist, to ensure that these PLOs are publicly accessible and readily available, and to establish aligned curricular maps and comprehensive assessment plans to best support learning and assessment. We also note that half of the undergraduate biology programs we examined that did publicly post PLOs only had them accessible in an academic course catalogue, and not on the departmental or program webpage. The placement of PLOs in a catalogue likely limits their visibility, as prospective students are more likely to read about a department and program on their webpages, and we anticipate that current students would similarly rely more on program-specific webpages and documents rather than the academic course catalogues. In addition, given the importance of aligning course learning outcomes to PLOs, we also urge instructors to convey the alignment of course learning outcomes to PLOs by placing PLOs on course websites (e.g., home pages on learning management systems) and syllabi.
Characterizing Undergraduate Biology PLOs
We also characterized the range of skills, competencies, and goals that undergraduate biology programs listed in their PLOs. Unsurprisingly, we identified that the most common theme in PLOs is having students demonstrate their mastery of biological content knowledge, followed by students being able to think through and apply the process of science and know how to communicate scientifically. There was no other skill or competency that was shared in the majority of institutions’ PLOs, indicating that there is wide variability in what undergraduate biology programs include in their PLOs.
Indeed, we identify 17 themes across these PLOs and highlight a few significant points. First, we identified some PLOs focused on specific lab skills, such as the ability to use instrumentation or perform specific techniques, while other PLOs instead focused on independent research and the process of conducting such research. The number of institutions with PLOs encompassing lab techniques was more than double that of institutions that had PLOs that included independent research and research skills (48.1% and 20.6%, respectively). Multiple calls, including Vision & Change, have emphasized the importance of developing students’ research skills and how such research can help students increase their abilities to think critically (National Research Council, 2003; AAAS, 2011). Given this, undergraduate biology programs may wish to reflect on their choice of PLOs and the relative importance of both developing lab-based technical skills and using those skills to develop research competency. We urge programs to provide supports and opportunities for students to conduct independent research or participate in course-based undergraduate research experiences (CUREs) and to ensure that their PLOs reflect the importance of these skills.
We also similarly note a difference in several themes by institution type. For instance, far fewer associate’s colleges listed research skills in their PLOs as compared with other institution types. This may be because faculty at 2-year institutions are not expected to conduct research and thus are usually not given time or resources to do so (Schinske et al., 2017). As such, there may be lower emphasis in the curriculum and fewer opportunities for students to conduct independent research. There are similarly lower frequencies of PLOs at 2-year institutions encompassing fieldwork-based skills or computational skills. We speculate that this difference may be due to faculty at 2-year institutions having higher teaching loads and potentially lower amounts of institutional resources (Schinske et al., 2017), which may serve as barriers for implementing courses that involve fieldwork or computational biology. However, more work—such as surveys or interviews with department chairs and the faculty in biology programs who contributed to the writing of PLOs—is needed to elucidate the reasons for these differences.
In addition, we identified that more than half of the PLOs were written to describe LOC skills, despite Vision & Change and other national calls emphasizing the need to promote critical thinking and associated HOC skills (National Research Council, 2003; AAAS, 2011). These results match surveys of course learning objectives conducted in other STEM disciplines: for example, nearly 90% of learning outcomes across psychology programs were found to encompass LOC skills (Bumpus et al., 2022). Taken together, these results indicate that programs may still be emphasizing or teaching LOC skills, consistent with work that has found that many introductory biology courses are focusing on LOC skills and that upper-division biology classes may still retain this focus on LOC skills (Momsen et al., 2010; Cleveland et al., 2017). This has led to recent calls for biology programs to develop program objectives that focus on HOC skills (Cleveland et al., 2017), and we echo such calls for undergraduate biology programs to reflect on their PLOs and determine whether there are ways to integrate more HOC skills into their PLOs and curricula. Our results also indicated that doctoral universities had a significantly higher frequency of PLOs with LOC skills than the other types of institutions. More work is needed to explore the reasons behind this difference. However, we note that there are both typically larger class sizes (as well as total numbers of students in programs) and an increased expectation of research productivity from faculty at doctoral universities (Tanner and Allen, 2006). As such, we speculate that differences in size, the different emphases on teaching, and potentially different amounts of pedagogical training among faculty at the different types of institutions may be contributing to this difference.
Our results show that all six core competencies from Vision & Change were represented in our sample of PLOs. However, there was major variation in the frequency of these competencies (Table 4), suggesting uneven coverage across institutions. For instance, fewer than 10% of institutions incorporated any PLOs that conveyed the abilities to conduct modeling and simulation, and fewer than 20% included PLOs addressing science and society or the interdisciplinary nature of science (see Table 4 for examples of these PLOs). These results suggest that there is an urgent need for programs to examine whether their curricula are addressing each of the core concepts and competencies suggested in Vision & Change and to examine whether programs are implementing the evidence-based recommendations listed in Vision & Change.
Implications and Suggestions for the Biology Education Community
We provide several suggestions and implications of our work for institutions, program directors, chairs, instructors, and the biology education community:
Provide more resources and training to create PLOs, curricular maps, and robust assessment structures. Our work demonstrated that nearly half of all undergraduate biology programs may not have PLOs, aligning with past work finding high frequencies of colleges and universities that do not have learning objectives or assessment plans. We call on colleges and universities to provide more training and resources for faculty to develop such PLOs, curricular maps, and assessment protocols, an urgent need also identified by provosts (Kuh and Ewell, 2010). For example, organizations like the National Institute for Learning Outcomes Assessments (www.learningoutcomesassessment.org) can provide an invaluable set of resources for faculty and administrators. We similarly see an opportunity for the biology education community to develop freely available resources tailored specifically for our discipline to support creation and review of PLOs, curricular maps, and assessment protocols. For instance, the new CBE-LSE Evidence-Based Teaching Guide on learning objectives (Orr et al., 2022a,b) provides an examination of the literature surrounding learning objectives, and provides instructors several resources for developing course learning outcomes. Similar resources for developing PLOs, building and examining curricular maps, and conducting assessment, may be useful.
Review existing PLOs to ensure that they convey intended learning outcomes and align with national standards. We note that half of the PLOs were written as a LOC skill and speculate that programs and faculty, if queried, would likely intend for their students to gain HOC skills. Given this potential disconnect, programs should review their existing PLOs to ensure that they accurately reflect the intended levels of learning and identify opportunities to incorporate more HOC skills throughout the curriculum. Similarly, we saw uneven coverage of the Vision & Change core competencies across institutions’ PLOs. We urge programs to consider the recommendations of Vision & Change, and use developed tools like the BioCore Guide (Brownell et al., 2014) and BioSkills Guide (Clemmons et al., 2020) to check whether these concepts and competencies are being introduced and taught in the curriculum, and revise PLOs if necessary. We highlight these steps in more detail in our Recommendations for writing, revising, and using PLOs section below.
Support opportunities for students to develop research skills, field-based skills, and computational skills. We found unequal coverage of PLOs that emphasized research skills, field-based skills, and computational skills, which were represented in particularly low frequencies across the PLOs for 2-year institutions. We urge the biology education community to develop additional curricula and resources for programs to support students’ development of these skills in order to best meet the needs of students. For example, there have been recent calls highlighting the growing importance of computational skills in biology (Rubinstein and Chor, 2014; Stefan et al., 2015; Wright et al., 2020). Similarly, Vision & Change and other national calls have highlighted the importance of integrating undergraduate research experiences in biology (AAAS, 2011), given the many affective and cognitive benefits for students who participate in research experiences (Lopatto, 2007; Russell et al., 2007; Jones et al., 2010; Adedokun et al., 2013; Auchincloss et al., 2014; Dolan, 2017; Schinske et al., 2017). Thus, additional resources for programs and faculty to develop these skills (e.g., promoting accessible professional development for implementation of CUREs or workshops that provide an overview of key computational skills and instructional strategies for teaching such skills) may be beneficial.
Recommendations for Writing, Revising, and Using PLOs
In addition to these recommendations for the biology education community, we also provide a general set of steps here for chairs, program directors, and faculty looking to write new PLOs or revise existing ones. We note that there have been very few empirical studies examining best practices for learning objectives at the program level, with most work instead focusing on course and specific learning objectives. The CBE-LSE Evidence-Based Teaching Guide on writing and using learning objectives provides a useful summary of this past work (Orr et al., 2022a,b). However, many of the same principles for writing course-level learning objectives likely apply to writing and revising PLOs, and we draw upon recommendations provided in Vision & Change and other calls for reform.
Use Vision & Change and Aligned Resources
First, we urge programs to use the Vision & Change core concepts and competencies to guide the revision and writing process of PLOs. The development of the core concepts and competencies in Vision & Change was designed as a “consensus framework” for undergraduate biology programs’ curricula and conveys foundational knowledge and skills for biology majors (AAAS, 2011). We encourage biology programs to use these concepts and competencies when writing or revising PLOs, echoing the call from Vision & Change to use the core concepts and competencies as a foundation for writing PLOs relevant for the programs’ students and aligned with institutional goals and themes. In addition, the report also includes a table showing the alignment of different disciplinary practices with the core competencies to further guide programs’ use of these core competencies.
There are several other resources aligned with the Vision & Change core concepts and competencies that biology programs can use when writing, revising, and assessing PLOs. First, programs can use the BioCore and BioSkills Guides, both of which were developed through an iterative process that involved surveying biology instructors nationwide (Brownell et al., 2014; Clemmons et al., 2020). The BioCore Guide offers a specific list of principles aligned with the Vision & Change core concepts, while the BioSkills Guide provides a list of measurable PLOs aligned with the Vision & Change core competencies, sample course-level learning outcomes that align with each of the possible PLOs, and examples of classroom activities for each course learning outcome (Brownell et al., 2014; Clemmons et al., 2020). There are also validated assessments available that are aligned with these core concepts. For instance, GenBio-MAPS is designed as an assessment that can be deployed longitudinally to measure the change in students’ mastery across the core concepts in a general biology program (Couch et al., 2019). Similar tools are available for molecular biology, physiology, and ecology/evolution (Smith et al., 2019).
In addition to these guides and tools, existing initiatives dedicated to supporting undergraduate biology programs’ reform efforts can help with developing and/or examining PLOs. For instance, the Partnership for Undergraduate Life Sciences Education (PULSE) is a group sponsored by the National Science Foundation, Howard Hughes Medical Institute, and the National Institute for General Medical Sciences whose mission is to promote the evidence-based practices highlighted in Vision & Change (Dou, 2018). PULSE offers a set of Vision & Change rubrics that programs can use to evaluate their curricula; the rubric includes several items that guide faculty toward reflecting on their PLOs and assessment efforts (Brancaccio-Taras et al., 2016; Branchaw et al., 2020). In addition, PULSE offers workshops and site visits from PULSE Ambassadors, which can both support programs looking to establish or revise PLOs or make curricular changes based on assessment of PLOs.
Ensure That PLOs Are Clear, Actionable, and Measurable
PLOs should convey a set of skills or competencies that a student should be able to accomplish following completion of the undergraduate biology degree (Orr et al., 2022a). As such, PLOs should encompass a specific set of skills that are observable or measurable. Faculty can use Bloom’s taxonomy as a framework for identifying appropriate verbs for each Bloom’s level for PLOs (e.g., for a list of possible verbs for PLOs that are aligned with Bloom’s levels, see Towns, 2010). We encourage programs to emphasize HOC skills (and thus verbs aligned with the appropriate Bloom’s level) when writing these PLOs, given that our work identified that there are many existing PLOs that cover LOC skills. In addition, we note that there are some commonly used words that may not be clear or actionable. For example, there is no consensus on what cognitive level the word “understand” conveys when used in learning objectives, making it challenging to design appropriate assessments for any learning objective that states a student should “understand” something (Hsu et al., 2021). Thus, we encourage faculty, instructors, and chairs to ensure that all PLOs use more specific, actionable verbs that describe a skill that can be directly assessed.
Align Course-Level Learning Objectives with PLOs and All Assessments with These Learning Outcomes
Once PLOs are established, programs should discuss the PLOs with their instructors to ensure alignment of course-level learning objectives with PLOs. PLOs serve as one of the broadest levels of learning objectives and should encompass a nested set of course-level learning outcomes that align with each of the PLOs (Orr et al., 2022a). Similarly, instructors should have specific instructional learning objectives that align with different course learning outcomes. Thus, PLOs serve as a key part of the hierarchical nature of learning outcomes, wherein instructional learning objectives align with course learning objectives, which in turn align with PLOs (which may correspondingly align with institutional goals and learning outcomes).
Assessments should likewise be aligned with each of these learning objectives. For example, at the course level, instructors should check that each assessment question aligns with a specific instructional learning objective and a course learning objective. Assessments used to measure student progress and mastery of PLOs should also be aligned with the specific skills and competencies that the PLO states. The PULSE rubrics (Brancaccio-Taras et al., 2016; Branchaw et al., 2020) serve as a useful resource for programs to evaluate their use of PLOs and assessments, as they provide additional guidelines to facilitate alignment of PLOs, course learning outcomes, and assessments. In addition, the CBE-LSE Evidence-Based Teaching Guide on learning objectives offers further recommendations for how instructors can use learning objectives, many of which can apply to both course-level learning objectives and PLOs (Orr et al., 2022a,b).
Ensure That PLOs Are Shared Extensively with All Stakeholders and Used to Improve Student Learning
Our results showed that many colleges and universities may have PLOs but may not be sharing them with students, faculty, and staff, and that many programs do not put such PLOs on their departmental home pages. We urge every biology program to clearly communicate these PLOs with all stakeholders to improve teaching and learning and to ensure that PLOs are used to guide teaching, learning, and assessment. For instance, programs may ask instructors to list PLOs on their course webpages and syllabi and show the alignment of course learning objectives with such PLOs. In addition, programs should regularly be reviewing curricular maps, alignment of course learning outcomes with the PLOs, and assessment data to identify opportunities to improve curriculum and co-curricular supports. PLOs and the course-level learning objectives that align with such PLOs should serve as a foundation for any curricular decisions. Similarly, assessments of such PLOs and course-level learning outcomes should be used to examine student progression and mastery of learning objectives and to identify opportunities to make curricular and co-curricular changes to better support student learning.
Limitations and Future Directions
There are several limitations of our work. First, we sampled institutions to ensure even coverage across the different institutional types. However, we note that these institutional types, on average, enroll different numbers of students. Our sampling scheme thus may not be representative of all undergraduate biology majors’ experiences. Second, we only examined publicly available PLOs, and it is possible that some programs have PLOs that are posted internally. Similarly, we were not able to gain insight into how programs apply these PLOs for their curricular design and assessment, which would require feedback from faculty and access to curricular maps and assessment reports. Future work is needed to examine how programs design and implement PLOs across their curricula, the impact of PLOs on course learning outcomes and instructional activities, and how faculty and students perceive and are influenced by such PLOs.
We also note that an undergraduate biology program listing a particular skill or competency on a PLO does not necessarily mean that the program is teaching or emphasizing that PLO. It is possible, for example, that faculty in some undergraduate biology programs have not examined their PLOs recently or have not used curricular maps to inform their teaching, leading to a lack of alignment between the enacted curricula and the PLOs, which only represent the intended learning outcomes. Indeed, past work has highlighted how there can often be disconnects between what an instructor lists as learning objectives and what is actually taught and assessed in a course (Clemmons et al., 2022). More work is needed to examine alignment of course learning outcomes with PLOs across a range of institutions to provide insight into the fidelity of such PLOs.
Finally, we acknowledge that there is great variation within institutions at each level of Carnegie Classification and that there are likely many cultural, institutional, and societal factors that may influence each undergraduate biology program’s mission, values, and PLOs. We were not able to account for these factors in our work, and future work that relies on surveys and interviews of the chairs, deans, and/or faculty who wrote the PLOs is needed to explore this variation and factors that shape the choice of PLOs for each program.
ACKNOWLEDGMENTS
We thank Desiree Forsythe and Kate Hill for invaluable feedback on this project. N.C. was supported by the Schmidtern program, Chapman University’s Schmid College summer undergraduate research program.
