Assessment of the Effectiveness of the Studio Format in Introductory Undergraduate Biology

INTRODUCTION

Introductory biology is a course taught at almost all universities, most often in a traditional lecture/laboratory format (Sundberg et al., 2005). A number of efforts have been made to infuse more active learning into these classes, particularly into the large lecture components, for example by incorporating small cooperative discussion groups (e.g., Ebert-May et al., 1997, Klionsky, 1998), case studies (e.g., Brickman, 2006), or cooperative groups and personal response systems (“clickers”; e.g., Armstrong et al., 2007). A Science Policy Forum in 2004 urged that widespread changes should be made in the way that introductory science courses are taught, including incorporation of active participation in lectures, discovery-based laboratories, and assessment of the effectiveness of these courses (Handelsman et al., 2004). This paper reports the process of such a transition in introductory biology at Kansas State University (K-State) and the outcomes measured to date.

The K-State introductory biology course, Principles of Biology, is taught to approximately 1500 students each year and serves both majors and nonmajors. It was created in 1968 and used the then-current audio-tutorial (A-T) format (Postlethwait, 1970). In this format, students listened to audiotaped lectures and instructions and carried out labs on their own, with assistance, if desired, from graduate teaching assistants and faculty members staffing the classroom (“A-T lab”). The advantages of this system were that students could work at their own pace; there was an integration of the lecture and laboratory components of the course; and students were able to come to the A-T lab at times of their own choosing during the week. The disadvantages included a lack of student-faculty continuity (which was ameliorated to some extent by incorporating weekly recitations taught by faculty members); lack of a mechanism to encourage student attendance; and a feeling of isolation for students, who sat alone in carrels listening to tapes. The expected contact time for students was 5 h per week: two 2-h blocks in the A-T lab and a 1-h recitation.

Faced with the need to do an equipment upgrade and recognizing that the A-T format was no longer state-of-the-art pedagogy, the K-State Division of Biology completely restructured the course in 1997. Our entire faculty provided input and we chose to use a studio format, as pioneered at Rensselaer Polytechnic Institute for physics instruction (Wilson, 1994; Pipes and Wilson, 1996). The studio model features integration of elements of lecture and laboratory, interactive computer software, teamwork, multiple learning modalities, an investigation-driven approach, and instructors as facilitators rather than lecturers. Students are required to take more responsibility for their own learning than in a traditional lecture.

We chose the studio model over a traditional lecture/lab format or a modernized version of A-T that used computers instead of tape recorders to deliver the content because we wanted to create the most effective learning environment for the course, and also retain the engagement of most biology faculty members in teaching it. The transition took 3 yr of planning and was supported with funding from internal and external sources. Features of the A-T and studio models as used at K-State are compared in Table 1.

Here we report on the studio conversion process and its outcomes. We present data on student exam performance, student grades, student and instructor perceptions and attitudes, and student performance in subsequent biology courses for the course in the studio format and its previous format. Collectively, these measures demonstrate that the studio format is at least as effective as A-T for introductory biology instruction. We also report information on gains in student learning for the studio format and show that the studio format compares favorably with gain of learning in three courses taught in more traditional lecture/lab formats.

STUDIO BIOLOGY COURSE

Our Principles of Biology studio course is a one-semester, mixed majors and nonmajors course that covers the broad areas of evolution, ecology, biochemistry and cell biology, genetics and molecular biology, energetics, plant biology, and animal biology, in 2-wk blocks for each. A multiple-choice exam follows each module. Each studio section of 80 students is staffed by two faculty members, two graduate teaching assistants, and in some cases, one or two undergraduate teaching assistants. Each 2-h class (students attend two per week) begins with a quiz over assigned reading that also serves as a mechanism for tracking attendance. Faculty members or graduate teaching assistants then give a mini-lecture of 10–15 min that serves to introduce the basic concepts and experimental approaches that the students will use to investigate that day's material. During the largest portion of each class period (1–1.5 h), students work in pairs (two students per computer or microscope) or groups of four, with instructors moving around the room, answering and asking questions. Most classes feature a combination of traditional “wet-lab” activities and computer work, including simulations, models, and investigations. Each student purchases and brings to class a custom studio manual in which to record answers to concept questions, data from classroom exercises, and drawings and notes from mini-lecture and lab activities. Thus, students are exposed to teamwork and active learning, and the multiple learning modalities used provide formats friendly to students with a variety of learning styles (see, e.g., Fleming and Mills, 1992). The last 10 or so minutes of each class period are a “wrap-up” session in which the instructor reviews the important concepts, and students share data and summarize their findings. Table 2 shows the content of the Energetics module of Principles of Biology as an example of the different types of content included in the course.

ASSESSMENT MEASURES

Principles of Biology has been taught exclusively in the studio format since summer 1997. During this time, minor changes in course content and format have been made, such as the addition of study guides (available to students online) and optional weekly review sessions. We have evaluated the success of the course in this format by 1) analyzing student performance on examination questions used before and after conversion to the studio format; 2) examining final course grade distributions from the A-T and studio years; 3) comparing student and instructor perceptions and attitudes toward both the A-T and studio formats; 4) monitoring student performance in subsequent biology courses; and 5) measuring student learning in the studio format by administering an objective test at the beginning and end of the course and comparing this with student learning in other types of biology courses at our institution and another public university in Kansas. We report on each of these assessment measures and their outcomes below.

Student Performance on Exam Questions Used in Both the A-T and Studio Formats

To compare student learning in the former A-T and current studio formats, we reviewed historical records of student performance on common and similar exam questions used for multiple semesters in both formats of the course. We selected 15 objectives that have consistently been part of the course content and identified all questions that addressed these objectives. These questions were not necessarily identical (though in some cases they were) but were similar and covered the same material. Some of these questions also covered the same objectives as those in our pre- and postassessment tests (see below). Table 3 shows student performance on all 15 objectives and overall, comparing A-T and studio. The overall mean performance of A-T students and studio students did not differ significantly (p = 0.63). Only two objectives (characteristics of diffusion and definition of a population) showed significant differences between A-T and studio, with student performance on these objectives being higher in the studio format.

Final Course Grade Distributions in A-T and Studio Format Semesters

As a way of determining whether student performance in the course was comparable in the A-T and studio eras, we have collected data from multiple years of fall and spring semester final grades. It should be noted that the student population of K-State is heterogeneous, with the institution having had open admissions until 2001 and relatively liberal admissions standards subsequently (Requirements for K-State, 2008). Data from the K-State Registrar's Office indicate that there was no significant difference (as measured by two different nonparametric tests) between the mean ACT score for entering freshmen for the 5 yr before the qualified admissions policy went into effect (1996–2000) and the 6 yr after the change (2001–2006) (data not shown). Thus a difference in K-State student preparation or aptitude, over the time frame covered by our analyses, is probably not a variable that needs to be considered in regard to our conclusions.

Another issue that should be considered in evaluating the course final grade data is the fact that students take this course for a variety of reasons, including the fulfillment of a distribution requirement. Thus the population of students enrolled in the course is also fairly heterogeneous. The final grade data are shown in Table 4. The mean grades are very close; 2.15 for A-T and 2.12 for studio. However, the grade distributions differ significantly; the majority of this difference is in the proportion of students earning D's (higher in the studio years) and F's (lower in the studio years). We do not know the reason for this difference; two possibilities are 1) that the more favorable student-faculty ratio in the studio format allows some marginal students to pass the course with a D who might have otherwise failed; and 2) that students in recent years might be more likely to withdraw and retake a course that they believe they will fail. The second possibility can be directly examined. We analyzed the patterns of student withdrawal over 14 yr of Principles of Biology, including the last eight semesters of the A-T version of the course and all semesters of the studio version for which withdrawal data were available. These data are illustrated in Figure 1. Figure 1A shows the withdrawal data for each semester and 1B shows the analysis broken out by three groups of semesters: the last eight semesters of the A-T format, the first eight semesters of the studio format, and the last 12 semesters of the studio format. The pattern is complex but the following conclusions can be made. 1) Withdrawal rates (formal withdrawal from the course between week 5 and 10 of the semester; students receive a “W” on their transcript) averaged ∼8% over the last eight semesters of the A-T version of the course. 2) Withdrawal rates during the first eight semesters of studio were ∼10%. 3) Withdrawal rates in the subsequent 12 semesters of studio were ∼5%. The difference between the median withdrawal rates in A-T and most recent studio semesters is significant at the P < 0.005 level in an unpaired t test. These data are thus more consistent with the first hypothesis above, indicating that fewer marginal students are withdrawing from the studio course, and are possibly passing the course with a D.

Student and Instructor Perceptions and Attitudes Toward the A-T and Studio Formats

Student Perceptions and Attitudes.

In conjunction with Drs. Amy Gross and William Pallett of the K-State IDEA Center, we created a series of surveys that were administered to students enrolled in Principles of Biology in the last semester in which it was taught in the A-T format (spring 1997) and the first two spring semesters it was taught in the studio format (spring 1998 and 1999). IDEA Center personnel administered the surveys and analyzed the results. These instruments consisted of a presurvey administered during the first exam period, about 4 wk into the semester, and a postsurvey administered during the final exam. Response rates for the presurvey were 87%, 91%, and 96% for the spring 1997, 1998, and 1999 semesters, respectively. These rates do not include students who dropped the course; in other words, presurvey data are reported only for those students who received a grade, including a “W”. Postsurvey response rates, also based on the number of students who received a grade in the course, including those who withdrew, were 69%, 81%, and 74% for the three semesters listed above. The presurvey collected information about the college in which the student was enrolled; their previous experience with biology courses and prior enrollment in Principles of Biology; whether they had taken high school and/or college chemistry and physics; their primary reason for taking the course; the amount of time they expected to commit per week to the course; and their expected grade. Students were also asked to respond to their agreement with the statement “I really want to take this course,” and to evaluate the reputation of the course across campus in terms of its quality and degree of difficulty. A final set of items assessed student interest in various aspects of a typical biology course, including “doing experiments,” “working with other students,” “problem-solving,” “learning biology concepts”, “hands-on learning,” and learning about how biology relates to the “real world.” The latter set of items was also asked on the postsurvey. This instrument also asked students to self-report on the total time committed to the course per week and their expected grades; to respond to the statement “I learned a lot in this course”; to indicate how likely they were to read more biology-related topics in news media or popular magazines; and to comment on their level of interest in biology relative to the start of the course. The postsurvey included questions on to what extent students felt that a course instructor knew them and how much they interacted with other students. Students were asked how much they liked the course format and whether they were likely to recommend the course to other students. Finally, students were asked to report their level of skill or knowledge in various aspects of a biology course: “using lab equipment,” “using computers,” “writing lab reports,” “solving biology problems,” and their overall knowledge of biology.

Student respondents did not differ over the three semesters in which surveys were administered in terms of college enrolled, previous biology experience, prior enrollment in the course, chemistry or physics background, or reason for taking the course. There was also no difference in the grade expected by students in the presurvey (approximately 90% expected to receive an A or B); this had changed by the postsurvey, in which students had much more realistic expectations, with a significantly smaller percentage expecting A's in the studio format. (Actual final grade distributions were discussed.) Significant differences were found in self-reported learning and interests, with 64% of A-T students agreeing with the statement “I learned a lot in this course” versus 49% (1998) and 58% (1999) of studio students indicating moderate or strong agreement. There also was a lower percentage of students who reported that they were likely to read more biology-related topics and who indicated that their interest in biology had increased in the Spring 1998 and 1999 (studio) semesters as compared with the Spring 1997 (A-T) semester. To elucidate possible factors, in addition to course format, that influenced student attitudes, postsurvey responses were adjusted for students' initial levels of skills and interests as measured on the postsurvey. Significant differences between students enrolled in the course in the A-T and studio formats were found only for their self-reported skills in “doing experiments” and “working with others,” in which the studio students rated themselves higher; and “writing lab reports,” in which the A-T students rated themselves higher. Perhaps not surprisingly, students' major was a factor that significantly impacted skills and interests, with students majoring in biology and related areas reporting more skill and interest in these areas in both course formats. The major also was a factor in impacting student comfort with the course, whether in the A-T or studio format. Student motivation as inferred from the statement “I really want to take this course” and student ability as inferred from ACT composite scores also significantly impacted student learning, interest, and skill development.

In the time that has passed since these student attitude surveys were conducted, there is considerably less angst associated with the studio format. One of the areas that students continue to grumble about is the required attendance for studio sessions. However, this reinforces the idea that students are expected to be participants in their own learning in the studio and seems to be necessary for many college freshmen (or seniors). Many are at first uncomfortable with the course format, but this changes as the semester proceeds, and we have had a number of students express disappointment on learning that other biology courses are not taught in the studio format. A few students simply mark the time to complete the course and never really become engaged. It is possible that such students account for a portion of the D's and F's observed.

Faculty Attitudes.

Faculty attitudes toward the A-T and studio format were surveyed during the last semester of A-T and the first four semesters in which the studio was taught. An instrument dealing with Principles of Biology in the A-T format was administered to all biology faculty in spring 1997, with a similar survey with questions pertinent to the studio format administered to those who taught in the new format in fall 1997, spring 1998, fall 1998, and spring 1999. Response rates were good in spring and fall 1997 and then fell off quickly with time, presumably because some faculty members had been asked to complete the survey twice or more. It was not possible to know how many individual faculty members were represented in the responses collected. Faculty members were in general strongly receptive of the change. Not surprisingly, most felt that the course was an effective learning experience for students in both formats. Some faculty members had concerns about the change, which required them to be knowledgeable about the entire content of the course (e.g., a plant biologist would have to teach some classes about animal biology, and vice versa). In the A-T format, faculty members had the option of working in the A-T lab only during selected portions of the semester; such an arrangement is not available under the studio format. Other concerns included learning to use instructional technology and anticipation of an increased time commitment to the course. As time has gone by, these fears have largely been alleviated by faculty retirements, hiring of a younger group of faculty more comfortable with technology, and extensive support for faculty in the form of weekly staff meetings, staff notes, and an online site providing access to studio Web and print materials.

Student Performance in Subsequent Biology Courses

Of those students who take Principles of Biology as part of a science or prehealth profession major at K-State, nearly all go on to take one of three other biology courses: Organismic Biology, Structure and Function of the Human Body, or General Microbiology. We obtained data from the university registrar on student performance in these classes for students who completed one of these three classes in fall or spring 1998. There were three groups of students in these classes: those who took Principles of Biology in the A-T format; those who took it in the studio format; and those who took the course elsewhere (generally at a community college). The average grades of these three groups of students were compared with and without adjustment for ACT scores (data not shown). There was no significant effect of the Principles of Biology course format on performance in the three subsequent courses, particularly when the grades were adjusted for ACT score. Interestingly, there was a difference seen for students who did not take the introductory biology course at K-State; their grades in Organismic Biology (the next course in our core sequence) were generally lower, even when ACT scores were taken into account (data not shown).

Student Learning in the Studio Format

Assessment tests based on multiple-choice questions similar to those used in exams have been used in Principles of Biology for several years to gauge student learning. We report here data from four regular semesters (Fall 2002–2003 and Spring 2003–2004) and one summer session (2004). The same test was given at both the beginning (pretest) and end (posttest) of the semester. The test was not given for credit; the pretest was on the first day of class; the posttest was given on the last day of class. The tests consisted of 16 questions, two from each of the content modules of the course. The questions were based on objectives used in both the A-T and studio versions of the course. Questions were chosen such that the test has approximately equal numbers of easy (most students who have taken any previous biology course should know the answers) questions and more difficult (most students should not know the answers before taking the course) questions.

The “gain in learning” (g) metric has been used by others to measure student learning and is determined as the difference between the posttest score and the pretest score divided by the difference between the highest possible score and the pretest score (Hake, 1998; Cummings et al., 1999; Roy, 2003; Sundberg, 2003). Hake (1998) found that the average gain in learning for students in traditional courses in Physics was g = 0.23. In Table 5, we report results from these pre- and posttests. The gain in learning varied greatly from question to question, with consistently low values for quantitative and complex topics such as dilutions and photosynthesis and high values for concepts such as the ultimate source of energy in ecosystems and details of structures. The overall gain in learning for all questions and all semesters was a mean of 0.47 with a SD of 0.04.

Comparisons of Studio Student Learning with Other Courses

We cannot retroactively compare studio student learning with student learning in the A-T format; however, we have compared student learning in the Principles of Biology studio with that of students in other K-State biology courses, who also are given assessment tests at the beginning and end of the semester. The gain in learning calculated from these assessment data allowed us to compare learning in these more traditional K-State classes to the Principles of Biology studio. The other courses are:

• BIOL 455 (General Microbiology), a traditional semiweekly lecture/lab course and a required core course in our microbiology curriculum. The lecture is delivered by a K-State biology faculty member; the labs are delivered by graduate students.

• BIOL 450 (Modern Genetics), a lecture course with three meetings per week augmented with weekly recitation sections, required in all three K-State biology curricula. Lectures and recitations are delivered by K-State biology faculty members.

The data for the Fall 2006 semester, shown in Table 6, indicate that the g-value for BIOL 198 is higher and significantly different from the g-value for BIOL 450, and is also higher and significantly different from the g-value for BIOL 455. The g-values for BIOL 450 and 455 are not significantly different from each other.

We recognize that these are different assessment tests taken by different cohorts of students; however, all three are at the same university, and BIOL 198 is a prerequisite for the other two courses. BIOL 450 and 455 are taken primarily by life science and prehealth majors and show reasonable gains in student learning. We think that it is interesting that the more introductory studio course, taught to a more diverse group of students, shows a greater gain in learning than the two traditional format courses taught to students who have selected a major in the field.

We also compared student learning in introductory biology in the K-State studio format with that for a comparable course taught in a traditional lecture/lab format at Fort Hays State University (KS), another public institution in our state serving a comparable student population. Identical assessment tests were given in Spring 2007 at the start and end of the semester to students at both institutions. Data are shown in Table 7. To keep the sample sizes similar, a representative (median) K-State section was chosen; g-values for K-State sections ranged from 0.401 to 0.524 that semester. Initial test scores for these classes were similar and were not significantly different; this indicates a similar level of preparation of students in the classes at the two institutions. The amount learned (g-value) was significantly different, and higher for the K-State studio-format class. Similar results (no significant difference in presemester test scores, and significant difference in g-values) were obtained when any Spring 2007 section of Principles of Biology was used as the comparison group (data not shown). These results, while limited by the fact that they reflect different courses at different institutions, suggest that introductory biology education in the studio format may result in greater learning and retention compared with introductory biology education in a traditional lecture/lab format.

DISCUSSION

Principles of Biology is the sole introductory biology course for biological sciences and prehealth professions majors at K-State, and serves as the final natural science with laboratory elective for many students from across the university. It was taught for 30 years in an A-T format. A number of studies done at other institutions have compared A-T instruction to conventional (lecture) courses. A meta-analysis of multiple studies by Kulik et al. (1979) reported that A-T courses were somewhat more effective in promoting student achievement than conventional courses, but did not improve course evaluations or reduce student withdrawal. Our experience with the A-T biology course is consistent with these findings, although we do not have comparative data for a lecture version of Principles of Biology.

In 1997, we began a restructuring of Principles of Biology to use the then-novel studio format. It was our hope that the student-instructor interaction, active learning, and multiple learning modalities integral to the studio format would provide a superior learning environment for the course. The analysis of student performance on exam questions and student grades in the A-T and studio formats, and the qualitative assessment data on student attitudes and responses to the course in the two formats reported above, support the idea that the change in format did no harm, despite the reduction in contact hours per week (5 h per week in A-T versus 4 in studio). Another factor that should be considered in interpreting these data are that the course in the last year of the A-T format was in a mature form, the product of many years of modification. The student attitude data for the course in studio format were gathered very early in the implementation of the course, which continues to evolve as we formatively use feedback from students and faculty to modify content and delivery, and add components to enhance student learning. The latter include a series of learning objective–based study guides and weekly review sessions. Our self-published studio manual is in its ninth revision. We have used three different texts since conversion to the studio format. This format lends itself readily to minor modifications as new materials become available.

Published evaluations of studio physics at Rensselaer Polytechnic Institute have yielded mixed results in terms of effectiveness. Quantitative measures of student learning showed no improvement of the studio approach over lecture/lab (Cooper, 1998; Cummings et al., 1999). Student comments identified the fact that responsibility for learning is shifted from the teacher to the student as both strengths and weaknesses of the studio format (Cooper, 1998). This parallels our experience with students, some of whom greatly enjoy the biology studio and others who would prefer someone to lecture to them.

Roy (2003) compared student learning in a studio format versus interactive lecture demonstration for genetics and evolution and found that studio resulted in higher learning gains. Traver et al. (2001) analyzed student performance in a studio offering of biochemistry and although they did not measure learning gains directly, found significant pretest–posttest improvement. Student attitude surveys reported that most students responded favorably to components of the studio environment including use of technology and collaborative learning, and reported gains in self-efficacy that were related to knowledge gain as measured by posttest performance.

How does studio biology compare with the conventional lecture/lab format and the A-T format in promoting student learning? The gain in learning values reported in Tables 5 through 7 for studio biology and other lecture/lab courses suggests that the studio may be superior in producing student learning and retention. While the current data set is limited and we do not have (and cannot obtain) comparable data for the course in the A-T format, the findings of the relative similarity of A-T and conventional courses in promoting student learning (Kulik et al., 1979) suggest that the studio would also be superior to A-T, even though this learning might not necessarily have translated into higher grades in the studio format (Table 4). As was discussed above, the student population is heterogeneous and their reasons for taking the course are diverse.

The studio conversion process was in essence a large educational experiment that is still ongoing. It was an expensive and time-consuming undertaking, and had to be all-or-nothing because of the reconstruction of the classroom in which the course is taught. We believe that the student learning gains that we have consistently observed in the studio format indicate that this effort was a success. This project led the way for development of additional mediated instructional environments at K-State, including a studio version of physics. The studio biology course provides a model that might be considered for introductory biology at other institutions across the country, as there are now more options available to make the process less expensive and easier to implement. These include having students provide their own laptop computers and making use of software packages, laboratory exercises, and assessment instruments developed by others and available online. A partial listing of such resources is provided by Handelsman et al. (2004).

REFERENCES