| HOME | HELP | FEEDBACK | ARCHIVE | SEARCH | TABLE OF CONTENTS |
|
|
|
||||||
|
||||||||
ARTICLES |
* Department of Microbiology and Molecular
Biology, Department of Physiology and
Developmental Biology, Department of
Integrative Biology, and Department of
Instructional Psychology and Technology, Brigham Young University, Provo, Utah
84602
Submitted November 12, 2002; Revised May 21, 2003; Accepted May 28, 2003
 |
ABSTRACT |
|---|
 TOP ABSTRACT INTRODUCTIONCOURSE DESIGNRESULTSDISCUSSIONAPPENDIX AAPPENDIX BAPPENDIX CAPPENDIX DREFERENCES |
|---|
Key Words: Rasch analysis • item response theory • student outcome • student attitude • student confidence
|
INTRODUCTION |
|---|
|
TOPABSTRACTINTRODUCTIONCOURSE DESIGNRESULTSDISCUSSIONAPPENDIX AAPPENDIX BAPPENDIX CAPPENDIX DREFERENCES |
|---|
It may often be assumed, incorrectly we believe, that students will naturally acquire the habit of scientific thinking in the course of reading cell biology texts or listening to traditional descriptive presentations. While the ability to analyze experimental data may be intuitive for a small number of our students, it is not for most. Importantly, college students identify their most influential teachers as those who taught them to "think like professionals in the field" (Light, 2001). At schools with small, laboratory-intensive courses or abundant research opportunities, students effectively obtain some measure of skill in this practice of science. Where enrollments are large, however, and budgets for hands-on experience limited, nonlaboratory courses must provide much of this intellectual training.
The number of undergraduates majoring in the five departments of the College of Biology and Agriculture at Brigham Young University (BYU) exceeds 2,200. Many of the degree programs in this college require courses in molecular biology, genetics, and cell biology, which are deemed core subjects. A significant number of additional students also enroll in these lecture courses, for example, premed students majoring in the social sciences or humanities. As a result, sections of these courses with greater than 100 students are common. Laboratory exercises are not an integral feature of these courses, although students commonly take one or two separate laboratory courses that introduce the relevant experimental techniques. Still, the hands-on research experience provided by this system is minimal (cost limited), and insufficient to provide adequate training in the design, execution, and interpretation of experimental biology.
Helping students acquire skill in scientific reasoning has become the focus of the cell biology course at BYU. We report here our efforts at achieving this goal in the large nonlaboratory classroom through the use of innovative didactic strategies and examination questions that promote and assess analytical problem-solving abilities. In addition, we have sought rigorous evidence to test the effectiveness of this teaching/learning system and to determine whether it offers advantages over more traditional methods. Specifically, we have addressed the following questions:
|
COURSE DESIGN |
|---|
|
TOPABSTRACTINTRODUCTIONCOURSE DESIGNRESULTSDISCUSSIONAPPENDIX AAPPENDIX BAPPENDIX CAPPENDIX DREFERENCES |
|---|
Course Objectives
The primary focus of the course is to assist students to acquire and
strengthen the skill of correctly interpreting data generated from
experimental research in cellular biology. This is accomplished in the context
of investigating the conceptual principles that inform various topics
(membrane transport, signal transduction, etc.). At the end of the course,
successful students should be able (1) to attend a research seminar on a topic
in cellular biology, follow the presentation, and understand the speaker's
arguments and conclusions; and (2) to read a published review or update
report, designed for a general biology audience, with a similar degree of
comprehension.
Assessment
Student performance during the course is assessed through one final and
four midterm examinations. Midterm exams contain three data analysis problems
and three conceptual problems, each worth the same number of points (15 in our
scheme). Each data analysis problem consists of a paragraph describing an
experiment related to one or more of the topics covered on the exam. The
paragraph is accompanied by graphical and/or tabular representations of the
data. Two versions of the problems have been tested. In "constructed
response" versions, students are required to construct a series of
one-sentence conclusion statements supported by those data (example shown in
Fig. 1). In "selected
response" versions, students choose correct interpretations from a list,
allowing machine scoring (Appendix B). Both versions are equally effective at
discriminating student abilities, as judged by score distributions, but we
have preferred the constructed response versions requiring students to
generate rather than select answers. The conceptual problems ask students to
create a visual model or other description that reflects understanding of
basic cellular processes discussed in the course. For example, a student might
be requested to diagram the secretory pathway that explains the mechanisms
involved in targeting proteins to various compartments.
|
The final exams are comprehensive and include problems covering a larger scope than those used in midterm exams, but with similar formats. All exams are administered in the university testing center with no time limit imposed. Exam results and feedback on performance are shared liberally with students, but copies of the exam are not permanently returned. To avoid biases in student performance due to instructor personality, we used data mostly from semesters in which the course was team-taught. In a few cases, data were pooled from several semesters to average performance from sections taught by different instructors.
Didactic Methods
Use of Text. The texts used have been Alberts et al.,
Molecular Biology of the Cell, 3rd ed., and, more recently, Lodish et
al. (2000) Molecular
Cell Biology, 4th ed. Students are expected to come to each class period
having spent about 1 hr reading selected textbook pages focused on the
fundamentals of the target subject. The course philosophy is that students
should assume the primary responsibility to acquire the basic facts
(vocabulary, names and biochemistry of relevant molecules, overall mechanistic
features of processes, etc.) rather than relying on lecture from the
instructor. Instead, some class time is spent providing clarification and
correction.
We provide an extensive supplementary packet, a topic outline, containing annotations for each reading assignment. An example for the topic addressing regulation of the cell cycle is shown in Appendix C. Students are encouraged to derive a "big picture" summary as they focus on concepts that have the highest priority and decide which details need not be committed to memory. Interestingly, students tend to be intimidated by figures and diagrams and forgo carefully studying them in favor of following the text explanations. In contrast, we instruct students, if time-limited, first to focus their attention on the figures. A self-corrected quiz (with answers) covering the reading assignment is included in the topic outline. The intent is to encourage students to obtain feedback and self-assess their understanding before they meet a graded assessment by their teacher.
Class Period Agenda. A 5-min, graded quiz is administered at the beginning of each class period or on-line during a 24-h window before the period begins. The quiz usually consists of three to five items of fact to recall from the assigned reading and one or two "milestone" questions designed to test comprehension of a fundamental conceptual issue from a previous topic. An example relative to the regulation of protein function is presented in Appendix D.
The remaining 45 min of the class period is divided into alternate segments of instructor presentation and application exercises in which students work cooperatively to solve problems and practice data analysis. In the former, the teacher's role is to assist students to develop an accurate conceptual framework for the subject at hand (e.g., providing historical background for the experiments, presenting classical data not reported in the text, clarifying difficult concepts, and correcting misconceptions). We minimize lecturing and, instead, involve students in these presentations through Socratic dialogue or small-group discussions. A common didactic strategy is to require students to draw simple diagrams illustrating their understanding of concepts.
More than half of each class period involves group practice (two to four) in solving Application Exercises—analytical problems printed in the topic outline (similar to the ones used in exams; see Figure 1). In general, the task is to interpret the data and reach appropriate conclusions. As supporting activities, students are frequently required to explicitly identify the question asked by the researchers, draw flow diagrams that illustrate the salient details of the experimental design and protocol employed, or state in their own words a description of the experimental results. During this time, instructors and teaching assistants roam the classroom and participate in various individual groups by asking and answering questions. The didactic principle is to engage every student in active communication.
After Class. Additional problems comparable to the application exercises are provided for practice outside the classroom. Solutions to the majority of these problems are made available for student review, and the responses to a selected few are collected periodically and graded.
Additional assistance from the faculty is provided in formal weekly mentoring sessions with 20–30 students. Most of each hour is spent practicing data analysis with more individual attention than can be provided in class. We are therefore able to diagnose particular weaknesses and offer customized suggestions. Furthermore, these sessions promote a collegial spirit that, anecdotally, has appeared to improve class morale. Although the time required may seem excessive, we find that 3–4 h per week providing this kind of help is more productive than the same amount of time spent with fewer students during conventional office hours.
|
RESULTS |
|---|
|
TOPABSTRACTINTRODUCTIONCOURSE DESIGNRESULTSDISCUSSIONAPPENDIX AAPPENDIX BAPPENDIX CAPPENDIX DREFERENCES |
|---|
|
|
Improvement in Student Ability to Draw Conclusions from Experimental Data
A general assessment of student performance in formulating conclusions from
experimental data was obtained by use of a pre- and posttest containing three
data analysis problems (comparable to the problem shown in
Figure 1). The pretest was
administered during the first week of the course as part of a homework
assignment. Students were instructed to answer the problems without using any
resources other than their own efforts. They signed a statement on the cover
page of the assignment indicating that they had complied with that
expectation. Points were awarded toward the final grade as an incentive to
complete the assignment, but neither feedback on performance nor answers were
provided to students. For the posttest, the same problems were administered
again as part of midterm or final exams near the end of the course (i.e., at
least three fourths of the course completed).
The overall nature of the responses to the pretest problems suggested that the students took the assignment seriously and made an honest effort to answer correctly. Those few that did not attempt any of the problems were excluded from the study to avoid bias. Of the 271 persons included in the data set, 33 did not attempt an answer to all three problems. Of the 45 total problems left unanswered (i.e., 5.5% of 813 total problems), 67% contained a remark such as "I don't understand," "I'm not sure," or "I'm unable to answer." Table 2 demonstrates that there was substantial improvement in performance on each of the problems. The total scores at the end of the semester were more than double those at the beginning (65% of total points compared to 27%).
|
These data, however, cannot distinguish whether elevated scores on the posttest were due to improvement in student analytical skills or simply acquisition of relevant background information. In any course, one expects that students will know more information at the end of the term than they did at the outset. When trying to teach students to think better, one must expand the assessment question beyond asking whether they acquire the basic information to whether they improve in their ability to use that information in a meaningful cognitive process. The former is easy to address; the latter requires assessing performance independent of knowledge acquisition. For the purposes of this study, "improvement" is defined as increased ability to think scientifically. To investigate improvement in these terms, we sought additional means of assessing changes in performance that would factor out the influence of background information. Accordingly, we compared progress in student performance among course midterm examinations, where the facts and concepts pertinent to the data analysis problems were provided during the period immediately preceding the exam. Students' ability to solve data analysis problems on the midterm exams thus should depend only on progress achieved during the course.
A factor complicating this assessment is that not all exam problems that assess cognitive skills are equally difficult, despite instructor efforts to the contrary. This dilemma can be addressed by using Item Response Theory to estimate exam difficulty independent of student ability (Bond and Fox, 2000). We applied this theory to our midterm exam data using a Rasch analysis (Embretson and Reise, 2000), a complex iterative calculation requiring computer technology that has proven increasingly useful in the field of educational measurement. We refer readers to the cited references for more detailed information. The nature of the analysis required two adjustments to exam scores (for analytical purposes only and not for assigning course grades). The first adjustment was to score student responses dichotomously (i.e., completely correct or completely wrong). Rather than treat entire problems (each one sixth of the exam) in this fashion, we divided each problem into segments (individual conclusions in data analysis problems and discrete parts of the answer in conceptual problems) and scored those segments dichotomously. Second, Rasch analysis automatically and necessarily excludes all students that answered either all or none of the segments on an exam correctly (even though those students would have received partial credit in the scoring system used for assigning grades). These scores are excluded because they are outside the range of competencies assessed by the exam and therefore cannot be distinguished from infinite ability or infinite incompetence.
Table 3 displays the estimated difficulty of the first and last midterm exams for conceptual and data analysis problems. The conceptual problems were easier on the last exam compared to the first (i.e., a more negative score on the logit scale; see note to Table 3). In contrast, the data analysis problems were more difficult.
|
Rasch analysis also provides an estimate of student abilities calibrated by exam difficulties. The analysis suggested that student abilities increased similarly for both types of problems (Table 3). Due to the large range of student abilities and the relatively small improvement, this change was significant only at about the 90% confidence level (p = .12 for data analysis problems and p = .10 for conceptual problems). Nevertheless, we considered these estimates to be preliminary, because they did not represent the entire class or the full range of partial credit present in the original raw scores (due to the required exclusions and dichotomous scoring described above). Accordingly, we completed our analysis by applying the exam difficulty data (Table 3) to interpret distributions of the raw exam scores used for grading in the course. For both data analysis and conceptual problems, the average raw score increased from the first to the last exam (Figure 3). The larger apparent increment for the conceptual problems may be explained by the fact that on the last exam those problems were easier (Table 3). Conversely, the small improvement in average raw scores on data analysis problems is magnified by the fact that these problems were more difficult. The shapes of the distributions also support these conclusions because, in both cases, the distribution became more skewed toward improved performance (see skewness numbers in the legend to Fig. 3). This finding supports the preliminary argument from the Rasch analysis that student analytical ability increased.
|
We next addressed the issue of whether the observed improvement in student performance was the result of a systematic impact on the class, one that could be attributed to course design and pedagogy. To explore this question, we devised a method (termed "student mobility profile") to visualize changes in relative student performance between the first and last exams. The following describes the logic upon which this method is based.
Relative changes in student performance are first quantified by calculating
standard Z-scores (displacement from class mean normalized to class
standard deviation). Improvement relative to the class average is demonstrated
by an increase in one's Z-score. Therefore, if every student performs
the same relative to the class mean and standard deviation on both the first
and the last midterms, the change in Z-score will be zero for all
students, even if the class mean is different for the two exams. If, however,
one student improves from the bottom of the class to the top, that student's
Z-score would increase greatly. The Z-scores for the
remainder of the students would diminish slightly, not because their
performance declined, but because they did not improve as did the one student.
If several students improve, they will improve by different amounts if the
distribution of their abilities is random. In this case, a graph of increments
of change in Z-score (abscissa) versus percentage of class showing
improvement greater than each increment (ordinate) would reveal an exponential
decay (Figure 4). That is, many
students will improve by at least a small amount, but as larger increments are
considered, the number of students achieving at least that level of
improvement will decrease. When the entire class is considered, some students
improve, some remain the same, and some decline in performance between the two
exams. Thus, a plot of the entire class results in two exponential decays, one
on the positive side and one on the negative
(Figure 4). The intercept of
the ordinate on the positive side represents the sum of all students for whom
the change in Z-score was 
0. The intercept of the ordinate
approached from the negative side represents the sum of all students for whom
the change in Z-score was <0. Therefore, the sum of the
two intercepts will equal the total number of students in the class. The shape
of the decay is determined by the average increment of improvement (or
decline). When large improvements (or declines) are very rare, the curve will
be sharp. A broad curve results when many students improve (or decline) by a
large amount. We define the minimum degree of change exhibited by one half of
the students on either side of the graph as a "mobility
coefficient." Thus, two mobility coefficients are generated per class,
one on the positive side and one on the negative. The mobility coefficient is
analogous to a half-life. Because the number of students in the class is
fixed, a change in the mobility coefficient on one side of the graph relative
to the other will require reciprocal changes in the intercepts (see cases 2
and 3 in Figure 4).
|
If the probability of either improvement or decline in Z-score is the same for each student, the curves in the positive and negative directions will be symmetrical, giving identical intercepts and mobility coefficients for both sides of the graph. This condition is labeled "case 1" in Figure 4. Functionally, it represents a class in which improvement or deterioration is determined by random events that affect each student independently (e.g., changes in motivation, health, employment, and financial status). Alternatively, a negative pressure applied generally to the class (e.g., teacher neglect, general decline in class morale), would make it more likely that students would decline than improve. This asymmetry has two effects on the class. First, the average size of decrements in performance will be greater than the average size of improvements. Second, because the class mean will be lower on the second exam than it would have been without the negative pressure, the Z-scores of some students will be raised by a small amount even though their performance did not really improve. The graphical consequence of these effects is an asymmetry in the mobility profile. The mobility coefficient will be greater on the negative side than on the positive, but the intercept will be greater on the positive side. We have defined this scenario "case 2." Case 3 is the converse of case 2, in which a positive force on the class raises the probability of improvement higher than would have been achieved through random effects. Although the mobility profile does not address the issue of whether the total class performance improved during the semester (because Z-scores are normalized to class average), it does identify whether changes in performance were the result of random (case 1) or systematic effects (cases 2 or 3). An important advantage of the mobility profile analysis is that it applies to intrinsic properties of a course. Different courses can therefore be compared even though they focus on different topics and/or administer different exams.
Figure 5A applies this approach to student performance on one semester's data analysis problems. As predicted, the number of students both with improved and with diminished Z-scores decayed exponentially over increasing increments of change. We fit the data to a first-order exponential decay and computed the mobility coefficient and the intercept. The asymmetrical shapes of the positive and negative curves in Figure 5A are consistent with the pattern of case 3, suggesting that a systematic pressure promoting improvement of skills occurred in the course. Data from several semesters are summarized in Figure 5, C and D. The results supporting case 3 for data analysis problems were reproducible, i.e., the mobility coefficient was greater on the positive side of the profile (Figure 5C), and the intercept was greater on the negative side (Figure 5D). The conceptual problems serve as an internal control, since our efforts focused much less on helping students improve their skills in memorizing conceptual information. Indeed, consistent with case 1, no difference between the positive and the negative sides of the profile were observed for either the mobility coefficient or the intercept (Figure 5, C and D).
|
Comparison to a Course Not Offering Practice in Data Analysis
Students in our course and those in a separate cell biology course taught
at the same curricular level were administered data analysis problems.
Specifically, these problems used data from a study on expression of the
globin gene (Kadonaga and Tjian,
1986) and from work on the interaction of growth hormone and EGF
receptor tyrosine kinase systems (Yamauchi
et al., 1997). Both courses cover the information on gene
regulation and signal transduction necessary to understand the background for
the problems. The second course was taught with the majority of class time
spent with lecture presentation of information by the instructor including
multiple elaborate visual images. The primary objective was to convey a
state-of-the-art description of cellular mechanisms and processes.
Consequently, students in this second course were not given directed practice
in data interpretation tasks. Nevertheless, the text used for this second
course contains a detailed description of the original experiment and data
upon which the globin problem was based
(Cooper, 2000, pp.
244–247). This description was part of the required reading assignment
for students in that course. Table
4 demonstrates that performance on this problem was significantly
higher for the group that practiced data analysis tasks. The same was true for
the problem on receptor tyrosine kinases
(Table 4).
|
Effect of Data Analysis Practice on Acquisition of the Basic Concepts of Cell Biology
This question was addressed by administering two exams covering the same
subjects (equilibrium and free energy, allosteric and covalent regulation of
protein function, protein turnover, antibodies, membrane structure, membrane
transport, and membrane potential). The first exam was a scheduled midterm in
the course focusing on data analysis problems as described under COURSE
DESIGN. Student preparation was the same as that typically employed in the
course, with emphasis on data analysis skills. The second exam focused
entirely on recall of factual information and was administered one week later
as a surprise exercise with no additional preparation or return to the subject
matter.
Distributions of class performance on these two exams are shown in Figure 6. Even though students had not studied specifically for the recall exam, they performed better on it than on the data analysis exam (mean improved by 9%, standard deviation decreased by 6%). The important issue was whether performance on the data analysis exam predicted achievement on the subsequent recall exam. As shown in Figure 6C, a significant relationship between performance on the two exams was demonstrated (p < .0001, r = .6).
|
|
To assess attitudes about different learning strategies, students were asked to rate their opinion of the effect of courses emphasizing data analysis and those emphasizing recall of factual information on a series of qualitative benefits and risks. These benefits and risks were later divided into two categories to facilitate interpretation of the results. "Academic features" included questions about general utility and relevance of courses and their ability to provide intellectual stimulation. The "personal implications" category focused on the effects of courses on student psychological well-being and the achieving of practical goals. Student attitudes regarding academic features were somewhat more positive about courses that focus on analytical reasoning than courses emphasizing recall (5.2 ± 1.2 vs. 4.5 ± 1.0, mean ±SD, p=.0004, by analysis of variance, n = 119, assessed at end of the semester). In contrast, attitudes were more positive about recall-type courses when one considered personal implications (4.6 ± 1.2 vs. 5.2 ± 0.8, p < .0001, n = 119). In both cases, these attitudes were virtually identical to responses obtained at the beginning of the course.
Specific attitudes about the course were assessed using an anonymous questionnaire at the end of each semester. The responses to selected questions, summarized in Table 6, suggested an interesting dichotomy in reactions to the course. On the one hand, there was a strong expression of support (about 80%, items 1 and 2) for the value of the analytical approach to learning and its personal value in meeting future goals. Nevertheless, many students were not in favor (items 3–5) of repeating the same emphasis or using it in another course. We attribute this apparent contradiction to the rigor required and a sense that the academic background of most students had not included preparation for this approach to education.
|
|
DISCUSSION |
|---|
|
TOPABSTRACTINTRODUCTIONCOURSE DESIGNRESULTSDISCUSSIONAPPENDIX AAPPENDIX BAPPENDIX CAPPENDIX DREFERENCES |
|---|
In many traditional classrooms, the teacher as authoritative source presents information and models the language and practice of the discipline for the listening student whose role is primarily passive, usually restricted to making notes. The decision to introduce active problem solving and increase the number and kind of instructor–student and student–student interactions forces a redefinition of the teacher's role. In our system, the learning cycle begins with the students acquiring the fundamental information about a new topic in cell biology through preclass study. Putting students in control of sorting, prioritizing, and assimilating the basic facts is disconcerting to many at first. They have become accustomed to a comfortable rubric in which, during a lecture, the teacher organizes a hierarchy of knowledge and ranks what is most important, and they become uneasy when given full responsibility for these aspects of learning. Our use of annotated reading assignments has seemed very effective at this stage; we are able to focus the students' attention on a smaller, more manageable subset of potential topics. The graded quizzes at the beginning of (or before) each class period provide a strong incentive to come to class prepared. Students know that they can expect the faculty to provide an inquiry-based exercise that will serve to clarify a particular topic, but not remedial explanations of the basics for those who have not done the reading. They also realize that for much of the lecture period they will be in an active working mode that will not permit them to remain silent.
Solving an Application Exercise problem in groups of two to four students is one good way to get them talking. With 50 simultaneous conversations focused on the correct interpretation of the data in a figure, the classroom is a very noisy place. When we inject ourselves into one of these discussions, the goal is to answer a question or offer a discreet hint without robbing the students of the opportunity to solve the mystery for themselves. It can be something of an intellectual epiphany for the person who resists the temptation to resort to getting help from someone else and persists through the process of independent effort to resolve the problem. In this setting, the teacher becomes coach, offering suggestions for improvement in technique. "So, Mary, can you speculate about the meaning of these data: the level of radioactive protein remains constant in the animals with high substrate levels, while counts in the controls go down?" This also describes the teacher's function in the more intimate and informal confines of the mentoring session with 20–30 persons. Here the interaction may be even more effective because of its more personal nature and the more economical use of time.
Even when our purpose is to introduce a topic or devote some minutes to clarifying a concept (by projecting a figure from the text, for example), we attempt to talk less and let students carry the conversation. They should also learn visual as well as verbal articulation. We continually urge students to construct simple drawings of experimental protocols (e.g., DNA footprinting) or summary diagrams of all the mechanistic elements in a process (transcriptional regulation, for example). The rule of thumb is to get brains actively engaged with visual symbols that can demystify abstractions.
It is not uncommon, near the conclusion of the traditional lecture, for the teacher's invitation to raise questions to be greeted with silence. The best explanation for this lack of response is that at this point in the learning process most students do not yet know what they do or do not understand. Unfortunately, many have not developed a self-assessment strategy that allows ideas to be solidified or misconceptions and omissions to be identified and corrected. The biology teacher has a real opportunity to help correct this deficiency. What is needed is a confrontation between student and teacher that will require the student to demonstrate that understanding really has occurred.
Because students have not yet learned to confront the ideas effectively on their own, we confront them by inviting them to be accountable. In this way, they acquire the skill of self-confrontation and can practice it independently in the future. This is not to suggest a combative, "in your face" scenario, but an interaction with the teacher, in the spirit of helpfulness, that stimulates a student to communicate. Any pedagogical device that helps the learner to articulate an idea ought to be tried. "Tell the person sitting next to you what the question was that this experiment attempted to answer." "Please come to the board and teach us how a G-protein works.""Everyone draw a new curve on that figure to represent the result you think will occur under the following new experimental conditions."
Finally, it is our experience that we are better equipped to offer individuals a meaningful evaluation of their performances following an exam consisting of data analysis problems of the type described in this paper. While more traditional objective exam questions may permit a teacher to diagnose deficits in factual knowledge, we can more readily pinpoint a weakness in an important cognitive skill and prescribe a specific remedy. Common weaknesses include an ability to cite the name of a molecule or process without knowing what it really is or does, having an inadequate overview of a major concept, misunderstanding the conduct or purpose of an experimental methodology, not reading the axes of a set of coordinates correctly, being able to restate an experimental result—"The slope of the line changed when the new reagent was introduced"—without drawing a meaningful conclusion from it. The student benefits from both the requisite preparation before the exam and the feedback afterward.
Proof for the efficacy of commercial instructional materials, touted to improve the educational experience for students, usually takes the form of anecdotal endorsements like, "Highly praised by our test sample of participants!" Such evidence is not compelling. But we would like to have reasonable assurance that the difficult work of course restructuring will result in genuine improvement. Thus we have provided examples of the kinds of empirical evaluations that are appropriate to assess the effectiveness of course objectives, design, didactic strategies, and examinations.
Student Ability to Draw Conclusions from Experimental Data Improves During the Course
Based on the data shown in Tables
2 and
3 and
Figure 3, it is clear that
students improved during the semester in their ability to answer data analysis
problems. Furthermore, the data in Figure
5 suggest that the improvement was systematic and not explainable
by the usual random effects of individual circumstances and motivation.
Providing empirical documentation that students really do increase in
analytical ability has proven to be more subtle and sophisticated than we
initially supposed. Moreover, depending on one's point of view, the degree of
change our students have achieved in one semester may appear to be modest. We
would argue, however, that thinking well is hard work, and as helping others
to think well is equally difficult, any progress in this direction should be
celebrated.
Directed Practice of Data Analysis Problems Offers a Significant Advantage in Developing Skill at Scientific Reasoning
Performance on data analysis problems was better among students whose
course required practice of the requisite skill than among those without such
practice (Table 4). This
observation does not mean that students in the "acquire
information" course did not learn concepts that could be applied to data
analysis problems. For example, if we assume that performance on the pretest
problems is representative of all students prior to cell biology instruction,
we can compare the data in Tables
2 and
4 to consider whether there was
some apparent gain in student ability in the "acquire information"
course. Indeed, students in the "acquire information" version
scored an average of 43% on data analysis problems
(Table 4) compared to the
average of 27% observed for students not yet exposed to either cell biology
course (Table 2). This result
suggests that gaining a conceptual understanding permits students to utilize
whatever level of inherent analytical skill they possess. Intensive directed
practice in data interpretation added significant additional gain. This skill
is not acquired serendipitously; the course must be intentionally designed and
managed in order for meaningful improvement to take place.
Despite the Emphasis on Analytical Skills and the Resulting De-emphasis on Transmission of Factual Information, Students Still Acquire the Basic Information of Cell Biology
The data in Table 3 and in
Figures 3 and
6 demonstrate that our students
performed well when tested on their recall of factual/conceptual information.
In fact, our experience suggests that students assimilate the basic facts of
the subject better in an experimental context, while practicing data analysis,
than they would if the facts were presented descriptively in a traditional
lecture format. Certainly, extensive research has validated the idea that deep
and well-retained learning require active practice
(National Research Council,
2000). Importantly, the typical element of intense study
immediately prior to the exam was absent in the exercise illustrated in
Fig. 6B. Thus, one might argue
that not only is there no risk to teaching in this manner, but there may be a
long-term benefit with respect to information retention. In support of this
interpretation, students who performed better on the data analysis exam tended
to perform better on the recall exam
(Figure 6C).
These Methods Have Varying Effects on Student Attitudes and Confidence
One of the challenges of restructuring a course is maintaining positive
student attitudes. Students often find that our course does not match their
expectations; the focus on scientific reasoning is unfamiliar and difficult
for most of them. Some argue that analytical skill is a genetic legacy that
should not be graded; "Unfairly, this kind of an exam tests I.Q., and
doesn't reflect the effort I made in preparation." While it is probably
true that a small number of students come to our course with a strong native
aptitude for analysis, the data summarized in Tables
2 and
4 convince us that few solve
these kind of problems readily and that most individuals, even the most
gifted, make substantial progress. Furthermore, there is clear evidence
(Table 5) that the course
promotes confidence in dealing with biological information (text, graphical,
diagrammatic). We note, however, that our assessment reveals a contradictory
set of affective responses to the course experience
(Table 6). Nearly all of our
students endorse the benefitof improved reasoning, but the rigorous exams that
require new skills (which most prior courses have not helped to develop)
generate some feelings of frustration and resentment. There is, of course,
some concern about the impact of grades on personal goals such as acceptance
into postgraduate programs. Teachers should not adopt these sorts of methods
believing that all students will be pleased or converted.
Notwithstanding student frustrations with the rigor and novelty of the approach, instructor enthusiasm, charisma, and willingness to give personal help can do much to alleviate these apprehensions. Nevertheless, there is a limit to what students can accomplish in a single semester. We have frequently heard this concern: "Why haven't we been exposed to this kind of learning earlier?" This suggests that the next level of reform for teachers who value these analytical skills is curricular. It may be necessary for departmental or even campus-wide groups of faculty to redesign entire programs with the view of an early introduction of instruction that systematically promotes skill in scientific reasoning.
|
APPENDIX A |
|---|
|
TOPABSTRACTINTRODUCTIONCOURSE DESIGNRESULTSDISCUSSIONAPPENDIX AAPPENDIX BAPPENDIX CAPPENDIX DREFERENCES |
|---|
|
|
APPENDIX B |
|---|
|
TOPABSTRACTINTRODUCTIONCOURSE DESIGNRESULTSDISCUSSIONAPPENDIX AAPPENDIX BAPPENDIX CAPPENDIX DREFERENCES |
|---|
Based on these data what mutation(s) is(are) possible in the variant cells?
|
|
APPENDIX C |
|---|
|
TOPABSTRACTINTRODUCTIONCOURSE DESIGNRESULTSDISCUSSIONAPPENDIX AAPPENDIX BAPPENDIX CAPPENDIX DREFERENCES |
|---|
A. Before-class assignment: pp. 863–879
|
APPENDIX D |
|---|
|
TOPABSTRACTINTRODUCTIONCOURSE DESIGNRESULTSDISCUSSIONAPPENDIX AAPPENDIX BAPPENDIX CAPPENDIX DREFERENCES |
|---|
|
ACKNOWLEDGMENTS |
|---|
|
FOOTNOTES |
|---|
¶ Corresponding author. E-mail address: William_Bradshaw{at}byu.edu.
|
REFERENCES |
|---|
|
TOPABSTRACTINTRODUCTIONCOURSE DESIGNRESULTSDISCUSSIONAPPENDIX AAPPENDIX BAPPENDIX CAPPENDIX DREFERENCES |
|---|
Bardeen, M.G., and Lederman, L.M. (1998). Coherence in
science education. Science 281,178
–179.
Bond, T.G., and Fox, C.M. (2000). Applying the Rasch Model: Fundamental Measurement in the Human Sciences, Mahwah, NJ: Lawrence Erlbaum Associates.
Bonwell, C.C., and Eison, J.A. (1991). Active Learning: Creating Excitement in the Classroom. 1. Washington, D.C., George Washington University, School of Education and Human Development. ASHE-ERIC Higher Education.
Brown, M.S., and Goldstein, J.L. (1974). Familial
hypercholesterolemia: Defective binding of lipoproteins to cultured
fibroblasts associated with impaired regulation of 3-hydroxy-3-methylglutaryl
coenzyme A reductase activity. Proc. Natl. Acad. Sci. USA
71,788
–792.
Cooper, G.M. (2000). The Cell: A Molecular Approach, 2nd ed., Washington, DC, and Sunderland, MA: ASM Press and Sinauer Associates.
Crick, J.E., and Brennan, R.L. (1982). GENOVA: A Generalized Analysis of Variance System (computer program and manual), Iowa City, IA: American College Testing Program.
Cronbach, L.J., Gleser, G.C., Nanda, H., and Rajaratnam, N. (1972). The Dependability of Behavioral Measurement: Theory of Generalizability for Scores and Profiles, New York: John Wiley and Sons.
Embretson, S.E., and Reise, S.P. (2000). Item Response Theory for Psychologists, Mahwah, NJ: Lawrence Erlbaum Associates.
Kadonaga, J.T., and Tjian, R. (1986). Affinity
purification of sequence-specific DNA binding proteins. Proc. Natl.
Acad. Sci. USA 83,5889
–5893.
Karp, G. (1999). Cell and Molecular Biology: Concepts and Experiments, 2nd ed., New York: John Wiley and Sons.
Kouzarides, T., and Ziff, E. (1988). The role of the leucine zipper in the fos-jun interaction. Nature 336,646 –651.[CrossRef][Medline]
Light, R.J. (2001). Making the Most of College: Students Speak Their Minds, Cambridge, MA: Harvard University Press.
Lodish, H., Baltimore, D., Berk, A., Zipursky, S.L., Matsudaira, P., and Darnell, J. (1995). Molecular Cell Biology, 3rd ed., New York: Scientific American Books.
Lodish, H., Berk, A., Zipursky, S.L., Matsudaira, P., Baltimore, D., and Darnell, J. (2000). Molecular Cell Biology, 4th ed., New York: W.H. Freeman.
National Research Council. (2000). How People Learn: Brain, Mind, Experience and School, Washington, DC: National Academy Press.
Russell, I.J., Hendricson, W.D., and Herbert, R.J. (1984). Effects of Lecture Information Density on Medical Student Achievement. J. Med. Educ. 59,881 –889.[Medline]
Smeyne, R.J., Klein, R., Schnapp, A., Long, L.K., Bryant, S., Lewin, A., Lira, S.A., and Barbacid, M. (1994). Severe sensory and sympathetic neuropathies in mice carrying a disrupted Trk/NGF receptor gene. Nature 368,246 –249.[CrossRef][Medline]
Strasser, A., Harris, A.W., Corcoran, L.M., and Cory, S. (1994). Bcl-2 expression promotes B-but not T-lymphoid development in scid mice. Nature 368,457 –460.[CrossRef][Medline]
Wickman, K.D., Iniguez-Lluhl, J.A., Davenport, P.A., Taussig, R., Krapivinsky, G.B., Linder, M.E., Gilman, A.G., and Clapham, D.E. (1994). Recombinant G-protein beta gamma-subunits activate the muscarinic-gated atrial potassium channel. Nature 368,255 –257.[CrossRef]
