Gauging the Gaps in Student Problem-Solving Skills: Assessment of Individual and Group Use of Problem-Solving Strategies Using Online Discussions

INTRODUCTION

The ability to solve problems is one of the salient characteristics of successful scientists. In recognition of this, much of the literature on how science education should be changed for the twenty-first century emphasizes the necessity of providing exposure to, and regular practice with, a scientific approach to problem-solving strategies (National Research Council [NRC], 2003; Voet et al., 2003; Heron and Meltzer, 2005). Consequently, if educators are to improve science education, we must be better able to discern deficiencies in the problem-solving strategies of our students. However, in large classes where an instructor may not interact with all students individually, it is often difficult to determine whether students know how to proceed when confronted by a scientific problem, let alone whether they actually use any strategy. To address this obstacle, the authors have taken advantage of the technological option of online asynchronous discussion groups in our large-enrollment biochemistry classes, using Problem-Based Learning (PBL) cases as a tool to provide practice in solving problems (Barrows, 1998; Duch et al., 2001; Koschman, 2002; Torp and Sage, 2002; Anderson et al., 2005). To monitor students, we have developed a tracking method that allows us to assess students' problem-solving approaches and provides us opportunities to help them progress.

Earlier work indicated that using PBL cases provided students exposure to and practice in problem solving that translated into higher scores on authentic assessments of these skills (Anderson et al., 2005). We recognize that PBL can be criticized for the high level of faculty effort required, and we do not imply that PBL is the best or only way to introduce problem-solving strategies; however, this method is a successful tool for generating discussion, through which students' critical-thinking strategies can be followed.

PBL has been variously modified for use in distance learning situations for more than 10 yr (Naidu and Oliver, 1996; Bonk et al., 2001). Evaluations of the use of online discussions (based on PBL cases or on other prompts) have pointed out some disadvantages of students not being in face-to-face contact during conversations: lack of facial cues from other discussants, delayed rather than immediate responses (in asynchronous discussions), and decreases in overall group learning. Conversely, online discussions can be beneficial to some students who are normally shy, or who have English as a second language. Asynchronous online communication allows these students more time to develop their responses and to participate more confidently (Bullen, 1998; Sage, 2000; Orrill, 2002; Ronteltap and Eurelings, 2002; Zhang and Peck, 2003; Stromso et al., 2004).

There are very practical advantages of electronic communication even in traditional on-campus classes, and the major benefits to this study of holding PBL case–based discussions online were in fact logistical; all students in a large class were able to “talk” within their small groups despite their incongruent schedules, the lack of small rooms in which to meet, and limited number of tutors. In addition, the use of online discussions had the procedural advantage of allowing the group conversations to be kept as permanent transcripts, making them available for multiple evaluations, and in fact for potential longitudinal assessment.

MATERIALS AND METHODS

Course Structure and Case Design

Over the past 3 yr, we have used PBL cases with biochemical problems as the basis for online discussions during 10 classes of 60 to 150 students in three different biochemistry courses: both classes of a two-semester sequence for our biochemistry majors (two times each); and a one-semester service course, primarily for premedical and prepharmacy students (six times).

Students were presented with four to six PBL cases per semester through the course website. Many of our PBL cases were originally written for use in face-to-face small group discussions (Anderson et al., 2005), and needed only minimal modification to fit the online format. Additional cases were designed specifically for electronic implementation.

The initial case presentations never provided sufficient information to allow an immediate solution of the puzzle. An example of a vague initial case presentation is given here:

“CSI-Albuquerque”

In such a presentation, the students were provided with an engaging scenario, and their initial challenge was to identify the true nature of the problem and then develop a strategy to arrive at a solution. Each group engaged in an iterative, collaborative discussion to develop their experimental and analytical strategies to make progress with the problem. Without additional information, it was impossible to solve the case; consequently, the task was to develop that information-gathering strategy.

Asynchronous student discussions were carried out using the commercially available discussion board vBulletin (Jelsoft Enterprises), but there are several different products that could function equally well. As with any technology, it is important that whatever system is used, it is robust, usable, and supported by the vendor.

During the first week of each course, all students were required to register on the discussion site with their own unique user name and password and were then randomly placed into small groups of five to 10 students. Students were not allowed to view the discussions of other groups; they only had access to their own groups' postings. A short online tutorial on how to use the discussion board was provided for training. The first case in each class was graded only for participation and so served as a primer to help the students learn the technology.

During the course of the 3-yr study, we used two different case formats that varied in the amount of instructor interaction required. The decision of which format to use was based upon both the type of course (i.e., for majors or nonmajors), and on the student-to-faculty ratio for the class. One format required minimal faculty interaction with students. The cases were presented to the students, data sets necessary to solve the case were provided at particular times, and the instructors provided relatively little other input. At the end of the case the instructors provided a summary, and sometimes discussed the case in the lecture portion of the course. The second format required more interaction between the instructors and the student groups. The instructors actually became active members of the online conversations, and as such were able to question and probe student understanding of the concepts under discussion. Requests by the student groups for experimental results or other data had to be justified. This case format required daily input by the instructors; it took 1 to 2 h per day to adequately monitor and participate in 10 to 20 different tutorial groups. The authors view the less interactive format as still very valuable to our students simply because it increases communication skills in the “languages” of biochemistry and problem solving; however, the remainder of this article will focus on the more interactive of the two case formats, because it has the greater potential to provide insight into students' problem-solving strategies.

Assessing Group Performance in the Case Discussions

Student contributions to the group discussion (postings) variously took the form of initial and secondary hypotheses, requests for data from the instructor, biochemical information pulled from texts or other resources, analyses of data provided, and summaries or integration of the information available at each stage of the case. The instructors acted as guides to keep the online conversation on track, and as sources for data. Concurrent with the online case-based discussions, which lasted about 2 to 3 wk each, the students were exposed to text readings and traditional in-class presentations that included content pertinent to the case. However, the solution was not provided in lectures.

A portion of the course grade depended on performance in the case discussions; the percentage of the grade varied from 10% to 15% according to the class and year. The scientific content of each individual student posting was given a numerical score from 1 to 10, indicating its closeness to the final solution based on a previously defined grading rubric, which had been evaluated by several instructors, used with several classes, and modified as necessary in an iterative process. The grading rubrics became stable and easy to apply to new groups after two to three modifications; an example is shown in Figure 1. This rubric corresponds to the initial case presentation for CSI-Albuquerque above.

At the beginning of a case, it was expected that a contribution score would be low; a PBL case by definition starts as an ill-defined puzzle, without enough information present to allow an immediate solution. As the students in their groups defined the problem, asked appropriate questions related to their learning issues, and received specific data from the instructors, they were able to hone their postings more narrowly. Thus, the higher scores that were earned as the discussion progressed indicated that cumulative correct concepts had been expressed by the group; a student in a group could not receive a score of 9 unless all of the earlier partial correct answers had been stated by some member of the group. The highest contribution score achieved within the group as a whole was defined as the final group grade. Individual student grades were then assigned relative to the group grade, based on both participation and quality of individual contributions. A student who was consistently active in assisting the group to progress toward the final answer was assigned the final group grade, even if he/she was not the one who first reached the solution. Students who contributed only at the beginning or end of the case, or who simply copied/pasted from websites or from previous postings, were assigned grades equal only to their highest individual contribution score.

To evaluate reproducibility in applying the grading rubrics, two faculty members and three senior biochemistry students acting as teaching assistants reviewed the basic science supporting the case and discussed the application of the grading rubric. All five individuals then independently graded the same groups. At the end of the case the grades assigned by the five graders were compared.

Assessing Individual Performance in the Case Discussions

Some students who had seemed to be successful at solving problems as part of a group subsequently failed course exam questions that relied on problem-solving skills. In these instances the transcripts of the group discussions were reanalyzed for a further assessment of the strategies of these students. For this supplemental analysis the scientific method was used as a way to categorize student contributions; each posting by an individual student was classified as belonging to one of the following steps in the problem-solving/critical-thinking process, defined by the authors as “Domains”:

	Is the student generating reasonable hypotheses?
	Is the student proposing an appropriate investigative strategy?
	Is the student correctly analyzing supplied data?
	Is the student integrating conclusions from the data with known biochemical mechanisms or facts?
	Is the student reflecting on their own contributions, or planning to address potential new problems as a result of their proposed intervention? (This final question is our attempt to probe the student's metacognitive abilities.)

In some cases it was necessary to add a sixth category (“other”) if a student's contribution could not be assigned to any of the above domains. Because such evaluations of a student's contributions took place well after the completion of the case discussion, this domains-based assessment did not impact the student's grade.

Student Opinions

Student opinions on the educational usefulness of the online case discussions were collected using standardized questions on the university-required, end-of-course evaluations. Results of these surveys were not provided to the faculty until after final grades had been submitted. Students majoring in biochemistry at this institution have many chances for significant interaction with faculty members. Consequently, an educationally productive relationship is usually established between student and faculty. Typically, students are very honest and do not hesitate to voice their opinions if they are dissatisfied with some element of the curriculum. When appropriate, this informal information was also collected and evaluated by the instructors.

RESULTS

In our 3 yr of using this approach we consistently found that students were essentially experts in the use of the discussion board by the end of the first case. They became facile in posting, downloading, and attaching any type of media in support of their arguments. In classes of approximately 100 students there were typically fewer than 5% who had any difficulty learning how to get online or how to use the discussion software. The online training program provided for students did not appear to be needed. (The use of the discussion board software was also demonstrated in the lecture component of the classes). For those students having software problems, a 10- to 15-min individual tutorial with a faculty member usually corrected the problem.

Assessing Group Performance in the Case Discussions

Student contributions were graded by application of the grading rubrics each time the instructor entered the discussion and surveyed the progress of the group. Sequential contributions were assigned a score, based on the rubrics, and the score was immediately recorded on a spreadsheet. This allowed a chart to be automatically constructed that plotted the individual contribution score versus sequential contribution number, generating a group discussion profile. The results of such an assessment are shown in Figure 2. Successful groups showed a pattern of sequential postings that had a trend of increasing scores over the course of the case, and, although the students in the specific group shown did not ultimately arrive at the correct solution of the problem (as indicated by final posting scores of lower than 10), the progress of the group is clear.

As shown in Figure 2, the different graders were relatively consistent in assigning a score, and identical conclusions about either group or individual performance could be derived from the grades of any of the five graders. In a similar analysis of three other different online cases, the r between assigned grades for any pair of the five graders was 0.83 ± 0.07, which suggests that there was a considerable degree of reproducibility and consistency in applying the grading rubrics.

The type of sequential bar graph presentation shown in Figure 2 was used to determine different groups' levels of success. Figure 3 compares the performance of four groups in the solution of the same online case. Differences related to group process and problem-solving strategy were readily apparent.

Assessing Individual Performance in the Case Discussions

The sequential bar graphs of contribution grades allowed the instructors to identify groups in need of faculty intervention. They also allowed us to assess individual student performance within a group. Figure 4 highlights the performance of an individual student (red bars) within the group. In this case, the student performance paralleled group performance; both the group and the one student discussed issues that were, increasingly, focused on the correct solution to the problem.

The student pattern profiled in Figure 4 seemed to suggest that he/she was learning how to apply a problem-solving strategy by working with the group. However, when that student was required to solve a similar problem single-handedly, the student failed. To reconcile this inconsistency, each contribution of the student was re-evaluated, not for closeness to the solution as defined by the grading rubric, but in the context of the problem-solving/critical-thinking domains defined earlier.

The domains analyses for several different individual students are shown in Figure 5. Each subfigure illustrates one student's sequential postings during one PBL case. Each contribution was categorized by both problem-solving/critical-thinking domain, and for the closeness to the solution of the problem as defined by the grading rubric. In this presentation, a color value rather than a number indicates the rubric score; lighter colors represent contributions that were further from the solution and the darker colors represent contributions that were closer to the solution to the case. After applying the domains analysis to many students (≈50 over the course of the study), several common but different patterns for problem-solving strategies have become evident. We saw these patterns in the first year of this study and have since seen them many times. Figure 5 illustrates four of the most frequent.

We have used this type of domains analysis to evaluate some students over several semesters of courses that used the online group discussions. We have found that most students take cues from the feedback of their instructors and peers and become more practiced and comfortable in applying a common problem-solving strategy, the scientific method. However, there are some who do not learn from their lack of success and who repetitively apply the same failed strategies. Without direct faculty intervention, these students do not develop problem-solving skills. Figure 6 depicts one such student's attempts, in six different cases, over two semesters.

Student Opinions

Student evaluations of the online PBL cases varied according to class population (majors or nonmajors), but in general, ≈60% of the students in all classes found the case studies to be helpful for their understanding of biochemical concepts, and also an enjoyable experience. Some students even expressed the opinion that the cases were excellent practice for developing the types of communication skills that they would need as professionals. Those with truly negative comments (≈10% in all classes) voiced concerns that were similar to those reported in other studies (Bullen, 1998; Sage, 2000; Orrill, 2002; Ronteltap and Eurelings, 2002; Zhang and Peck, 2003; Stromso et al., 2004): the online case discussions took too much time; it was too impersonal a way to communicate; it was too difficult to understand what was expected; content information was not provided to students as in other classes; and technological difficulties or inequities, such as restricted access to computers, made the process unfair. By far, the most common negative comment was that students did not trust their peers to contribute correct biochemistry content.

DISCUSSION

To initially determine whether the interpretations of the student data could be used for accurate or constructive individual feedback, the authors conducted a mini-experiment; two faculty members, who had not previously interacted with this population of students, were provided student problem-solving/critical-thinking profiles of the type described in this article. Both faculty were first given an explanation of what the profiles were based upon and then were asked to identify deficiencies in student performance and problem-solving strategies based solely on these profiles. The opinion of an instructor who had actually worked with the students in classes was considered as the “gold standard.” There was remarkable consensus between the instructors who knew the students and those who based their assessments solely on the profiles. Although the results of this comparison are not statistically significant, the congruence in instructor evaluations about student abilities suggested that the domains-based profiles did have validity and provided the authors the impetus to expand the investigation and continue to evaluate the profiles.

In each semester there were only a few students with difficulties accessing the discussion site, and these difficulties were usually caused by browser settings or individual computer firewalls. More problematic than technological glitches were our students' expectations and attitudes concerning the case discussion assignments. Most had never been asked to do this kind of learning before and approached the problem by trying to solve the case with only the limited amount of information initially provided. They searched websites for pattern-matched solutions, rather than developing a strategy for gathering and analyzing data, despite what we, as instructors, thought were very clear suggestions for how to use the precepts of the scientific method in the process of solving the problem. We eventually found that if we allowed students to flail around unproductively, sometimes failing in the first case, and then used the lecture part of the course to role-model problem-solving strategies, the learning was more efficient, effective, and robust.

Constructing the problems and grading rubrics for the online PBL cases is not a time-intensive task. The first year a case is used, more faculty time is involved, but this first use allows us to refine the grading rubrics and the responses to group requests for data. During subsequent years, cases are used with alterations only in the data and the final solution. For example, in the CSI-Albuquerque case presented earlier, the elderly professor has variously died of deliberate poisoning by arsenic and malonate, and of beri-beri brought on by alcoholism and poor nutrition.

We chose the graphical representation of the concentric “target” rubric because it made it easy to grade very quickly, a practical necessity because of the many student postings. In addition, when students were shown the grading rubrics after a case, they appreciated the conceptual simplicity of working “inwards” to the ultimate answer. In some cases, the concentric rubric could be superimposed over a metabolic map, and the process of focusing in on the correct biochemical process became clearer.

As shown in Figure 2, the five different graders were quite consistent in applying the rubric to assign contribution scores. However, it was instructive to focus on the postings for which there were conflicting grades. These tended to be contributions in which one of two situations was occurring: either the student discussed multiple aspects of the case, and different graders focused their grading on different components of the contribution; or, the student was clearly guessing, and happened to get close to the correct answer with a speculation. Overall, though, these results showed an excellent interrater reliability, and indicated that even nonexpert but senior students can be instructed relatively rapidly in the application of the grading standards.

Figure 3 compares the performance of four groups working on the solution of the same online case. Panel A shows the performance of a group that was ultimately successful in solving the problem. The consistently increasing contribution scores indicated a group that was working well and needed little intervention by the faculty members. Panel B shows an example of a group that was not progressing. There were few postings overall, and many of the contributions were unfocused and lacking in content. Close to the end of the time allotted to the case (at about Contribution Number 15), the faculty mentor, to push the group along, provided a bolus of data. The figure shows the then-rapid resolution of the problem. Panel C illustrates a group that was not progressing toward the solution, despite many student postings. However, in this case the faculty mentor did not intervene in a timely manner. Some of the students seemed to become frustrated and dropped out of the discussion, and thus out of the learning experience. Panel D shows an example of a group that appears to have not gone through the iterative process of solving the problem, but had heard about the correct answer from another, already finished, group. The dynamics in this type of situation (which we have seen several times), are interesting; often other group members become angry with the student revealing the correct answer, because they do not get the satisfaction of solving the puzzle on their own. In the first years of this study, Pattern D was also a fairly common pattern seen in groups containing students who were repeating the course because of a failure in the previous year. As mentioned above, we now use similar case presentations, but with different data and solutions each semester.

Figure 4 shows a nested analysis of performance of an individual student (red bars) within a group. We have found that this type of presentation is extremely helpful in identifying students who do not understand or respond to the ideas and solutions of the group; in such a case, the individual student contribution scores remain low in contrast to the increasing scores of the other participants.

However, as mentioned earlier, the student pattern profiled in Figure 4 initially suggested to us that he/she was in fact learning how to apply a problem-solving strategy by working with the group, but this turned out not to be the case. The post hoc domains analysis made it clear that the student was not participating in all phases of the problem-solving exercise. The student mostly summarized what other group members had posted, and in fact was seldom if ever the first to propose a hypothesis, to suggest a relevant test, or to evaluate or integrate data.

The domains analyses for several different individual students are shown in Figure 5. A pattern of responses such as that shown in Figure 5A suggested to us that a student was not comfortable with hypothesis generation, or able to evaluate experimental results, but preferred to investigate; i.e., to ask for multiple experiments to be done, whether they were relevant to the problem or not. We call this a “shotgun” approach to problem solving: by asking for every possible test or procedure to be run, a student might, with luck, receive information that will lead to the solution to the problem. Such a student appeared to be unable to (or unwilling to) design a logical experiment-based approach and did not know what to do with acquired data.

Patterns such as that shown in Figure 5B led us to diagnose a student as a “summarizer.” The ability to summarize, which is one aspect of the Reflect domain, is an integral part of a successful problem-solving strategy that is particularly helpful in a group, but not as useful a skill when it is the single tack taken in the solution of a problem. Often, when such a student's postings were read in their correct sequence along with the rest of the group's conversation, this evaluation became more definite; such a student's reflective postings almost always came after several other students' contributions of hypotheses, evaluation of data, and integration of several pieces of information. For example, a summarizer would show a pattern such as seen in Figure 4, red bars only; in the group context the postings can be seen to occur after other, high-scoring postings.

Figure 5C shows the domains analysis pattern of an individual student in a very inactive group. The student was forced to solve the puzzle essentially as an individual, because the rest of the students were not contributing. This student was very methodical, and successful, in applying a problem-solving strategy to the solution of the case: repetitively cycling through identifying a hypothesis; designing experiments to test the hypothesis; evaluating data from the experiment; and then checking the validity of the results by comparing them to the known relevant basic science. This pattern mimics the classical scientific method, and it has been a fairly rare individual student pattern in the 3 yr of the study; it only occurred when there were few contributions from other members of a group to detract from the single student's problem-solving strategy.

Finally, we have identified a few students who were actually working at an expert level (NRC, 2000), whose postings spanned several domains at once, making leaps of understanding without using defined steps (Figure 5D).

The generation of domains analysis profiles for individual students allows faculty members to focus their efforts where they are needed most, i.e., on students who are unable to learn a productive problem-solving strategy without direct and vigorous instructor intervention. Figure 6 depicts one such student, in six different cases, over two semesters. It appears that the student played the same role in each of the six group discussions: that of the summarizer. He/she seldom used other approaches to the puzzle. This student was also unable to work his/her way through similar problems on individual assessments.

It is important to note that during the initial implementation of the online cases, the faculty focus was on optimal development and presentation of the cases and on monitoring the biochemical content learned within the discussions. The methods described here for analyzing individual contributions were not in place from the beginning, and, because of faculty time commitments, earlier group discussion transcripts were simply archived for later analysis. Consequently, early intervention with students having difficulties in solving the group problems was not possible in the first 2 yr of the study.

However, as our skill and experience with the domains analysis technique improved, and with the emergence of some typical patterns, we found ourselves using these data to help students improve their problem-solving strategies. Moreover, sharing the data with the students has now become an excellent method to discuss the necessity of this skill for their future. Our understanding of both the breadth of different problem-solving strategies as well as intervention strategies is only in its infancy. Nevertheless, we feel that the tool described in this report has permitted a significant improvement in both our understanding and ability to assist students with their approaches to problem solving, as well as providing students with a new self-assessment tool.

We are currently in the middle of a long-term study on the various possible effective interventions for students with specific problem-solving difficulties. Although an extensive description of these strategies is beyond the scope of this article, some successful interventions are emerging. For example, the typical summarizing student depicted in Figure 5B often responds to iterative, individualized questions about their approach to a problem from the instructor. A student who has difficulty implementing appropriate investigative strategies often does so because he/she does not understand the context of the inquiry process. This type of student may need to be immersed in a laboratory or other real-world setting to fully comprehend the importance of this problem-solving domain. Regardless of what domain(s) require remediation, it should be noted that the process is usually unique to the student, and intervention may be faculty intensive. However, we find that simply providing the individualized data regarding performance on the pattern of use of the problem-solving domains is helpful in focusing students in their specific educational needs, and in guiding faculty on the effective implementation of their courses.

An important component of traditional PBL is peer evaluation, in which students are given the opportunity, and responsibility, to provide critical feedback to others in their learning groups. Although formal peer-assessment sessions were not a previously defined part of our online group process, we gathered anecdotal evidence from our observations as members of the online groups and from comments from many students representing all 10 online classes over 3 yr.

First, the roles that individuals adopted within the online virtual groups only sometimes reflected their usual class behavior. Often a leader within a group was the quintessential “smartest kid,” who had the superior scientific background and perhaps possessed natural problem-solving ability. But, we also saw exactly the opposite: the student who had achieved high grades in other courses by relying on superior memorization skills did not understand, or want to learn how to use, a different learning strategy. This opened up the leadership role for previously less-confident students. Some of our best problem-solving strategists in the online discussions were older or nontraditional students, who had experience in working through puzzles in other disciplines but had not realized that the same skills could be applied to biochemistry. In other cases, good but quiet students found a new voice when their assertions could be backed up with scientific evidence, and they were not interrupted in the process of (electronically) explaining their point.

Second, each group seemed to police both their own group's social behavior and the scientific credibility of the postings. There have only been a few instances in 3 yr of an impolite or even abusive student within a group. In one case the rest of the students requested a move to another group, leaving the abuser alone as a group of one. In another, the responses from the other members of the discussion were very critical of that one student's derogatory posting, and the student apologized online to the rest. In the case of extremely irrelevant, thoughtless, or unscientific postings (a far more common occurrence than rude student members), which ignored previous progress that the group had made on the problem or presented incorrect biochemical principles, the common response of the other group members was to ignore such contributions. Conversely, positive comments within the groups for a particularly insightful or thorough posting were very common; the groups often developed a team attitude, with encouraging and constructive conversation by far the norm. At least one online group per class became a face-to-face study group by the end of the semester.

Finally, there were some students, as mentioned earlier, who just did not trust their peers to be able to provide correct or useful information through the discussion groups. This factor also seems to interfere with learning in classical PBL face-to-face groups. We have seen, in most situations, that trust is built over time, but it is not universal.

The major benefit of the implementation of the online, small-group problem-solving experience has been twofold. First, it provided students with a forum to discuss and apply their biochemistry learning. It was not uncommon to see references to earlier lecture examples in the student posts, or for a student to jump forward in the assigned readings to understand a biochemical concept in the online discussions. However, the opportunity to analyze a student's problem-solving strategy is the major advantage of the online cases. All of the authors are experienced tutors in UNM's medical school PBL tutorials, and we agree that the data obtained in the online discussions reported here allow a far more precise and constructive method of student assessment than is possible in the face-to-face setting. The domains analysis is the rate-limiting factor for this type of assessment, and we do not do it for all students. Students who can successfully demonstrate any successful problem-solving strategy are usually left alone. Only when students repetitively fail in independent problem-solving tasks are their discussion contributions analyzed as described above.

No correlation between student performance in the online problem-solving sessions and course examinations has been established. In addition, over the 3 yr of the study, our department did not offer concomitant, comparable biochemistry classes without the online PBL cases; therefore, the question of whether the online group discussions improved overall biochemistry learning, as measured by conventional exams, cannot be answered. However, we have seen no significant differences in overall student exam grades on our standardized final exams in any of the courses that used the PBL online cases (data not reported). It is clear our students have not been harmed in their learning of biochemistry content.

We judge that this lack of correlation does not tell the whole story, however, because the standardized exams measure mostly content knowledge, rather than application of content. We believe that the assessment tool must reflect the learning experience. Most traditional exam questions, written in single-best-answer format, are limited in their ability to evaluate problem-solving skills; this type of evaluation focuses more on recall of content, and may, at best, only identify a student's ability to engage in simple problem recognition, rather than the ability to critically investigate a scientific problem. We are continuing longitudinal studies with our biochemistry majors to determine whether the consistent use of PBL case discussions throughout a six-course curriculum leads to measurable increases in productive use of problem-solving strategies. For these long-term studies, assessment tools that specifically evaluate the skills practiced in the online discussions have been developed and are being included in each course; the results will be the subject of a separate communication.

We have seen differences related to the gender, age, and cultural backgrounds of our students in terms of perceived benefit from the use of the PBL cases, and we are attempting to quantify this. The University of New Mexico Department of Biochemistry and Molecular Biology has a very high percentage of underrepresented minority students. Incorporation over the past 5 yr of a variety of constructivist teaching modalities (including the PBL cases) into our biochemistry curriculum has resulted in the establishment of what we consider to be a multicontextual learning environment (Ibarra, 2001). Concurrently we have seen an increase in the graduation rate for our underrepresented minority students, with no decrease in the average exit examination scores on the American Chemical Society's Biochemistry Certification Exam for the graduating biochemistry majors.

In conclusion, we believe the use of online PBL- cases is a way to get small-group discussions into our large biochemistry classes, to encourage students to use their basic biochemical knowledge in practical contextual situations, to develop the ability to integrate different pieces of their knowledge, and, most importantly, to practice their problem-solving skills. We have developed an illustrative tracking method to monitor the use of problem-solving strategies. This allows us the opportunity to help our students increase their proficiency in this critical scientific skill so necessary for successfully competing in the twenty-first century global marketplace.

REFERENCES