Response to “Interpret with Caution: COPUS Instructional Styles May Not Differ in Terms of Practices That Support Student Learning,” by Melody McConnell, Jeffrey Boyer, Lisa M. Montplaisir, Jessie B. Arneson, Rachel L. S. Harding, Brian Farlow, and Erika G. Offerdahl
To the Editor:
The article by McConnell et al. (2021) brings attention to an issue that I, as a leader in the development of the Classroom Observation Protocol for Undergraduate STEM (COPUS), have been increasingly concerned about. That issue is the ascribing more meaning to COPUS results than is appropriate or correct. There is an increasing tendency to see COPUS results as a general measure of teaching quality and as a tool to be used in teaching evaluations. McConnell et al. provide data showing that there are some specific important differences in teaching that COPUS fails to distinguish, but here I want to emphasize that this conclusion applies more generally. As discussed in the “fine print” of the original COPUS article but now frequently overlooked, the COPUS was created from the beginning to be an instrument with quite limited capabilities in order to meet some very pragmatic constraints. It is important that users of the COPUS recognize these limitations.
The limitations of the COPUS can be better understood by looking at its history. The COPUS came out of a larger effort to characterize the teaching of science courses to measure the impact of the Science Education Initiative (SEI) at the University of British Columbia and University of Colorado in bringing about the adoption of research-based teaching practices. We developed the Teaching Practices Inventory (TPI) as an instrument that could provide a practical but sufficiently detailed characterization of the teaching practices used in a course. This covered all aspects of the course, including learning objectives, tests and grading policies, homework, and so on. After a lot of work and testing, we managed to refine this TPI instrument so that the great majority of the items on it involved objective responses and allowed the instructor’s responses to be independently checked, if desired. The exceptions were the subset of items (∼10%) that applied to activities in the classroom, specifically what fraction of the class time was devoted to what types of activities. I worried that instructors might be quite poor at making such estimates, and so we needed some way to check those estimates. I had the SEI education postdocs do a series of observations of classes to explore what information we could reliably obtain from classroom observations. This experimentation involved using a variety of published observation protocols in a variety of courses and looking at interobserver consistency. The goal was to determine what could (and could not) be determined by observers with modest amounts of training, recognizing that if this were to be carried out on a regular basis by departments, it would only be practical if only about an hour or less of training was needed for the observers to get consistent results.
Through this process we determined that no existing protocol met these needs. In some cases, protocols did not adequately capture the information that was needed, while in many others they required a level of nuance and judgment that could only be achieved after many hours or days of training. As we developed our own protocol that would meet the constraint of good interrater reliability with modest training, we found it necessary to make it simpler and simpler, essentially making it a blunter and blunter instrument.
The final result, COPUS, was an observation protocol that achieved our primary goal of allowing quickly trained observers (including nonexperts in the discipline) to generally characterize how the students and the instructor were spending their time in class, primarily as a check on the TPI information. However, this was done with the explicit recognition that this goal was achieved at a cost of not capturing many issues that are important in determining the quality of instruction, including those discussed in McConnell et al. (2021). I believe it is likely that many of these missing elements can be captured by the TPI information, but that is a different issue, as the COPUS has come to be used so widely on its own. As a side note, somewhat to my surprise, we found that instructors’ TPI time estimates were reasonably consistent with COPUS data from their classes.
Echoing the caution expressed by McConnell et al. (2021), I note here that the COPUS should be recognized for what it is and is not. It is a blunt instrument that is easy to use reliably and captures some aspects of classroom activities that reflect teaching quality, but it misses many others.
