Teaching Assistant Attention and Responsiveness to Student Reasoning in Written Work

INTRODUCTION

In a 2009 Science editorial, Bruce Alberts (2009) lamented, “Rather than learning how to think scientifically, students are generally being told about science and asked to remember facts.” He wondered, “What would it take to get scientists to teach their college courses” in ways that “students at all levels engage in active inquiry and in depth discussion,” in the interest of students’ learning “to generate and evaluate scientific evidence and explanations, to understand the nature and development of scientific knowledge, and to participate productively in scientific practices and discourse” (p. 437).

Part of the answer to Alberts’s question is for science instructors to recognize and support the beginnings of scientific reasoning in their students (Robertson et al., 2016). That means looking past simple canonical correctness—Did students get the right answers?—to consider how students try to make sense of phenomena and ideas for themselves: Are students assessing information for meaning, plausibility, and coherence? Are students finding gaps and inconsistencies and posing their own questions? Instructors can help students learn to reason scientifically by identifying, highlighting, and helping them refine the beginnings of such thinking. But learning to notice and respond to students’ reasoning can be challenging.

Research in mathematics and science teaching has begun to consider how instructors can learn to value and focus attention on students’ reasoning to cultivate disciplinary practices. In mathematics, there has been a great deal of work on “teacher noticing,” both during classroom discussions (Sherin, 2001; Miller, 2011) and through careful reading of students’ written problem solving (Kazemi and Franke, 2004). These studies have found that teachers can notice but that they require support in orienting their attention to value student reasoning. Similarly, in science, there have been studies of “responsive teaching” (Levin et al., 2009; Robertson et al., 2016), of how instructors listen for, interpret, and respond to students’ reasoning. This research has demonstrated that expertise in listening and responding to student ideas is not simply related to more experienced teaching. Even novice teachers can identify productive disciplinary thinking in their students if they understand it to be a worthwhile activity.

So far, however, there has been little attention in science education to how instructors recognize, interpret, and respond to students’ reasoning in written work. While some research has studied the value of writing for learning science (Applebee, 1984; Scardamalia and Bereiter, 2006), the primary focus has been on how writing can develop students’ understanding of specific content (Keys et al., 1999; Hohenshell and Hand, 2006; Gunel et al., 2007). In giving priority to canonical correctness, this work has paid less attention to how students try to make sense for themselves.

Noticing students’ reasoning in written work can be challenging, because so many other aspects of their writing can draw a reader’s attention. Students’ language and grammatical proficiency, organization, and writing style can obscure their attempts to make well-reasoned scientific arguments. Moreover, lines of thinking that are well reasoned but canonically incorrect may receive less attention than those that communicate the expected correct ideas. While it may be desirable to help students develop better writing skills, it is important that instructors do not focus on skills to the exclusion of engaging with the substance of students’ scientific reasoning. To illustrate what we mean by attending to reasoning, we present a brief analysis of an introductory biology student’s lab report, pointing out the instances of scientific reasoning evident within it.

Evidence of a Student’s Reasoning in a Lab Report

In Box 1 we provide excerpts of a lab report written by an introductory biology student, “Micky” (a pseudonym). The lab was an experiment to measure the impact of various diets on the growth rate of tobacco hornworm (Manduca sexta) larvae. The lab manual presented to students the expected narrative around this experiment, such as explaining how plants “produce an array of secondary chemicals” and “herbivores have adapted-evolved mechanisms allowing them to consume these chemicals.” The manual also included questions directing students to consider why “plants rely mostly on chemical defenses” and “the advantage of having an inducible defense.”

The manual guided students to set up an experiment to measure the effect of chemicals on larval growth over 1 week, giving students choices among several possibilities. Micky’s group chose to compare tobacco hornworm growth with artificial diets containing the addition of either nicotine or salicin, with a nonsupplemented diet as a control. Nicotine is a chemical produced by the hornworm’s typical host plant, tobacco, while salicin is a phenolic compound produced by many other plants.

Micky’s report (Box 1) shows several potentially fruitful lines of scientific reasoning. First, Micky identified the relationship between the hornworm and tobacco plant as “a classic example of a plant–insect co-evolution relationship.” She correctly described how tobacco produces nicotine as a defensive chemical and how the hornworm has been able to evolve the ability to digest and use nicotine in its own defense against predators. Note that this first paragraph essentially paraphrases information from the laboratory manual and from a published article. For this reason, it is not convincing as evidence of Micky’s own reasoning. A teaching assistant (TA) might notice that Micky is correctly describing the study system, but there is not much of Micky’s scientific thinking to notice or respond to.

In the second paragraph, Micky introduced a new idea, that nicotine could also be a source of nitrogen for insect herbivores. Earlier in her report, Micky had written, “Nitrogen is one of the most important elements for life because it is a derivative of amino acids, which build together to make proteins.” Thus, Micky was arguing that, rather than simply tolerate nicotine, hornworms may be able to use it as a source of nitrogen. This idea is still in line with the main narrative of plant–insect coevolution, because it has hornworms reappropriating the secondary chemical for their benefit. However, it is not in the lab manual, and the hypothesis that hornworms would grow larger in the presence of a toxic chemical is unconventional.

That Micky proposed this idea is stronger evidence of Micky’s own reasoning. It presents both a challenge and an opportunity for the TA: the challenge to notice and consider the unconventional ideas and the opportunity to encourage Micky’s treatment of lab as a context for generating and evaluating original hypotheses and explanations, by engaging with Micky’s ideas. The TA could, most simply, acknowledge Micky’s having generated a novel, biologically plausible hypothesis. More, the TA could raise questions for her to consider: In what way does her prediction account for any costs associated with processing nicotine? Does she have a reason to believe that hornworm growth is limited by nitrogen and therefore that larvae that ingest more nicotine would grow larger? Does she account for the nitrogen content of the other diets in her experiment?

In the next section, Micky provided a possible explanation for why salicin might reduce larval growth. She described salicin as part of a group of chemicals with “more variety” and indicated that “this variety would cause a barrier to evolution,” but she did not explain how. It seems she was referring to an idea from the lab manual that identified salicin as part of a larger group of chemicals called phenolics, “the most common and diverse chemicals encountered by herbivores.”

A TA attending to Micky’s reasoning could ask for more elaboration. What exactly does she mean by a “barrier” to evolution here? Could she elaborate on the link between variety and benefit? (Perhaps she was thinking the variety of structures within the phenol group makes it difficult for herbivores to evolve the necessary pathways to digest each version.) What are the potential costs to producing chemicals of this type? Our point here is that, in order for Micky to improve her argument and develop it more fully, she would need to consider questions like this, and in order for her to consider these questions, instructors need to be looking for lines of argument that have potential but are in need of improvement—not just checking to see that students provided the expected story.

We present this brief analysis of Micky’s report to illustrate, first, evidence of student reasoning in lab reports, that is, the beginnings of their learning “to participate productively in scientific practices and discourse” (Alberts, 2009, p. 437). Second, we use it to illustrate the challenges in identifying these beginnings for instructors. For novice instructors especially, it can be difficult to recognize and interpret productive possibilities in students’ writing, because students’ reasoning can appear unconventional. Finally, we use it to highlight opportunities—to show how engaging with these beginning ideas could lead students to deepen their scientific thinking.

Studying Instructor Attention to Reasoning

There has been a great deal of work in K–12 teacher preparation focused on helping instructors develop abilities for eliciting, attending, and responding to student thinking. This work encourages instructors to learn to notice student thinking (van Es and Sherin, 2002; van Es, 2011); to elicit, recognize, and respond to student ideas (Hammer and van Zee, 2006; Levin et al., 2012); and to use artifacts to recognize student ideas in written work (Goldsmith and Seago, 2011). These studies have reported changes in instructors’ patterns of noticing and responding with professional support from facilitators and peer communities like video clubs. For example, van Es and Sherin (2002) found that, with the help of a facilitator, teachers learned to notice student reasoning present in video recordings of classroom conversations. Without the distraction of other classroom-related events, the teachers could identify moments when student thinking was present. Van Es (2011) proposed a framework based on one such video club in which teachers noticed student reasoning; participants’ noticing was sorted into four levels: baseline noticing, mixed noticing, focused noticing, and extended noticing. Inexperienced teachers tended toward baseline noticing, and teachers who noticed student thinking tended to prioritize classroom activities and focus largely on understanding students’ frames of mind.

Other work has examined the benefits to students of increased teacher noticing and responsiveness (Robertson et al., 2016). This research builds on a substantial body of work that has demonstrated that the form of instructor feedback impacts student learning (e.g., Black and Wiliam, 1998). For example, open-ended questions tend to stimulate sustained engagement and deeper thinking than closed-structured evaluative responses. Research on teacher noticing and responsiveness highlights the importance of feedback that is focused around nascent disciplinary ideas and practices. In science class, for example, feedback should be directed at those ideas that have the greatest potential to lead to scientific thinking (Russ et al., 2009; Coffey et al., 2011). Such a focus can support a classroom culture that promotes engagement with authentic scientific practices. According to Rosebery et al. (2010), responsive teaching can ensure that many heterogeneous ideas are available for consideration, making it more likely that more students will make meaningful connections to the scientific content.

There has been less work focused on graduate student TAs, and that work is mostly situated in physics (Goertzen et al., 2010a,b; Marshman et al., 2017). The findings of these studies suggest that TA beliefs about learning and teaching affect how they respond to students, either in classroom discussions or on paper. For example, Goertzen and colleagues (2010a,b) found that physics TAs mostly paid attention to superficial markers of canonical understanding—key words, phrases—rather than to the substance of students’ reasoning. They found that this behavior stemmed from a range of beliefs held by the TAs about whether their role in the classroom is more to support students struggling with concepts for themselves or more to guide them to canonical explanations. Marshman et al. (2017) showed that what TAs noticed and responded to in students’ written work differed by course. TAs were more likely to attend to conceptual understanding in an upper-division quantum mechanics class than in introductory physics, where they were more likely to attend to correctness.

In this article, we focus on how TAs engage with students’ lab reports in an introductory biology course. TAs are important to study in this context, because at many universities they provide much of the detailed feedback students receive on their lab reports. At the same time, they rarely receive pedagogical training that goes beyond the basics of university policies and basic classroom management (Luft et al., 2004; Schussler et al., 2015; Gormally et al., 2016). To make recommendations for TA training that addresses student reasoning, we first wanted to understand what biology TAs do without explicit training.

We have two purposes in this article. The first is to examine where graduate student instructors focus their attention in reading and commenting on students’ writing without explicit training. Do they notice and respond to students’ scientific reasoning? The answer to that question, we show below for the TAs in our study, is mostly that they do not. We suspect this is true broadly at our university and others.

However, we also show evidence that, like novice K–12 teachers, the TAs in our study have the ability to notice and respond to students’ scientific reasoning. This generated our second purpose, to understand what influences TAs to attend and respond to student reasoning in lab reports. In this, our purpose was to analyze the moments when TAs were attending and responding to students’ reasoning so that we might better understand what drew their attention in these moments. Progress in this respect could inform strategies for professional development and course design to promote TA attention and responsiveness to student reasoning.

METHODS

Study Setting

We conducted this study at a private liberal arts university in New England during the spring semester of 2014. Five TAs of a group of 12 for a large-enrollment (∼280 students) introductory biology course for biological sciences majors and nonmajors participated. We assigned each TA a pseudonym (Abby, Betty, Chris, Dana, and Ed). They were master’s (Abby, Chris, Dana) and doctoral (Betty, Ed) students in the Department of Biology (Abby, Betty, Chris, Ed) or Education (Dana). Abby and Chris had taught lab sections for the Department of Biology in a previous year. Abby had been a high school science teacher before entering the program. Betty, Dana, and Ed had no prior teaching experience.

All had taken a required pedagogy course offered by the Department of Biology the prior semester. That course met weekly and included readings and discussions to help TAs feel comfortable in the lab classroom setting, focusing on approaches to introducing topics and leading discussions, classroom management, lesson planning, and assessment of written work. The course did not, however, address attention to student reasoning. The session on assessment emphasized attention to the learning goals of the assignment. In the labs we describe below, assessment focused mainly on style, form, and the correctness of information.

Additionally, Abby had taken a course in the Department of Education on learning and teaching science. That course included a strong emphasis on the nature and importance of instructors’ attention to student thinking. It included a series of assignments to interview students and laypeople about their reasoning around questions about natural phenomena and activities surrounding analysis of students’ thinking as evident in classroom videos, all informed by readings from education research that offered a theoretical background on the nature of student reasoning in science.

The introductory biology course labs met weekly over the course of the semester, and students attended 10 labs. For this analysis, we focused on instructor feedback on lab reports written for a lab in which students investigated the impact of diet on the growth rates of hornworm larvae (Manduca sexta). This was the second of two lab reports assigned during the semester and was worth 25% of the lab grade (6.25% of the course grade). Students read background information about hornworms and plant secondary chemicals. For the lab, they chose two of five treatments: an artificial diet containing all necessary nutrients (the control diet), a high-cellulose diet, or the control diet with the addition of one of three secondary chemicals (nicotine, rutin, or salicin); the treatments were selected to mimic what M. sexta might encounter in nature. The TA in each section guided students in setting up controlled experiments. Students collaborated in groups of three or four to conduct the experiments, but they prepared their own lab reports.

The laboratory manual included tutorials on how to access and evaluate scientific information, develop and test hypotheses, design experiments, and analyze data. Students completed these tutorials with TA guidance during class time or as homework in the weeks leading up to the experiment. The lab manual also devoted a section to explaining the purpose and organization of information within each section of a lab report. For example, the introduction is meant to “establish context to your study and convince the reader of its value.” To do this, the recommendation is to “start with the big picture,” then to “focus on your study system,” and finally, to “state your hypothesis and predictions.” For the discussion section, the manual instructed students “to summarize and interpret your results in the context of your hypothesis and other studies.” The materials and methods and results sections focused on giving examples of how to write, “Do write [the section] in past tense,” and what kind of information to include, “Start with a general statement concerning the results” (emphases in original).

Students could elect to receive support for writing their lab reports in individual office hours with their TA. Office hours—for the few who attended—typically focused on data analysis and proper formatting of figures and tables. TAs read, commented on, and assigned grades to the students’ reports. The lab coordinator provided a rubric (Supplemental Material 1) to TAs as a guide for grading but gave no explicit directions on how to use it.

Data Collection

Our data included 1) student lab reports with TA comments and grades from each of the five TAs and 2) semistructured TA interviews about their thinking as they read over the marked reports. We collected all of the lab reports from each of the five TAs’ lab sections (a total of 125, or an average of 25 reports per TA). Because of the in-depth nature of the data analysis, we analyzed four lab reports per TA section, with the reports being selected by the TAs during the interviews. We asked TAs to include at least one high-grade (>90%), one low-grade (<75%), and one average report in the set of four. These made up the subset that underwent the full data analysis, although other lab reports for each participant were checked to ensure that markings included in this subset were typical of that individual, meaning that the number and type of comments made on other lab reports were similar to those found in the subset.

During the interviews (60–90 minutes each), each TA was asked to speak aloud about what he or she noticed while rereading each of the four reports. The first author (C.F.C.H.) conducted the interviews, framing the interview as an opportunity to gain a better understanding of TA attention with students’ written work. She was a fellow graduate student not associated with teaching the course, and so not in a position to evaluate the TAs. Even so, she avoided providing any facial or verbal cues that could have been perceived as approval or disapproval. Instead, she tried to encourage TAs to share what they wanted and followed what caught their attention as they read their students’ reports. The goal was to elicit as much of the TAs’ thought processes as possible. C.F.C.H. used the following questions to guide the interviews:

• Can you please talk me through this paper?

• What kinds of things did you notice about what the student wrote?

• Are there any things you noticed but did not comment on?

When necessary, C.F.C.H. interjected with questions to ask for clarification. For example, if the TA mentioned that something was important, C.F.C.H. followed up with a question to probe for more information, such as “Why do you think that was important?” Or, if the TA mentioned something—such as the grading rubric—several times during the interview, C.F.C.H. would follow up with “Did that affect how you looked at this paper?,” to gain a sense of where TAs focused their attention and why. We audio-recorded and transcribed interviews for analysis.

Data Analysis: Written Comments

We developed a coding scheme to analyze TA markings on lab reports. These markings, as illustrated in Figure 1, included symbols for punctuation, single words, and longer comments.

Our process for developing the coding scheme followed the constant comparative method (Charmaz, 2006). We first separated comments into a spreadsheet that included the TA marking and the surrounding context of what the student wrote. We then generated a set of categories from initial reads through the data set and two coders (C.F.C.H. and J.S.G.) applied those categories to code a subset of the data; this was followed by discussion and refinement of codes. In some instances, it was difficult to decide between two or more possible categories. We discussed these borderline cases and used them to further refine the coding scheme. In some cases, a comment fell into multiple categories, in which case the two coders discussed the comment, either coming to consensus on a single code or double-coding a response that fulfilled criteria for both categories.

We iterated this process: describing categories, two coders applying them independently to data, comparing results, and refining categories to minimize ambiguity. Table 1 summarizes our final coding scheme, including explanations of our judgments in clear and in borderline instances. After three cycles, independent coding for these categories yielded 87% agreement across 28% of the data set. A single coder (C.F.C.H.) then applied this scheme to the full data set.

Our purpose in coding TA markings was to quantify TA attention to style and form, correctness, or reasoning. Some markings, such as checkmarks, question marks, or underlined passages with no comment, and single words, such as “good,” we coded as ambiguous; these did not contribute to our analyses of TA attention.

There was a great deal of variation in markings for each category, and we coded markings in the context of what the student wrote (Table 1). We coded markings that addressed genre-specific elements of scientific writing as style or form. Style comments concerned, for example, conciseness, verb tense, formal language, parenthetical references to visual aids, appropriate acronym introduction and use, proper organization of sections, as well as requests to include or exclude specific details, such as background about the study organism. Comments on form addressed flow, grammar and punctuation, spelling, and clarity of writing. By the last, we mean a marking to note something technically unclear in the student’s writing, but without evidence of interest in interpreting the student’s meaning. By and large, we treated these two categories together, style and form, as concerning the formalities of scientific writing.

Comments on correctness assessed the report for alignment with canonical knowledge, including use of terminology and specific, factual information about the study system. Comments on reasoning, in contrast, focused on the students’ thinking as they were expressing it, such as to acknowledge or challenge students’ ideas, to encourage elaboration, or to note clarity of scientific ideas. By the last, we mean a marking that shows a specific attempt to interpret meaning.

Data Analysis: Interviews

The second goal of this study was to understand what influenced TAs to attend and respond to student reasoning. We conducted semistructured interviews to collect information about how TAs described their marking practices and motivations. In our analysis, we aligned the interview transcript with the excerpts of student reports and the marking that the TA was referencing. We then generated a descriptive account of how each TA’s attention shifted in real time in the context of the interview. This meant interpreting what TAs said in light of the lab report they were looking at, their own written comments, and, at times, the prompts and questions posed by the interviewer.

Two TAs, Abby and Ed, made the highest proportion of markings coded as attention to student reasoning. We chose to focus on them in follow-up qualitative case studies, because the data from their lab markings and interviews provided the most opportunities to see TA attention on student reasoning (Supplemental Material 2 and 3). Studying Abby and Ed in more depth could help us understand what may initiate, sustain, or interrupt TA attention to student reasoning.

We present here specific episodes from Abby’s and Ed’s interviews when their attention shifted to or from students’ reasoning, focusing on their markings and comments about the discussion sections of their students’ lab reports. For Abby, we saw the clearest evidence of a shift in attention as she reread Micky’s lab report. For Ed, we saw the clearest evidence in discussing his student Nora’s report.

As we noted earlier, Abby had prior experience teaching high school. She was in her second year in the program, and this was her second time as a TA for this lab course. Ed was a first-year graduate student who had no previous experience as a classroom instructor. Both had taken the pedagogy course required and offered by the Department of Biology; Abby had also taken a seminar on student reasoning in the Department of Education.

RESULTS

Results of Written Marking from All TAs

We describe here two findings based on coding TA marking data: 1) as a group, the TAs mainly commented on style and form; and 2) within the group, some individuals noticed and responded to student reasoning more than others.

There was a wide range in the total number of markings made by each TA (Table 2). Every TA commented mainly on style and form, from a low of 62% (Ed) to a high of 87% (Chris). Only 15% of all markings fell into the category of attention to reasoning. Some TAs commented on reasoning more than others: 7–9% of comments made by Betty, Chris, and Dana focused on reasoning, whereas the results for Abby and Ed were 24 and 26% of comments, respectively.¹

Shifting Attention to Student Reasoning in Abby’s and Ed’s Interviews

In our analysis of their interviews, we tracked how Abby’s and Ed’s attention shifted from moment to moment. We focused on the moments when they were attending to student reasoning as a potential place to understand what influenced their attention. When Abby and Ed attended to student reasoning in their interviews, what influenced their attention differed. Abby’s interview showed her attention shift from style and form to reasoning in response to an interviewer’s question. Ed’s attention to style and form and reasoning was more closely tied to the section of the lab report he was reading. In the remainder of this section, we use an in-depth analysis of Abby’s and Ed’s responses to a single student report from each of their lab sections to illustrate some of the main themes from the interview as a whole.

We first present Abby’s reflections on “Micky’s” lab report, which demonstrates Abby’s shifting attention. When Abby began discussing Micky’s report, she focused primarily on style and form. It was not until after the interviewer inadvertently raised a question about a student idea that Abby shifted her attention to reasoning. From that point forward in the interview, she continued to discuss the substance of the student’s ideas when opportunities arose. We then present Ed’s discussion of “Nora’s” lab report, which illustrates his desire for lab reports to look like “something that you could practically possibly submit as a scientific publication” (Supplemental Material 3, lines 267–277). As in a scientific article, Ed expected to see evidence of student reasoning in the introduction and discussion sections, and so he discussed reasoning as well as style and form as he reviewed these two sections. He expected the methods and results to conform to style and formatting standards but to include less student reasoning.

Abby’s Shifting Attention as She Discussed Micky’s Lab Report

Micky’s was the third lab report Abby discussed during her interview (see Supplemental Material 2). Initially, Abby was preoccupied by the correctness of Micky’s argument, stating that Micky’s claim—that plants have a “large” (Supplemental Material 2, line 614) amount of nitrogen—was incorrect. She seemed concerned about Micky’s confusion, because the experiment had “nothing to do with nitrogen” (Supplemental Material 2, line 619). As described in our analysis of Box 1, Micky’s idea was that some insects have found a way to extract nitrogen from the defense chemicals of plants to use as a dietary source of nitrogen. Abby missed this idea—both in her initial reading and in reflecting on the report in the interview. Furthermore, there is no evidence of her attempting to understand Micky’s reasoning at this moment. Instead, Abby initially decided there was no reasoning to follow.

As Abby continued to read, she attended mainly to structural conventions of lab reports and the lack of clarity in the writing. She commented that the introduction lacked an “internal logic or flow” (Supplemental Material 2, line 627) and noted Micky’s habit of listing facts: “They give you this laundry list of facts that have nothing to do with their experiment. So she did some of that” (Supplemental Material 2, lines 655–656). Abby continued her critique, pointing out “writing problems” (Supplemental Material 2, line 663) where Micky was “not using language in a precise way” (Supplemental Material 2, line 667).

About 10 minutes into her discussion of Micky’s lab report, Abby expressed confusion about Micky’s phrasing “in order to demonstrate the use of secondary chemicals.” The interviewer then asked Abby a follow-up question:

• Interviewer: What do you think she’s trying to say?

• Abby: I think she’s trying to say that the experiment was designed in order to observe the effect of different secondary chemicals on the caterpillars. And the second half of this sentence makes that a little more clear. (Supplemental Material 2, lines 669–675)

The full sentence in Micky’s report was “An experiment was designed and performed in order to demonstrate the use of secondary chemicals, specifically nicotine (inducible) and salicin, and observed the resulting effects on the growth of tobacco hornworm Manduca sexta.” The interviewer’s prompt evidently cued Abby to attempt to interpret Micky’s meaning. Abby decided Micky meant the lab was designed for students “to observe the effect of different secondary chemicals on the caterpillars.” From this point in the interview onward, Abby put more of her attention toward understanding Micky’s ideas as she continued to read through the report.

Previously, Abby had dismissed Micky’s idea about nicotine being a valuable source of nitrogen for hornworm growth. When she encountered this idea again, Abby began to refer to it as “the part where it gets crazy”; however, as she read “due to the high nitrogen content” aloud, she stopped herself, apparently now realizing why nitrogen featured so heavily in Micky’s introduction:

Abby maintained this shifted attention in the next part of her interview. She read another sentence aloud and then attempted, unprompted, to understand it. Micky wrote that it could be difficult for insects to evolve a tolerance of salicin, because it comes from a group of biochemicals with “more variety.” Rereading this, Abby tried to articulate what Micky could have meant,

Here, Abby proposed that Micky could be making an argument that the variety in the structure of this group of chemicals would make it unlikely that insects could evolve a way to metabolize all of them. With this interpretation, Abby now saw Micky’s hypothesis that “less [hornworm] growth will occur on the salicin [diet], when compared with that of the natural diet, nicotine” as a “reasonable hypothesis” (Supplemental Material 2, lines 707–708).

Abby then returned to Micky’s earlier idea about nicotine as a source of nitrogen, engaging with the idea, although not agreeing with it:

Abby articulated her reason for disagreeing with Micky at this point: Micky was “confusing the ability to, like, be okay with a chemical, as actually liking it and choosing nicotine or preferring a nicotine diet over just a control diet” (Supplemental Material 2, lines 718–719). Abby noted a lack of evidence for nicotine as beneficial and pointed out that Micky was not able to actually test her hypothesis, because she did not include a control diet in her experimental design.

During the interview, Abby noticed a comment she had written at this point in the report: “Are you saying that you think caterpillars fed nicotine will do better than the control caterpillars?” At the time of marking, Abby did not understand why Micky placed such importance on the nitrogen content of nicotine, but Abby described making this comment as an attempt to “get [Micky] thinking about that” (Supplemental Material 2, line 734). Instead of writing an evaluative comment, “something like, ‘obviously, the caterpillar fed nicotine will not do better than the control caterpillars’” (Supplemental Material 2, lines 739–740), Abby thought that “by virtue of asking that question she’ll think, like, ‘hmmm… there’s a reason she’s asking this question. Maybe I should rethink this’” (Supplemental Material 2, lines 737–738). She referred to this move as being a purposeful attempt to point out what she considered as flaws in Micky’s thought process, without addressing them directly (Supplemental Material 2, lines 860–863).

To us, the question Abby wrote in the margins is evidence of her noticing something “crazy” about Micky’s reasoning and responding to it. During the interview, when she was working harder to interpret Micky’s reasoning, Abby was able to see the sense in it.

Ed’s Attention in Interpreting Nora’s Report

In his interview, Ed talked through Nora’s report section by section, switching focus as he moved through each one (see Supplemental Material 3). English is Nora’s third language, and Ed described taking extra time to interpret her writing compared with that of native English speakers, which suggests another kind of cue for attention to student meaning.

In Nora’s introduction, she mainly summarized the M. sexta study system, providing details about its life cycle and posing some initial ideas about the relationship between diet, size, and fitness. While reflecting on this section, Ed addressed the “idiosyncrasy” of Nora’s written expression: how “it’s not a particularly neat phrasing,” but that he understood the substance of what she attempted to convey, because “it gets all the information across” (Supplemental Material 3, lines 484–486).

In the methods and results sections, Ed attended mainly to style and form. He attended to the clarity of her description of the experimental protocol (Supplemental Material 3, line 563): “she did not say how much cellulose—that one was high and one was not” (Supplemental Material 3, line 566). Ed pointed out the missing information because “others […] need to know the concentration you used if they’re going to test or replicate your results” (Supplemental Material 3, lines 568–569). Ed’s focus in marking was related to his sense of the purpose of these sections, to communicate methods for conducting an experiment or to present experimental findings that refer to the author’s proposed hypotheses and predictions.

As Ed read through Nora’s discussion section, he began to attend more to her reasoning. In reading her discussion, one of the first things he noticed was her novel idea about changing cellulose levels in plants as related to plant defense. He read aloud, “Further experiments can focus on different situations that might induce the increase or decrease of cellulose in the plant,” and remarked,

Even though Ed did not think Nora was correct, he was willing to consider and follow her idea:

Ed’s reflections in the interview are consistent with his written comments on the report. In the margins, Ed had written, “Really? Do plants produce more cellulose when attacked?,” in response to Nora’s ideas about inducible cellulose. In response to her research question about how plants regulate their defenses, he had written, “Interesting ideas—give some more detail!” Ed considered her idea for a follow-up experiment in which she proposed to study plant-regulated cellulose levels to be a genuine direction for future studies consistent with her prior reasoning about the role of cellulose in plants. Even though he did not think Nora was factually correct, Ed chose to focus on encouraging Nora’s line of reasoning rather than on correcting her.

In sum, both Abby and Ed were drawn to noticing, interpreting, and responding to students’ ideas more strongly in certain moments than others. Some features of those moments included the curiousness of a student’s idea, the TA’s knowledge about the student’s fluency in English, the particular section of the lab report, and cues in the context such as the interviewer’s questions. Abby and Ed noticed ideas that were in some way out of the ordinary or confusing, breaking with their expectations of what a typical student report looks like.

DISCUSSION

Science and mathematics education researchers have studied responsive teaching—the instructional practices of attending and responding to what and how students are thinking—in classroom interactions. With this study, we have begun to explore responsive teaching in written work, specifically in how graduate student TAs read and comment on undergraduate students’ lab reports. We are motivated both to improve students’ learning to practice science in undergraduate laboratories and to help graduate students develop expertise in teaching biology.

Our first, simplest result was that, overall, the graduate student TAs paid little attention to the substance of students’ reasoning in reading and commenting on lab reports. This was variable according to TA and motivated us to try to understand why. It is difficult to believe that TAs’ limited attention to student reasoning is primarily a matter of ability. The ability to attend to reasoning is continuous with the abilities they must use as learners themselves: reading and interpreting scientific arguments and engaging in discussions with peers. Also, we saw evidence of TAs attending to student reasoning in particular moments, and it became our second research aim to understand what influences TAs’ attention as they read lab reports.

In this section, we discuss what this study contributes to understanding possible influences on TAs’ attention to student reasoning. First, we give evidence and argue that TA attention is sensitive to context. Contextual features can vary in scale from the large-scale structure of the course as a whole, to guidelines for assessment, to moment-to-moment cues that arise as TAs are reading and interpreting students’ written work. These contextual features can impact how TAs frame and enact their instructional activities (Goertzen et al., 2010b).

We first describe contextual features that may support TA attention to reasoning. We then argue that various features of the course context may have the unintended effect of distracting TAs from attending to student reasoning. Finally, we turn to implications for lab design and TA professional development.

Our findings overall show that TA attention to reasoning is sensitive to context. There was some evidence of attention to student reasoning for every TA in our analyses of their written comments on lab reports. To understand when and how it occurs, we studied particular moments more closely. We chose Abby and Ed, because they presented us with the most opportunities to study moments of attention to student thinking. Looking across their cases, we suggest, first, that TA attention can be attracted by novel ideas within students’ writing and, second, that it can be influenced by features of the context of their reading.

Unexpected Ideas May Cue Attention to Student Reasoning.

A common feature of both Abby’s and Ed’s moments of attention to reasoning is that they were responding to an incorrect or unexpected idea. Abby first noticed Micky’s idea about nitrogen as part of the lab report because she found it confusing. This idea had captured Abby’s attention as she was marking the report, and she described it as “the part where it gets crazy.” Abby’s noticing led her to write a question on Micky’s report that was not explicitly evaluative, but rather simply asked, “Are you saying…?”

For Ed, too, the idea that caught his attention was one he had not expected: Nora was suggesting that, in response to herbivory, plants could increase the proportion of cellulose in their tissues. Like Abby, he thought Nora’s reasoning was incorrect; unlike Abby, Ed continued on his own to understand her reasoning.

Instructors may be most likely to detect student reasoning when they encounter unexpected or incorrect student responses (Cowie and Bell, 1999). Detecting student reasoning is only the first step, of course; once TAs detect incorrect or unconventional student reasoning, they must work to understand and then respond in a way that will provoke thought from the student, rather than just point out that the idea is incorrect or requires further explanation.

Responding to what is sensible in unconventional ideas may be particularly important, because the instances when students deviate from the expected narrative of the laboratory are more likely to represent students’ own thinking. Micky’s idea about secondary chemicals as a nitrogen source and the beginnings of her thinking about the effects of a variation in secondary chemicals are the best candidates for ideas that came from Micky. Similarly, Nora’s idea about inducible cellulose provided some insight into how Nora was making sense of the system. These ideas reflect thinking, not just blind parroting of what was described in the laboratory manual, and they are sensible within the context of a system of coevolving plants and insects.

These ideas also present opportunities for helpful feedback. Here, for example, after recognizing the students’ reasoning, the TA could prompt the student to compare the costs and benefits or to consider the possible mechanisms of the organismal strategies they are proposing. Micky proposed that nitrogen is a nutrient source; the TA could prompt her to consider the cost of metabolizing nicotine. Nora proposed inducible cellulose as a strategy for deterring insects; the TA could prompt her to consider by what mechanism cellulose content could be increased. By attending and responding to ideas like these, TAs could encourage students both to continue to have and pursue their own ideas and to refine, improve, and defend them.

Features of the Context May Cue Attention to Student Thinking.

Abby had noticed Micky’s “crazy” thought, but until the interview, she did not understand it at a level that would have let her provide helpful feedback. It was not until the interviewer asked, “What do you think she’s trying to say?,” that Abby worked to understand Micky’s reasoning. The interviewer did not help with the interpretation; she only provided the cue that it was something to do, which prompted Abby’s taking the time to decode Micky’s words for meaning. We suggest the shift was in her framing of what was taking place. During the initial reading, Abby framed her activity primarily as assessing a lab report, and her attention was guided by this framing. During the interview, she shifted to interpreting the substance of a student’s reasoning. Both were available to her as ways of framing her activity. She had practiced the latter in a science education seminar she had taken the year before.

Ed shifted his attention to focus more on student reasoning in the introduction and discussion sections of the lab reports, reflecting his expectations of the purposes of the writing in these sections. In other words, the organizational standards of scientific papers seem to have helped organize Ed’s attention. Because he did not expect students to share their reasoning in the methods section, he did not pay much attention to it, unless the reasoning did not follow a consistent, logical flow from the introduction through subsequent sections to the discussion.

There was also evidence in Ed’s comments during the interview that he spent more time working to understand Nora’s writing, because English was not her first language. The need to put time and effort into interpreting Nora’s writing also functioned to keep Ed’s attention on her reasoning.

Features of Context May Distract TA Attention from Students’ Reasoning.

Features of the overall context of the laboratory course and grading structure may help explain our first finding: in general, the TAs paid little attention to the substance of students’ reasoning. One prominent feature of that context was the rubric provided to TAs as a guide for awarding points in reading reports (Supplemental Material 1). If TAs simply divided their attention in a way that mirrored the rubric, we would not expect to see much attention to reasoning. Of 100 total points, only 20, those awarded for an initial statement of hypothesis/predictions and those related to the discussion section, could be reasonably interpreted as places to attend to students’ scientific reasoning. The remaining 80 points concerned the inclusion or exclusion of specific elements (e.g., dependent and independent variables in the title, description of the study system in the introduction, enough detail in the methods), writing style (clarity, use of past tense), or formatting (figures referenced parenthetically, citations properly formatted). With so many details about students’ writing to attend to, it is not surprising that the TAs in our study did not spend the time and effort needed to delve into understanding students’ arguments.

We have evidence from interview data that we have not analyzed here that TAs varied in how closely they followed the rubric in their marking process. For example, Abby felt “obligated” to follow the rubric, because it was a course-provided resource, yet felt it had a “distortional effect on grading” and compelled her to comment on things she might have otherwise left alone. Betty and Chris both felt that the rubric largely aligned with their sense of good scientific writing, but neither felt obligated to adhere to it strictly, and both reported excluding rubric items that they had not discussed in class. Dana and Ed both relied on the rubric to help award points, which Dana articulated in her interview as a need for grading to be “consistent and fair.” Ed shared a similar sentiment concerning fairness, but described spending more time on student thinking than was called for by the rubric.

Taken together, these comments suggest that, even if TAs did not use the rubric strictly to guide line-by-line commenting, the rubric may have helped shape their framing of what they were doing in reading reports. That is, on the whole, grading lab reports is about attending to many details of writing style and formatting. The instructions in the lab manual similarly framed the purpose of the lab report broadly as an exercise in making a lab report fit the structure of scientific writing. If we want TAs to attend to students’ scientific reasoning, then the overall structure of the lab, including explicit messages from supervisors and the lab manual and implicit messages communicated through the assessment structure, needs to be redesigned to place more emphasis on developing students’ scientific reasoning.

IMPLICATIONS FOR LABORATORY INSTRUCTION AND TA PROFESSIONAL DEVELOPMENT

The TAs in this study worked in the context of a lab course designed, primarily, to guide students in recreating a predetermined narrative. The grading structure emphasized the style, formatting, and correctness in students’ writing of that narrative. TAs may notice students’ original reasoning in this context, but it would be difficult to give these ideas much attention, as they are digressions from the core purpose.

Designing Laboratory Instruction and Assessment to Focus on Students’ Scientific Reasoning

To promote responsive teaching in laboratory instruction, we suggest, would require a course design that frames student reasoning—including their unexpected, original ideas—as a fundamental purpose for experiments and activities. Such reframing must attend both to what students do in lab and what is valued in assessments.

Writing lab reports can be an opportunity for students to develop their ideas into well-reasoned scientific arguments. However, lab reports are often assessed using grading checklists or rubrics that include many different aspects of scientific writing (Allen and Tanner, 2006). Rubrics that give weight to stylistic and genre-specific conventions of writing in science will focus student and TA attention on those conventions. Even when rubrics include both stylistic and substantive items, students and instructors may gravitate toward style and format (e.g., use of correct terminology, figure formatting), because those aspects are more straightforward to apply and identify (Tang et al., 2015). Both student and TA attention is limited. Assessment structures must therefore prioritize students’ scientific reasoning over more superficial concerns.

In our subsequent work, we have been reframing the structure of the introductory biology laboratory to emphasize student reasoning in activities and written assignments. Preliminary analyses indicate that this shift has increased the level of scientific reasoning in students’ writing as well as TA attention to that reasoning (C. F. C. Hill, unpublished data).

Recommendations for TA Professional Development Focused on Attending to Student Reasoning

Our study also suggests that TAs are likely to need practice and preparation to notice and engage with students’ reasoning. Studies in K–12 education have shown that identifying the productive beginnings of students’ disciplinary thinking is a skill that can be developed with guidance and practice (van Es and Sherin, 2002; Kazemi and Franke, 2004; Goldsmith and Seago, 2011). Many of the successful professional development programs developed at the K–12 level are organized around the collective analysis of artifacts—videos of students talking or written work. Part of what teachers need guidance and practice with is identifying the beginnings of students’ disciplinary thinking—­what it is in students’ work that looks like the beginnings of science, math, or history.

Graduate students may have more disciplinary expertise but may need guidance applying these skills to students’ work. In our study, we saw Ed successfully applying his skill at evaluating scientific arguments to his students’ writing. One role of professional development therefore could be to build on TAs’ skills as developing scientists, to recognize the science in what students write, and in particular, to notice the scientific merit in ideas that may initially appear unconventional. We saw that Abby, perhaps building on prior practice, was able to engage with Micky’s unconventional ideas when prompted by the interviewer. With additional practice, TAs could learn how to apply the scientific criteria that are likely already familiar to them—plausibility, coherence, explanatory power, evidentiary support—to student writing.

An additional challenge, as studies in physics education research have found, is that TAs may not expect their role as instructors to involve analyzing students’ scientific ideas. They may instead expect to steer students toward canonical interpretations and strongly value this correctness as an indicator of good teaching and learning (Goertzen et al., 2010a,b). In addition to providing opportunities to practice, professional development should also create opportunities for TAs to share and reflect on their values and expectations about teaching and learning.

Finally, as we and others have argued, TA attention is sensitive to context. Thus, responsiveness to student thinking is most likely when TA preparation aligns with the expectations and values communicated to TAs in the courses they are teaching. Despite Abby’s previous experience in the classroom and in a course emphasizing the importance of attention to student thinking, she seemed to focus her markings on aligning with expectations set by the lab course. For Abby, and perhaps for other TAs, teaching in a course that explicitly and primarily valued student thinking could shift her attention and effort.

LIMITATIONS AND FUTURE RESEARCH DIRECTIONS

In this study, we looked at marking patterns in four lab reports for each of five TAs—a very small sample size. One limitation of this sample size is that we cannot generalize patterns from this study to all TAs at all universities. Nevertheless, we have reasons to suspect that a similar result would be found if this study were replicated at other universities where TAs receive minimal professional development. One reason is that the dominant tradition of emphasizing the style and form of scientific writing in student lab reports is pervasive, and TAs are likely to encounter expectations for grading lab reports that look similar to the ones we have described. Moreover, TAs are likely to have experienced writing structured lab reports as students and, like many novice teachers, use this past experience to inform their teaching and assessment practices.

A second limitation of the small sample is that we do not have the power to explain variation in our data. We saw that some TAs attended to reasoning more than others, but we do not know why. We might expect that prior teaching experience could influence TAs’ attention to student reasoning, but we would expect that the nature of the experience would matter a lot. Abby had prior experience teaching high school for a few years, but we do not know enough about the specifics of that experience to know whether it would have included practice attending to disciplinary beginnings in students work. Chris also had prior experience teaching as a biology TA in the same introductory course, which we would not expect to account for developing expertise in noticing or responding to student thinking. Though it is possible that experiences in upper-division courses in which disciplinary thinking may be valued more highly could provide such practice. Finally, Ed, a first-year graduate student, had no prior teaching experience, suggesting that prior experience is not necessary for developing the skills of attending to student ideas.

Another possible explanation may have to do with difference in TAs’ values and expectations about teaching and learning in science, which we did not study directly. Future research on biology TAs is needed to better understand their underlying values and motivations for teaching, including how those values and motivations may shift with different teaching contexts.

CONCLUSION

Lab reports are often framed, for students and TAs, as an exercise in practicing formal scientific writing. Our results show that, without explicit training, TAs may focus their attention during marking on the stylistic elements of writing rather than on students’ scientific reasoning. Because written work is a promising space for students to develop and refine their scientific reasoning, TAs must to learn to recognize and engage with the beginnings of students’ scientific reasoning. Our results suggest that some TAs can detect the disciplinary substance in students’ writing—as we saw, Abby and Ed noticed Micky’s and Nora’s ideas. While more research on how TAs can learn to notice and respond to students’ reasoning is needed, our results suggest that an important first step could be to shift the purpose and assessment of laboratory to prioritize students’ scientific ideas. Professional development for biology TAs should provide them with collective practice identifying, interpreting, and responding to student ideas, both oral and written.

FOOTNOTES

ACKNOWLEDGMENTS

References