Concept Map Assessment Reveals Short-Term Community-Engaged Fieldwork Enhances Sustainability Knowledge

INTRODUCTION

Global environmental challenges are not only understandably large, but complex. To seek solutions, we need to develop future scientists with both an awareness and appreciation of these problems that goes beyond basic scientific literacy to an understanding of the interconnected ideas of sustainability (Spruijt et al., 2014). To this end, several national calls, such as Vision and Change in Undergraduate Biology (American Association for the Advancement of Science, 2011), have emphasized the importance of developing core competencies and student-centered pedagogies to prepare students for engaging in the real practices of science. Over the last decade, professional societies such as the Ecological Society of America (ESA) have built on this idea to directly advocate for teaching in ways that promote core (in this case, ecological) concepts, practices, and human elements through integrative, thematic approaches (ESA, 2018). In short, professional scientists are calling for the authentic, but indeed complex, scientific training of the next generation (Crawford, 2015). But, while sustainability education may be a way to prepare undergraduate students for the future (Sjöström and Eilks, 2018), we still need to improve our pedagogical approaches to sustainability education and assessment (Gough, 2018; Mintz and Tal, 2018; Olsson, 2018).

Often, the best way for students to gain skills is through hands-on application. This is the simple theoretical basis behind experiential learning, a pedagogical approach wherein students are at the center of real-world experiences developed to bolster their subject matter understanding and professional practice. This approach is established (Dewey, 1938) and organized into four key elements: 1) a concrete experience, 2) reflective observation, 3) abstract conceptualization, and 4) active experimentation (Kolb, 1984). The concrete experience in the typical ecology laboratory, for example, would be fieldwork to learn a set of tasks at a particular geographic location, which would be supported by assignments that would promote reflection and deepen conceptual learning.

Service Learning as a Pedagogical Strategy

While the typical experiential learning experience, as described, may support content learning, it can often be disconnected from the scientific needs of the community, giving students a false sense that their work is also separate from the needs of society (Cooper et al., 2019). To remedy this, instructors have begun promoting service learning, whereby students become civically engaged with the community, as a preferred means of experiential learning (Ehrlich, 1996; Bringle et al., 1999). The National and Community Service Act of 1990 (2010, p. 13) defines service learning as “a method under which students learn and develop through thoughtfully-organized service that is conducted in and meets the needs of a community and is coordinated with an institution of higher education, and with the community.” The idea is that this engagement both enhances the academic curriculum and fosters civic responsibility.

By its very nature, community-engaged service learning is particularly suited to create sustainability-minded citizen-biologists (Astin et al., 2000; Felton and Clayton, 2011; Curtis, 2018). This pedagogical approach narrows the experiences to those which serve the community’s needs via integration of academic work with work done off-campus, often through a nonprofit or government agency, focused on a civic issue. This work expands the traditional classroom experience by allowing students to apply their theoretical knowledge in practical ways to address concerns of a community and has been demonstrated to support discipline-specific academic enhancement, personal growth, and civic engagement in students (Ash et al., 2005; Clayton et al., 2013). While students experience increases in worth, fulfillment, accomplishment, and the satisfaction of contributing to society’s needs (Ferguson, 2015), they directly contribute to improved community social, economic, or environmental conditions (Riel, 2019).

Community-Engaged Fieldwork in Ecology Course Work

By its nature, an ecological laboratory is an ideal place to integrate scientific education with the scientific needs of the community (Simmons, 2000; Leege and Cawthorn, 2008). As ESA’s 4DEE Framework emphasizes (ESA, 2018), ecology fundamentally describes relationships, including human–environment interactions, which can be acutely demonstrated via experiential learning (Brew and Jewell, 2012; Fuller et al., 2014). Ecological research informs citizens about the health of the environments in which they live, which has direct consequences on their daily lives. Many of the established, basic environmental monitoring methods conducted by ecological professionals are approachable to and solvable by large groups of nonexperts (e.g., students, citizen scientists; Cohen, 1997; Dickson et al., 2010), as much of the work relies on extended hours of relatively simple human observations, as compared with the use of complex laboratory equipment. In particular, as governmental and community monitoring efforts can be resource limited, a large group of observers can accomplish work that would necessarily be eliminated or severely reduced due to a lack of resources; thus, students provide the human capital to keep environmental monitoring projects running—and relevant.

It is clear that a classroom of students can be quickly trained to meaningfully provide data to community members, but it is fundamentally important to assess the learning gains of students. The benefit of service learning is assessed generally through traditional methods (i.e., reflection and assessment tests), but the extent to which student theoretical conceptions within their discipline begin to integrate community knowledge as a result of these experiences is unclear (Astin et al., 2000; Ash et al., 2005; Woodley et al., 2019). Because we recognize that community engagement is a complex, multifaceted process of interacting parts, some of the learning gains students achieve can be missed when the methods lack the ability to capture the interactivity of the experience. For this reason, the application of concept mapping may be used both to facilitate learning and to assess gains made; this allows for students to visualize the connections built through the experiential process in a way that respects their connectivity (Sandee and Hicks, 2005; Youssef and Mansour, 2012).

Concept Maps as Assessment Tools

At their most basic level, concept maps consist of several concepts related to a topic, connected via links in a hierarchical or web-based form. In many cases, the links themselves are labeled to describe in words the relationship they represent (Novak and Gowin, 1984). Concept maps allow students to visually organize knowledge (Weinstein and Mayer, 1986; Novak and Cañas, 2008; Emmanuel, 2013) so that they can structure and integrate new concepts. As assessment tools, concept maps are advantageous, because, in addition to assessing the quality of the student’s understanding (through the number and relationships between concepts and examples), they make the structure of that understanding transparent through the way in which the concept map is organized. Structural features of the maps are thought to reflect knowledge characteristics such as depth, breadth, and connectedness (Novak and Gowin, 1984; Turns et al., 2000). Previous researchers have devised concrete methods and software for scoring the structural components of concept maps in an automated manner (Cañas et al., 2013), which allows for the rapid assessment of large numbers of concept maps with little concerns of interscorer reliability (Reiska et al., 2018).

Concept map assessments can also provide insight into content knowledge changes due to participation in a learning activity. These adjustments in content knowledge are of particular importance when trying to assess learning gains for interdisciplinary subject domains such as sustainability. Thus, sustainability education researchers have developed categorical scoring methods to evaluate the content of sustainability-related concept maps that require classifying concepts into sustainability categories and examining the links between concepts in different categories (Segalàs et al., 2008; Watson et al., 2016). This scoring method has been employed to assess the utility of different pedagogical approaches to increase sustainability knowledge (Segalàs et al., 2010).

Study Description

We examined student learning during a 2-week community-engaged fieldwork experience to determine the benefits of short-term service learning on sustainability knowledge. The community-based service-learning experience centered around a multidecade socioscientific issue of water quality in the same watershed as our campus. The tools required to assess water quality (e.g., aquatic biodiversity and chemical sampling) are well within the biology undergraduate curriculum and appropriate for the students chosen (those enrolled in an upper-level undergraduate ecology laboratory course at an R1 institution). Students were asked to create concept maps before and after the experience in response to the question “What is a sustainable community?,” using CmapTools software (Institute for Human and Machine Cognition, 2017). The maps were used to evaluate changes in student knowledge structure and content knowledge. We had students complete midexperience reflection questionnaires to further assess changes in content knowledge and evaluate how students personally felt about the experience. We also examined the reflections for expressions of service-learning objectives, specifically academic enhancement, civic engagement, and personal growth (Ash et al., 2005; Clayton et al., 2013), to understand how students thought about and applied what they were learning during the service-learning experience.

METHODS

Community-Engaged Fieldwork Experience

This 2-week lab unit targeted local stream health, as measured by macroinvertebrate biodiversity, chemical content, and physical stream characteristics. Students in the course were given historical data and a narrative on the sampling sites as a prelab (Supplemental Material), and over the 2 weeks, the students sampled the stream health per U.S. Environmental Protection Agency (EPA) and U.S. Fish and Wildlife Service (USFWS) protocols (EPA, 2017; USFWS, 2018). The environmental monitoring data generated by students (Supplemental Material) were done in collaboration with community members engaged in the long-term monitoring of their watershed; officials from city, state, and federal departments conducting year-round monitoring efforts; and course instructors. Although some of the community members were also professional scientists, the approximate ratio of scientists:community members:students working at the site at any one time was 1:2:4. After accounting for travel to field sites (approximately 30 minutes round trip), each week’s lab was approximately 2 hours long. During time at the site, collection of data and conversations happened ad hoc within and between groups of students, scientists, and community members, seeded particularly through the findings of interesting, anomalous data (e.g., finding a rarer species of organism).

Across the experience, students visited an unhealthy and a healthy stream sampling site (as determined by historical water-quality data from USFWS officials), and the data that the students took contributed to larger city data sets and clean-up advocacy efforts of the local stewardship group, from which the community members and scientists working with the students were recruited. These stream sites fell within the same watershed and were nearby to sites of some of the most economically depressed areas of the city with high rates of poverty and crime. The waterway itself is plagued with urban runoff, erosion, illegal tire dumping/litter, and combined sewer overflows, which have led to both public health threats and the prompting of the community members to fight for and restore their local streams.

Students completed a concept map before their experience on site (both fieldwork and interactions with those outside the course), a reflection between site visits (after 1 week’s experience), and a concept map postfieldwork (after both weeks and at the completion of processing their data). We used these assignments to then determine the learning gains of the students during this service-learning experience. This study was approved before data collection under protocol H18143 granted by the Georgia Institute of Technology’s Institutional Review Board.

Concept Map Assessment

Concept map analysis was performed to assess for changes in student knowledge as a result of the community-engaged fieldwork experience. Students were asked to create concept maps using CmapTools software (Institute for Human and Machine Cognition, 2017) before and after their fieldwork experience, centered around the question “What is a sustainable community?” We provided students with in-depth online instructions and resources on how to use the CmapTools software (Institute for Human and Machine Cognition, 2017) and construct concept maps (Supplemental Material). The concept map assignments were completed outside class time, but students were encouraged to reach out to course instructors with any questions. It is important to note that students had experience creating concept maps by hand as in-class activities in the requisite paired lecture course. Of the 64 students enrolled in the course, 59 of the students (Table 1) submitted a concept map both before and after the experience. These maps were analyzed using two different scoring methods to assess changes in knowledge structure and content.

Concept maps were evaluated using the concept map scoring method presented in Novak and Gowin (1984) to assess for change in student knowledge structure. Pre- and postexperience concept maps were autoscored using cmap-parse software (Pelkey, 2016), which calculates the number of concepts, the longest chain of concepts in a hierarchy (highest hierarchy), and the number of links between hierarchies in a given concept map (Figure 1 and Table 2). These subscores were then used to calculate the structural complexity of each concept map using the following formula from Novak and Gowin (1984), which gives more weight to variables deemed as higher-order learning:

The number of hierarchy links is subtracted from the number of concepts so that concepts in multiple hierarchies are not counted twice.

We also assessed concept maps using a categorical strategy developed by Segalàs et al. (2008) to analyze the content contained within sustainability-related concept maps (Watson et al., 2016). This content analysis scoring method first required each concept in a map to be assigned to one of six sustainability categories adapted from Watson et al. (2016) and defined in Table 3. The number of links between categories was then counted, as well as the total number of categories present in a map (Figure 2 and Table 2). A trained scorer (J.L.P.) manually evaluated each concept map blind to whether it was a pre or post map. The content complexity of each concept map was calculated using the following formula adapted from Segalàs et al. (2008):

We used Wilcoxon signed-rank test to assess for differences between pre and post concept maps for all measured variables (Table 2). Bonferroni adjusted p values are reported to account for the possibility of type I error due to multiple comparisons. All statistical analysis was performed using the coin package (Hothorn et al., 2008) in R v. 3.4.2 (R Core Team, 2017).

Reflection Assessment

Students completed reflections halfway through the experience to provide insight into student thinking. The reflection questionnaire was adapted from the Center for Serve-Learn-Sustain (Hirsch and Thurman, n.d.) based on the VALUE rubrics from the Association of American Colleges and Universities (Rhodes, 2010; Supplemental Material). Of the 59 students who completed both a pre- and postexperience concept map, 55 submitted a complete and legible midexperience reflection. We analyzed the qualitative content of statements contained within student reflections using a deductive approach (Cho and Lee, 2014). We started with the same sustainability categories used in the concept map content analysis scoring method (Table 3), and two readers (J.L.P. & E.G.W.) went through the reflections looking for ideas emerging from the responses. Previous meaningful experience with nature, career goals, and self-reflection were added as additional categories after the first read-through of the reflections. Each sentence/statement was then coded by one of the readers to match each of the nine potential categories, and the number of statements in each category was counted; these were compared with initially high interrater reliability (>90%), and conflicting scores were discussed to achieve 100% agreement across scores.

We also examined student reflections for expressions of service-learning objectives to assess how students thought about and applied the material captured in concept maps on a personal level. We noted the presence of statements in student reflections expressing service-learning objectives and coded them as academic enhancement, civic engagement, or personal growth.

RESULTS

Significant Gains in Knowledge Structure

Based on structural analysis of concept maps (Table 4), student knowledge breadth and depth appeared to increase as result of the fieldwork experience. The number of concepts included in concept maps (i.e., knowledge breadth) increased by 39% on average between pre- and postexperience assessments. The highest hierarchy level (i.e., knowledge depth) also significantly increased after the experience, with students adding one additional level to their longest concept chain, but there was no difference in knowledge connectedness (i.e., hierarchy links). Overall, postexperience concept maps had 30% higher structural complexity scores due to these increases in the number of concepts and highest hierarchy.

Statements included in student reflections demonstrated student expression of academic enhancement, with 13 students mentioning learning benefits of the community-engagement experience. For example, one student said: “We get to see what it’s like to observe nature in a way that’s different from what we’re used to. It made me feel like a real biologist doing hands on work.” Another student explicitly stated the intended desire of service learning which is to help students apply theoretical knowledge to problem solving:

Learning Gains in Ecological and Social Categories Led to More Complex Sustainability Content Knowledge

Sustainability content analysis of student concept maps demonstrated that the increased number of concepts being used by students postexperience were primarily in the ecological and social categories (Figure 3). Seventy-eight percent of students increased the number of ecological concepts included in their postexperience assessments, and 64% of students showed increases in the number of social concepts used. The number of concepts classified in the economic, technical, balance, or temporal categories generally remained constant between pre- and postassessments.

Qualitative content analysis of the midexperience reflections revealed that statements pertained mainly to ecological, social, and technical categories (Figure 3). However, the number of reflection statements coded as ecological was similar to the number of social statements. Also, the percentage of technical statements in reflections was twice as high as those in both pre and post concept maps. The added categories from the deductive approach did not make up a large portion of overall student reflection content, but did demonstrate students gained nonacademic benefits, with 44% of students including at least one statement of self-reflection, 27% mentioning previous meaningful experiences of nature, and 10% discussing career goals.

Self-reflection statements were primarily focused on examples of personal growth experienced, as expressed by this student:

More than half of the number of students included statements related to the service-learning objective of personal growth.

Content complexity of student concept maps as defined by Segalàs et al. (2008) increased postexperience due to increases in the number of concepts and the addition of more links between categories (Table 4). The number of sustainability categories included in student concept maps did not change after the experience, but the number of links between different categories in their postexperience concept maps was 41% greater. Most of these category links were between concepts in ecological and social categories. The content complexity of student knowledge, as calculated using Segalàs et al. (2008) complexity formula, was twice as high for student postexperience concept maps.

Overall, 24 students reflected on their role as a citizen or scientist in society, as demonstrated by including statements of civic engagement. This student felt called to action after the first visit to the stream:

Another student stated, “This experience has also made me think about how I can become more involved in local efforts to protect the environment and create a more sustainable community.”

Overall, 46 of the 55 submitted reflections included at least one statement relating to service-learning objectives, with 18 of those expressing two of the three objectives.

DISCUSSION

Concept map assessment demonstrated that our community-engaged fieldwork experience provided meaningful student learning gains. Service-learning experiences are typically semester-long projects, but our study illustrated that benefits in student learning can be seen in as little as 2 weeks. We found that student knowledge breadth, depth, and complexity increased based on changes in their concept map structure and content before and after the 2-week community-engaged fieldwork experience (Table 4). Additionally, evidence of meeting service-learning objectives was found in student reflections. Together, these results suggests that short but intense service-learning experiences, such as collecting ecological monitoring data alongside professional scientists and community members, can be effective at increasing not only student knowledge, but also impacting students personally.

We saw conceptual change in student knowledge based on increases in the number of concepts used and longer chains of concepts (i.e., highest hierarchy) in postexperience concept maps, leading to more structurally complex maps based on Novak and Gowin (1984) scoring method (Table 4). The concurrent increases in the structure and concept subtotals support the theory that, as the sophistication of a student’s understanding increases, so too does the complexity of the structure of his or her understanding (Markham et al., 1994). The statistically significant gain in highest hierarchies allows us to infer that the largest structural changes in students’ knowledge involved more clustering of specific ideas under overarching ideas, a structuring skill that seems to be positively related to an individual’s degree of expertise with a topic (Miller, 1956).

Concept mapping has also been used to reveal the extent of reorganization of a student’s knowledge structure. Rumelhart and Norman (1978) have described three types of changes in the way individuals reorganize knowledge: accretion, in which new knowledge is added to existing knowledge structures; tuning, in which the accuracy of the knowledge structure is changed; and reconstruction, in which new knowledge structures replace the old ones. We found that our students exhibited knowledge accretion, because they added concepts to their existing concept map structures, but did not increase connections between hierarchies, as illustrated in Figure 4, which shows the changes in a pre- and postexperience concept map submitted by a student.

While the learning gains made by students who participated in this community-engaged service-learning experience were statistically significant (Table 4), the simple addition of new knowledge into an existing knowledge structure represents a weak conceptual change compared with the strong conceptual change suggested by knowledge reconstruction (Carey, 1987). Indeed, Jones and Vesilind (1996) used these ideas to assess concept maps drawn by preservice teachers throughout the senior year of a program and saw knowledge accretion by students in earlier concept maps but knowledge reconstruction in later concept maps, which may suggest a deepening of knowledge with extended experience. In Jones and Vesilind’s work, most students interviewed cited that experiences in field placement opportunities were the reason for the changes in their maps. Thus, while students who participated in our short fieldwork exercise demonstrated changes in their knowledge, more experiences may have been necessary for our students to apply new conceptual frameworks to their thinking about sustainable communities. Repeating the data-collection activity in additional locations to present a broader spectrum of stream environments and local communities may help students reconstruct their knowledge. Additionally, extensions to the experience that may not require on-site work, such as viewing long-term water-quality policies in the area and associated environmental changes or similar case studies, may provide students a new perspective from which to approach the subject, while respecting semester course constraints.

Categorical concept map scoring methods provide information on student content knowledge that is of particular interest for complex and multifaceted topics like sustainability (Watson et al., 2016). We found that, when students were first asked to define a sustainable community using concept maps, their concepts pertained mainly to ecological concepts, but they did include some social concepts (Figure 3). After engaging in the 2-week service-learning experience, they increased the number of ecological and social concepts used and were better able to connect concepts from different sustainability categories, as evidenced by statistically significant increases in the number of category links (Table 4). Learning activities centered around socioscientific issues, such as stem cell research or legalization of marijuana, are shown to increase domain-specific knowledge and contextual interrelationships by providing students an opportunity to attach real-world significance to scientific concepts (Zeidler et al., 2011). Our students gained knowledge on ecological principles and social topics by engaging with the issue of water quality in urban communities and were able to make connections between these course-related concepts and the world they live in.

We found evidence that our short-term service-learning exercise supported personal growth and civic engagement by examining reflection responses. Critical reflection helps students draw connections between service-learning activities and the course material, which enhances the benefits of the experience (Felton and Clayton, 2011) and provides a valuable assessment tool (Ash et al., 2005). A reflection questionnaire issued to our students in the middle of the 2-week stream fieldwork experience allowed us to assess for gains related to service-learning objectives and to connect these to the conceptual change in knowledge supported by the concept map results. We found most students, more than 80%, spontaneously included at least one statement relating to one of the service-learning objectives, which suggests our community-engaged fieldwork experience was effective at producing the significant increases we found in concept map structural and content complexity.

Prior work supports that the assignment task, such as an essay or presentation, may shape what knowledge students reveal; for example, in a debate exercise about genetically modified organisms, students offered only scientific-based arguments, but in interview and online questions, these same students brought up their own personal opinions and the human element of science (Walker and Zeidler, 2007). While our work broadly supports the use of concept maps as an effective alternative to reflection assignments commonly used in service learning, it also suggests that the simultaneous use of multiple methods of assessment provides a more complete understanding of the effects of service-learning instructional approaches on student learning.

We believe that these short-term service-learning experiences can be incorporated into existing curricula beyond ecology classrooms and may provide a useful pedagogical tool for sustainability education at the high school and undergraduate levels. Educators need to examine the resources they have readily available (i.e., personnel, laboratory equipment, training) and reach out to local community collaborators to identify whether there is a tangible need their course work can meet. These collaborations can begin with discussions on campus within and between departments to develop projects that mirror the interdisciplinary and multifaceted nature of sustainability itself.

Limitations

Students were asked to complete concept maps and the reflection questionnaire outside class for participation credit, which may have affected the quality of their work. The assignments did not contribute significantly to the students’ overall course grade, so there may have been little incentive to produce complex maps that required additional effort; we acknowledge this potential limitation, but made this pedagogical choice following Allen and Tanner (2003), who argue that graded concept maps detract from the goal of concept maps for students to illustrate their own knowledge structures and may prompt students instead to seek a right answer.

The postexperience concept maps may have been more complex simply because students had more practice creating maps with the second map and not because of the service-learning experience. However, as noted in the Methods, students did semi-regularly create concept maps by hand in the paired lecture, so this effect would be due to more familiarity with the concept map software and not the idea of concept maps themselves. It is unclear whether there is a difference in concept map construction when students use software or traditional handwritten methods.

Additionally, students were provided with detailed instructions and online resources (Supplemental Material), but did not experience in-class training on creating these digital concept maps, which may have affected the perceived utility (Santhanam et al., 1998) and quality (Kinchin et al., 2005) of the maps. This suggests that our maps may underestimate changes in student knowledge structure and content if maps were rushed or simplified to satisfy completion for credit or limited by student technological abilities, rather than depictions of true knowledge. Future studies could investigate whether in-class introductions to digital concept maps or credit weight impact the quality of concept maps produced and thus quantitative results of concept map assessments.

Our results suggest that 2-week service-learning experiences have the potential to change students’ knowledge and enhance the ability of students to connect concepts. Given that our data were collected one time in one course at one institution, it is important to conduct similar studies at other institutions with larger samples of students to determine the replicability and generalizability of the learning gains seen in our study. Future research could make use of the same sustainability categories used in this study and others (Segalàs et al., 2008; Watson et al., 2016) to compare results across instructional approaches.

ACKNOWLEDGMENTS

REFERENCES