ASCB logo LSE Logo

Development and Effectiveness of an Educational Card Game as Supplementary Material in Understanding Selected Topics in Biology

    Published Online:https://doi.org/10.1187/cbe.13-05-0093

    Abstract

    The complex concepts and vocabulary of biology classes discourage many students. In this study, a pretest–posttest model was used to test the effectiveness of an educational card game in reinforcing biological concepts in comparison with traditional teaching methods. The subjects of this study were two biology classes at Bulacan State University–Sarmiento Campus. Both classes received conventional instruction; however, the experimental group's instruction was supplemented with the card game, while the control group's instruction was reinforced with traditional exercises and assignments. The score increases from pretest to posttest showed that both methods effectively reinforced biological concepts, but a t test showed that the card game is more effective than traditional teaching methods. Additionally, students from the experimental group evaluated the card game using five criteria: goals, design, organization, playability, and usefulness. The students rated the material very satisfactory.

    INTRODUCTION

    Many educators are concerned that students do not spend enough time engaged in independent thinking, group discussions, or active learning. Individuals are likely to learn more when they learn with others than when they learn alone (Michael and Chen, 2006). Johnson and colleagues (2000), in their meta-analysis of 164 studies of cooperative-learning methods, have found out that there is solid evidence supporting the benefits of cooperative learning.

    It has been observed that one common problem encountered by the students in the biological sciences is difficulty in understanding biological concepts. Many students become discouraged by the course because of the complex vocabulary they need (or they believe they need) to memorize in order to understand the subject. In effect, efforts should be made to reduce the total amount of factual information students are expected to memorize; reduce the use of the passive lecture format; and devote more effort to helping students become active, independent learners and problem solvers (Vander, 1994).

    Ellington et al. (1981) noted that educational games have favorable characteristics for use in science education. One of the main ways to use games in science is to reinforce basic facts and principles. Once the basic facts of a particular lesson have been taught, it is often necessary to reinforce the knowledge the students have just acquired by giving them some form of exercise in which they have to demonstrate their understanding of what they have learned by applying the knowledge to a specific situation. Traditionally, such exercises have generally taken the form of worked examples the students complete on their own, either as seat work or as homework assignments. However, it would be possible to achieve the same objectives by employing a game of some sort. Though it is certainly not recommended that all worked examples in a course be replaced by games, it is suggested that it might, in some cases, be beneficial to introduce a few carefully chosen games into a course in situations in which they seem to offer some distinct educational advantage over more traditional approaches. Using educational games helps enrich instruction by exposing students to a creative form of learning and introducing a fun element into the lesson, thus diminishing drudgery in learning contexts, as well as reviving flagging enthusiasm among students when attention is waning (Subramaniam et al., 1999).

    There have been studies on the use of educational games in learning biology (Cohen et al., 1989; Odenweller et al., 1998; Franklin et al., 2003; Rivera, 2006). Most of these attempts, however, looked only at students’ perception of the use of the said methods. There are fewer data on the effectiveness and usefulness of these games to enrich students’ learning. In this study, the researcher documented the procedure he used in the development of an educational card game as supplemental material for learning biology. The developed card game was then evaluated and tested for its effectiveness.

    MATERIALS AND METHODS

    The study was composed of two phases: phase 1 was the development of the educational card game, and phase 2 was the actual use, testing for effectiveness, and evaluation of the developed card game. The researcher combined the instructional content and objectives of the included topics in the format of a card game (Figure 1).

    Figure 1.

    Figure 1. Conceptual framework of the study.

    Figure 2.

    Figure 2. Sample playing cards. Left, nutrition and digestion; right, respiration and circulation.

    Figure 3.

    Figure 3. Sample card combination and justification.

    The following steps were used to develop and test the effectiveness of the educational card game:

    1. Preparatory stage. This included conceptualization and planning of the instructional material developed. The current methods of instruction were assessed together with the existing needs and problems encountered in the teaching–learning process. Research, preparation of materials, and other pertinent aspects of the development of the game were established in this stage.

    2. Game development. This included the design, development, and preimplementation of the educational card game. For this study, the researcher selected two main topics in biology: 1) nutrition and digestion and 2) respiration and circulation. The following steps were used in the development of the educational card game:

      1. Selection of the key terms from the selected topics. The researcher selected 90 terms from the reference material used to teach each topic. The researcher believes that the selected terms will provide a comprehensive review of the topic when instructionally combined during the game. Each term was repeated twice, for a total of 180 cards per set (topic). Duplication of terms allows the student to use the term in more than one application.

      2. Card design. The design of the cards was influenced by the cards used by Odenweller et al. (1998) in their card games for supplemental teaching of gastrointestinal physiology. Some modifications were done as defined by the next two sentences. Each of the terms was printed strategically on each of the four sides of a 6.1-cm by 8.6-cm board paper. An iconic picture for the topic was also included and was printed at the center of the card (Figure 2).

      3. Game rules, pilot testing, and revision. Most of the rules were derived from the popular card games tong-its and pusoy-dos (popular Philippine card games similar to rummy) and from a similar study by Gaugi and Hodges (1980). The main objective is for the student-players to form and justify combinations that demonstrate a clear conceptual relationship between the terms on the included cards. The player should be able to justify his or her combination (see Figure 3 for sample combination). Immersion in instructional content, playfulness, and peer interaction were among the top considerations in drafting the rules of the game. The researcher's previous biology classes were utilized to pilot test (preimplementation) the educational card game. Revisions, especially to the game rules, were done after the researcher's observations during the pilot test.

      4. Expert review and validation. After the initial development process was completed and before the actual use of the game, eight biology teachers (subject matter experts [SMEs]) were asked to review and validate the educational card game based on its goals and objectives, design, components and organization, playability and playfulness, and usefulness. A week before the evaluation day, the researcher gave each SME a copy of the rules of the game and handouts of the biology topics covered to ensure the evaluators were familiar with the rules of the game and the topics concerned. On the day of the evaluation, the researcher discussed the goals, objectives, and rules of the game before letting the SMEs play the game in small groups. The researcher personally distributed and collected the evaluation instruments. Generally, comments and feedback from the SMEs were positive.

    3. Testing for effectiveness. The researcher utilized the pretest–posttest experimental design to test the effectiveness of the card game. Two intact biology classes at Bulacan State University (BulSU)–Sarmiento Campus who were enrolled for the first semester of academic year 2011–2012 were the subjects for this part of the study. One of the classes was assigned as the experimental group (BSEd English 1, n = 53), and the other class was the control group (BEEd Generalist 1, n = 54).

      1. Sample populations were chosen based on similar mean IQ scores. Data on IQ scores were obtained with permission from the Guidance Center of BulSU. The mean IQ score of the experimental group (98.225) was statistically the same as that of the control group (97.200; t = 0.800 at 0.05 confidence level). Students who did not have an IQ score available and those who were not able to take a pretest or posttest were not included in the sample. Each test group included 40 students.

      2. Both groups took the pretest before the experiment. The mean pretest score of the control group was 19.222/50, while the experimental group had a mean pretest score of 20/50. The entry skills of both groups were statistically the same (t = 0.633 at 0.05 confidence level). The two groups were taught the selected topics through lecture and discussion. Reinforcement activities were given to the students after each lesson. For the experimental group, reinforcement was done through playing card games in small groups. The students were required to play the game during their break times. Submission of score sheets and pictures and videos of actual game playing were acquired. Traditional assignments and exercises were given to the control group. The posttest was administered to both groups after the topics in the study had been taught and reinforced. The mean gain score (difference of posttest and pretest scores) of each group was compared for significant difference, using the t test for independent means. This was done to determine whether the use of an educational card game can be an effective supplemental material in teaching and learning biological concepts.

    4. Evaluation of the educational card game. After the testing stage, students from the experimental group evaluated the card game. Evaluation of the educational game was based on five criteria: 1) goals and objectives; 2) design; 3) components and organization; 4) playability and playfulness; and 5) usefulness. A space for comments was also provided for the evaluators.

    Research Instruments

    Evaluation Instrument.

    The instrument used to evaluate the developed educational card game was a five-point rating scale from a similar study by Odenweller et al. (1998). The adopted instrument was then revised, mainly through rearrangement and addition of items based on other related studies (Rivera, 2006) and the literature (Hong et al., 2009), to suit the needs of the study. The questionnaire has been validated by two faculty members of the College of Science of BulSU who are specialists in biology.

    Pretest/Posttest.

    The researcher prepared a 60-item, multiple-choice test to evaluate the effectiveness of the educational card game. The test was made after making a table of specification (TOS). A TOS is a test blueprint that shows how the teacher came up with test questions (e.g., number of questions per topic relative to the amount of time it was taught, the level of thinking skills [knowledge, comprehension, application, analysis, synthesis, or evaluation] the student will employ to answer a question). The constructed test items were validated by SMEs who were Graduate School of BulSU students enrolled in a biology education class during the third trimester of the 2010–2011 academic year. Minor changes were made based on the comments and suggestions of the SMEs. This test was given to the students of BSEd-Physical Science 1 who took the lessons in advance to determine the reliability of the test. The test was then reduced to 50 items after eliminating the five most difficult and five easiest items through an item analysis.

    To determine the reliability level of the test, the researcher used Cronbach's alpha. Cronbach's alpha ranges from 0 to 1.00, with values close to 1.00 indicating high consistency. For a classroom exam, it is desirable to have a reliability coefficient of 0.70 or higher (Wells and Wollack, 2003). The test obtained an alpha value of 0.861, which is relatively good and acceptable for a classroom exam.

    RESULTS AND DISCUSSION

    Effectiveness of the Educational Card Game

    In testing the effectiveness of the educational card game as a reinforcement activity, a pretest–posttest experimental design was utilized. Table 1 presents the significant difference between the performance of the students in each group before and after the experimentation as shown in their mean pretest and posttest scores. A t test for dependent means was conducted to show whether there was a significant increase in score from pretest to posttest. For the control group, the mean posttest score of 28.45 yielded a mean gain score of 9.225 from their mean pretest score of 19.225. The t test computed a value of 8.940. Comparing it with the critical t value of 2.021, it can be determined that there was a significant increase in the performance of the students exposed to the traditional method. A study conducted by Candido (2000) also supported the effectiveness of traditional methods. Traditional methods are still an effective way of teaching and learning.

    Table 1. Significant difference between the mean pretest and posttest scores of students in each group

    MeanMean gain scoreSDComputed tdfCritical t (0.05 level of confidence; two-tailed)
    Control group (traditional)
    Pretest19.2259.2256.5268.940392.021
    Posttest28.45
    Experimental group (educational card game)
    Pretest2012.6755.75313.934392.021
    Posttest32.675

    For the experimental group, a mean gain score of 12.675 was determined from the mean posttest (32.675) and pretest (20) scores of the students. A t test for dependent means was used to determine whether a mean gain score of 12.675 is enough to say that the performance of the students significantly increased after playing the educational game. A higher computed t value (13.934) than the critical t value (2.021) means there was a significant difference between the mean pretest and posttest scores of the experimental group. It therefore can be said that the use of educational card games is an effective adjunct to traditional methods of teaching. There have been a number of researches conducted that support the effectiveness of using educational card games and other related constructivist approaches (Cohen et al., 1989; Lazarowitz et al., 1994; Candido, 2000; Burrowes, 2003; Marasigan, 2006; Rivera, 2006; Tuzun et al., 2009;Barclay et al., 2011).

    Summarized in Table 2 are the statistical data for comparing the mean gain scores of the students who were exposed to the educational card game (experimental) and those who were exposed to traditional teaching (control). A t test for independent means was used to determine whether there was a significant difference between mean gain scores of the two groups. The computed t value of 2.508 is higher than the critical t value of 1.96. This would mean there is a significant difference between the two mean gain scores. The result of the test favors the experimental group with its higher mean gain score of 12.675 compared with the 9.225 mean gain score of the control group. This further revealed that using an educational card game is a more effective method of teaching and learning compared with traditional methods. A similar study by Dimagiba (1997) also yielded the same result.

    Table 2. Significant difference between the mean gain scores of students from the experimental and control groups

    Mean gain scoreMean differenceSDComputed tPopulation of the sampleCritical t (0.05 level of confidence)
    Control9.2253.456.5262.508401.96
    Experimental12.6755.75240

    Evaluation of the Educational Card Game

    The researcher interpreted students’ responses based on frequency distribution, weighted mean, and a brief verbal interpretation in each of the given items and criteria in the evaluation instrument (Table 3).

    Table 3. Descriptive evaluation of the card game

    Frequency
    Items54321MeanVerbal interpretation
    Goals and objectives
    1The purpose and rationale for the game are fully explained.1522714.13Very satisfactory
    2The goals and objectives of the game are clearly defined.162544.27Very satisfactory
    3The game was thought provoking.929524.00Very satisfactory
    4The game encouraged student interaction.261364.44Very satisfactory
    5The game promoted discussion of key topics.192244.33Very satisfactory
    6The card game helps with my recall of concepts/terms.281524.58Outstanding
    Average mean4.29Very satisfactory
    Card design
    7Card size is appropriate.2715214.51Outstanding
    8Having terms printed on all four sides of the card is a helpful feature for the players’ handling of the cards.271534.53Outstanding
    9The picture printed on the card is representative of the topic.2118514.31Very satisfactory
    10The material used (paper) in the preparation of the cards is durable.13191123.96Very satisfactory
    11The deck of cards is compact and can be easily carried around.18151024.09Very satisfactory
    Average mean4.28Very satisfactory
    Components and organization
    12The directions were clear, concise, and easily understood.1220133.98Very satisfactory
    13The game emphasized key points of the topic played.152374.18Very satisfactory
    14The terms used were appropriate to my level of knowledge.92794.00Very satisfactory
    15The number of cards was appropriate.12211113.98Very satisfactory
    16The length of time required to play the game is reasonable.1821334.20Very satisfactory
    Average mean4.07Very satisfactory
    Playability and playfulness
    17The game provides opportunity for healthy competition and cooperation.261724.53Outstanding
    18The rules of the game provide players with equal conditions for a fair play.192154.31Very satisfactory
    19The rules of the game provide a set of options for flexibility in making decisions when playing the game.162724.31Very satisfactory
    20Playing the game was fun.2711524.40Very satisfactory
    Average mean4.39Very satisfactory
    Usefulness
    21The game was effective in reviewing the material.36904.80Outstanding
    22The game encouraged the players to dig deeper into the subject matter.242014.51Outstanding
    23Playing the game is a productive use of time.1917814.20Very satisfactory
    24Playing the game help me establish better relationships with the members of the group.251734.49Very satisfactory
    25I would recommend the game to my peers.1026814.00Very satisfactory
    Average mean4.40Very satisfactory
    Overall mean4.29Very satisfactory

    The first set of items is about the goals and objectives of the developed educational game. It can be seen from the data presented in the table that students find the goals and objectives of the game to be “very satisfactory” (average mean of 4.29). Overall, the students reported that the goals and objectives of the game were met. In fact, most items in this criteria set have received a “very satisfactory” rating from the users. Only item 6 received an “outstanding” mean rating of 4.58. This particular item, together with the results of the evaluation for items 4 (4.44 = very satisfactory) and 5 (4.33 = very satisfactory), were consistent with the findings of Odenweller et al. (1998), which showed that card games were successful in promoting learning, aiding concept recall, and engaging students in the discussion of topics. Item 3 received the lowest mean rating (4.00, although still very satisfactory). Items 1 and 2 also received relatively lower mean ratings. Presentation of the goals and objectives of the game were done before letting the respondents play the game.

    The next set of items presents the respondents’ views with regard to the design of the cards used in the educational game. The first two items of this criteria set received the highest mean rating. Both items 7 and 8 received “outstanding” mean ratings. Together, these two items influence the players’ handling of the cards. The respondents agreed that the size of the card is appropriate and the feature of having the terms strategically printed on all sides was helpful for the effective handling of the cards. It can be noted that among the items in this criteria set, it was item 10 that received a mean rating of less than 4 (3.96 = very satisfactory). Item 9, which is about the picture used in the cards, also received a relatively lower mean rating (4.31). Choice of a more durable material and a more representative (or specific) icon (picture) should be among the primary considerations of educators who plan to use educational card games. The last item in this criteria set, which is about the portability of the deck of cards, received a “very satisfactory” mark (4.09) from the students. This is in line with what Ellington et al. (1981) have noted: cards used in educational game should be compact and thus easily stored and carried around. In general, the card design has received an average mean rating of 4.45, which is “very satisfactory.”

    Items 12–16 are about the components and organization of the game. It can be seen that students consistently gave a mean rating of “very satisfactory” in all items of this criteria set. Although their ratings are both “very satisfactory,” items 12 and 15, which concern the playing directions and number of cards, received the lowest ratings. Some of the students found the rules and number of cards quite complicated. Odenweller et al. (1998) also reported a similar problem, reporting that students felt the number of cards was inappropriate. On the other hand, the students really agreed that the length of time required to play the game was reasonable (item 16: mean rating of 4.20). The respondents also agreed that the terms used were appropriate to their level of knowledge (item 14) and the game emphasized key points of the topic (item 13) as manifested by the mean ratings of 4.00 and 4.18, respectively. This criteria set has received an average mean rating of 4.07. The respondents find the way the game has been composed and organized “very satisfactory.” A similar study by Odenweller et al. (1998) also reported that students found the components to be clear and the game to be well organized.

    The fourth set of items deals with the playability and playfulness of the game. The respondents found the card game “outstanding” (4.53) in terms of providing an opportunity for healthy competition and cooperation. This is what item 17 was all about. The second highest mean rating of 4.40 was obtained by item 20 (Playing the game was fun). This is consistent with the results of the study about educational games conducted by Persky et al. (2007) and Rivera (2006). The two other items, 18 and 19, both received a “very satisfactory” mean rating of 4.31. These two items are both concerned with the effect of the rules on the playability and playfulness of the game. This result supports the claim of Hong et al. (2009) that playfulness of an educational game is influenced by the degree of uncertainty and flexibility in decision making, the level of challenge, equal conditions for fair play, and the level of interactivity.

    The last set of items is statements evaluating the usefulness of the educational card game. The respondents have agreed that the educational card game is “outstanding” in terms of reviewing the material and digging deeper into the subject matter, as manifested in items 21 and 22, respectively. These ratings are parallel with the result of the study of Beylefeld and Struwig (2007), which indicated that informal learning methods like the use of games can enhance the recall of factual knowledge. The respondents also agreed that playing the game was a productive use of time. Requiring the students to play the game a couple of times resulted in their break times being used effectively and productively. The respondents also recognized that the game helped them to build better relationships with other players, as shown by a “very satisfactory” rating (4.49) of item 24. Similar results were obtained in the studies by Jaipal and Figg (2009) and Beylefeld and Struwig (2007). Finally, respondents agreed that the game is worth recommending (item 25). A “very satisfactory” (4.00) rating was given by students to this item. An average rating of 4.40 (very satisfactory) was obtained by this educational card game based on its usefulness.

    The developed instructional material obtained an overall mean of 4.29, which is also “very satisfactory.” All the criteria acquired a mean rating very close to the overall mean of 4.28, except for the components and organization, which received the lowest ratings from the student evaluators. This is the criteria set concerning the rules, the biological terms included, the number of cards, and the length of play time, which the researcher thinks are the focal points of the material. Further development on these aspects is recommended for a more effective and acceptable educational card game. Usefulness received the highest rating (4.40) from the students, which shows that the card game is really useful in terms of learning and recalling biological concepts. This is consistent with the positive result acquired by the card game in the test for effectiveness.

    CONCLUSIONS AND RECOMMENDATIONS

    The mean posttest scores of the two groups both increased significantly as compared with their pretest scores. Both methods (traditional or using the educational card game) were found to be effective methods of enhancing understanding of biological concepts. However, students who used the educational card game had a significantly higher gain in scores than students who were exposed to the traditional methods. Thus, it has been discovered that the use of a student-centered approach like educational card games can improve student performance to a greater extent than traditional methods. The results of the evaluation of the educational card game showed that the students find the game in general to be very satisfactory as a supplemental material for reinforcing acquired knowledge and skills.

    Because the educational card game developed in this study received very satisfactory feedback from its users and is statistically supported as an effective supplemental material for reinforcing learning, teachers should consider using it. However, teachers are advised to come up with their own set of terms to be included in the set of cards based on the reference material they are using. Educators should formulate other rules aside from the ones presented in this paper so that students can also play the game in a different manner (one that is possibly more playful and effective). Likewise, educational card games can also be developed for other lessons in biology (e.g., photosynthesis and cellular respiration) and other disciplines. This researcher believes the nature of this card game is applicable to most if not all disciplines and recommends parallel studies, especially on a wider scale (larger population), to further establish the claim of the effectiveness of educational card games in reinforcing learning. A study on the effectiveness of educational card games for students of different ability levels or learning preferences, which was not part of the current study, could also be made.

    ACKNOWLEDGMENTS

    The researcher thanks the following for their participation and support in the conduct of the study: Dr. Erlinda Villamoran, the people of BulSU Graduate School and BulSU Research and Development Center, and my students and colleagues at Sarmiento Campus.

    FOOTNOTES

    The materials used in this study are available upon request to the author. Please send inquiries to .

    REFERENCES

  • Barclay SM, Jeffres MN, Bhakta R (2011). Educational card games to teach pharmacotherapeutics in an advanced pharmacy practice experience. Am J Pharm Educ 75, 33. MedlineGoogle Scholar
  • Beylefeld AA, Struwig MC (2007). A gaming approach to learning medical microbiology: students’ experiences of flow. Med Teach 29, 933-940. MedlineGoogle Scholar
  • Burrowes PA (2003). Lord's constructivist model put to a test. Am Biol Teach 65, 491-502. Google Scholar
  • Candido BO (2000). The effect of games lecture on the achievement of high school chemistry students with different ability levels. Master's thesis, Manila: Technological University of the Philippines. Google Scholar
  • Cohen A, Yaakobi D, Ben-Porat A, Chayoth R (1989). The effects of biology games on students’ anxiety and in their achievement. Int J Sci Educ 11, 387-394. Google Scholar
  • Dimagiba EM (1997). The use of games in teaching selected chemistry topics in college chemistry. Master's thesis, Manila, Philippines: De La Salle University. Google Scholar
  • Ellington H, Addinall E, Percival F (1981). Games and Simulations in Science Education, New York: Nichols. Google Scholar
  • Franklin S, Peat M, Lewis A (2003). Non-traditional interventions to stimulate discussion: the use of games and puzzles. J Biol Educ 37, 79-84. Google Scholar
  • Gaugi JF, Hodges D (1980). Using rummy to teach cardiovascular and respiratory physiology. Physiologist 23, 68-69. Google Scholar
  • Hong J-C, Hwang M-Y, Lu C-H, Cheng C-L, Lee Y-C, Lin C-L (2009). Playfulness-based design in educational games: a perspective on an evolutionary contest game. Interact Learn Environ 17, 15-35. Google Scholar
  • Jaipal K, Figg C (2009). Using video games in science instruction: pedagogical, social, and concept-related aspects. Can J Sci Math Technol Educ 9, 117-134. Google Scholar
  • Johnson DW, Johnson RT, Stanne MB (2000). Cooperative Learning Methods: A Meta-Analysis In: Minneapolis: University of Minnesota. Google Scholar
  • Lazarowitz R, Hertz-Lazarowitz R, Baird JH (1994). Learning science in a cooperative setting: academic achievement and affective outcomes. J Res Sci Teach 31, 1121-1131. Google Scholar
  • Marasigan E (2006). The use of games in teaching selected topics in chemistry. Master's thesis, Manila: Technological University of the Philippines. Google Scholar
  • Michael D, Chen S (2006). Serious Games: Games That Educate, Train and Inform, Boston: Thomson Course Technology. Google Scholar
  • Odenweller CM, Hsu CT, DiCarlo SE (1998). Educational card games for understanding gastrointestinal physiology. Adv Physiol Educ 20, S78-S84. Google Scholar
  • Persky AM, Stegall-Zanation J, Dupuis RE (2007). Students perceptions of the incorporation of games into classroom instruction for basic and clinical pharmacokinetics. Am J Pharm Educ 71, 21. MedlineGoogle Scholar
  • Rivera ET (2006). Games as enrichment activities in the study of the digestive, circulatory, and skeletal systems In: Master's thesis, Cebu City, Philippines: University of San Carlos. Google Scholar
  • Subramaniam R, Khang GN, Sai CL (1999). Word Juxtapoz—an innovative tool for promoting interest in biological education. J Biol Educ 33, 103-104. Google Scholar
  • Tuzun H, Yilmaz Soylu M, Karakus T, Inal Y, Kizilkaya G (2009). The effects of computer games on primary school students’ achievement and motivation in geography learning. Comp Educ 52, 68-77. Google Scholar
  • Vander AJ (1994). The excitement and challenge of teaching physiology: shaping ourselves and the future. Adv Physiol Educ 12, S3-S16. Google Scholar
  • Wells CS, Wollack JA (2003). An Instructor's Guide to Understanding Test Reliability, Madison: University of Wisconsin. Google Scholar