Article: Abramovich, S., Schunn, C. & Higashi, R. M. (2013). Are badges useful in education? It depends upon the type of badge and expertise of learner. Educational Technology Research and Development, 61(2), 217-232.

Summary of the Article

Abramovich, Schunn and Higashi conducted this study to find out whether badges with different types are effective for educational purposes or not for students with different performance levels. The authors mentioned that educational badges could be an alternative type of assessment, which may increase student motivation and learning. In the introduction section, they pointed out some motivational theories which they believed the most related to the educational badges. Although the authors mostly concentrated on the achievement goal theory, they also investigated if the interest level of participants to a subject (intrinsic motivation) and their expectations to be successful (expectancy value theory) could be interacted with educational badges.

For the study, they recruited 36 students at 7th grade and 15 students at 8th grade (N=51) from a charter school which serves a low-income suburb of a large east coast city in North America. They pre-tested students' math abilities on the subject of proportional reasoning and divided them into two groups: low-performers and high-performers. Then they measured the students before and after the experiment to understand their achievement goal orientations (mastery approach goals, performance approach goals and performance avoidance goals) by using the patterns of adaptive learning scales. In addition, they used five questions to measure expectancy and two questions for interest which were adapted from other studies, which were also collected before and after the experiment. Also, they surveyed the participant to learn their thoughts about badges at the end of the study. The authors stated Cronbach alphas of all surveys used in the study, in which the lowest alpha was 0.66.

For the experiment phase, they used CS2N intelligent tutoring system on the computer to teach the proportional reasoning subject. Students used this tutoring computer system for 20 min for each school day for almost one month. While using the tutoring system, students were provided badges throughout their tutoring process. Two type of badges were used for this study: skill badges (students obtained this badge when they mastered the measured skills) and participation badges (students obtained these badges when completing units).

This is a correlational study that the researchers looked at the relationships between low-and high-performing students, number of earned different badge types (skill vs. participation), students’ achievement goal orientations, students’ expectancy and interest. The authors conducted Pearson correlations between the number of earned badges and the different measures of motivation, prior ability, and pre-post changes in motivation. In addition, they used paired sample t-tests for pretest and posttest changes in motivation concepts.

They found that students with different performance levels (prior knowledge or interest in math) showed interests for different types of badges. That is, students with high performance level (as they have more prior knowledge and intrinsic motivation) gained more skill badges compared to low-performers, while students with low performance level (as they have less prior knowledge and less internal interest to math) earned more participation badges than skill badges.  When the authors analyzed their results in terms of motivational learning goals and badge acquisition; they found that there was a significant positive correlation between the earning more participation badges and the performance approach goal of low-performing students. Thus, these students wanted to earn more badges to outperform to others. However, for low-performing students earning badges did not influence concerns about their performance, which is the performance avoidance goal, but their interest in math was increased significantly between pre- and post-test. In addition, there was a significant negative correlation between the earning skill badges and the performance avoidance goal of high-performing students. Moreover, there was a significant positive correlation between the number of earned skill badges and expectancy for success in math for high-performing students. Both groups had positive opinions about the badges. In addition, authors concluded that their study also supported the notion that extrinsic motivators decrease learning, as badges (especially participation badges) did not motivate students for mastery to earn more skill badges. Also, they stated that skill badges could be used as an alternative form of assessment. Moreover, they wrote that practical implication of their study for instructional designers is that ability level and motivation level of learners have to be taken into account while choosing which badge types to include.

Evaluation of the Article

The introduction section of the article was well written for introducing what the educational badges are and why they should be used in education (such as an alternative assessment type, a tool to increase motivation and learning) to unfamiliar audiences. The authors benefitted from well-established motivation theories, which are the achievement goal theory and expectancy value theory, to unpack the effects of badges on learners’ motivation, expectancy and interest behaviors. They stated their research questions in the article, however they did not provide any hypothesis, which could be reasonable as there was not enough evidence about effects of badges in educational context.

The method section was written poorly that they did not included much information about participants such as how participants were chosen, their genders and age range that we only know they were 7^th and 8^th graders. While most of the studies about gamification tools are conducted in online learning environments with undergraduate students and it is nice that this study recruited middle school students. The authors recruited only 51 subjects, which is a very small sample that make it difficult to generalize the results. In addition to problems in participant section, the authors did not provide much details about the procedure of the study too. For example, it was not clear how many times they collected expectancy and interest questionnaires. Furthermore, they stated that students spent 20 min each school day with the tutoring but they did not write the conditions under which students interacted with the tutoring system: in their school or somewhere out of the school. And it was not stated when students used tutoring system: only in math lessons or in any convenient time of the day for researchers. If students pulled away anytime for the study, I think this would decrease motivational level of students (especially by low-performing students), such that they may not like to spend time with math tutoring during such as PE or Art classes time period. In addition, they stated that they did not mention about badges to the participants at the beginning of the study, so that students found out the badges during tutorial by themselves. They did not provide any explanations why they did this way and I wonder the reason behind it.

I found the result section a little complicated to understand, so their inclusions of tables and figures helped me a lot to understand what they wrote in this section. Since this is a correlational study, there were too many data points which is sometimes difficult to understand. For example, there are two conflicting findings, which took a lot of my time to understand. That is, they wrote that “for the low-performing students (n = 23), we discovered that performance avoidance motivation (e.g. One of my goals in Math is to avoid looking like I have trouble doing the work.) decreased by 0.48” (p. 225). However, they also stated later in the article that “For the low-performing students … total badges earned correlated with an increase in performance avoidance (r = 0.42, p<0.05)” (p. 226). I still did not understand if there is a decrease in performance avoidance motivation of low-performing students between pre- and post-measurement, how there is an increase in performance avoidance while earning badges. In addition, they recruited both seventh and eighth graders for this study, however they never provided any data if there was a difference between the results of seventh and eighth graders. We even did not know if all low-performers are seventh graders or it was a mix group from both grades. Moreover, the authors measured the achievement goal orientation of students at the beginning and end of the tutorial. Although they found some significant correlations between pre measured goal orientations and earned number of badges, they did not include the data for the correlation between post measured goal orientations and earned number of badges in the result section. I think that if the earned badges during the tutorial caused any changes in the achievement goal orientations of learners, post measured goal orientations would be more appropriate to find out correlation between them.

The discussion section was a little short but well written. They connected their arguments in the introduction section with those in discussion section. However, they did not mention about the limitations of their study.

There are several important issues in general about this article. First of all, this is a correlational study that we could not infer any causation, so there could be some confounding variables which may affect the learners’ responses to motivational measures and the number of badges that they earn during the tutorial. Secondly, the tutorial in this study was about one specific subject in math and whole tutorial lasted just a month, which is a short time period to observe any changes in motivational behaviors (their goal orientation, expectations and interest in math) of learners, which could affect the obtained results for this study. Lastly, the authors mentioned persistently throughout the article about the possible use of badges as an alternative assessment. However, they did not investigate if badges could be used as an alternative assessment type or not in this study. That is, they mentioned that skill badges may have a potential for an alternative assessment but they did not examine if this is true or not. They might have given a post test for the proportional reasoning subject at the end of the tutorial and looked at if there is a relation between the number of earned skill badges and students’ performance at the post test. For example, although low-performer did not get as many skill badges as high performers, they may still perform as good as high-performers at the post test of proportional reasoning, which could impair the claim that badges could be used as an alternative assessment type.

Finally, the results of the study could be important for the educators and instructional designers. They found that the effects of badges change based on the learners’ different prior ability level. Also, different badge types affect different learners’ motivation. I like their inclusion of different badge types (skill and participation badges), which attracted different learner types. Although they did not mention the self-efficacy concept in the article, I think that including different badge types for different ability users could be a good idea to increase motivation of students through increasing their self-efficacy beliefs as well. The authors stated that their study could provide some insights for the instructional designers how to include badges in the curriculum such as considering different ability and motivation level of learners. As there is increasing interest for badges and gamification tools in education, educators and researchers may benefit the findings of this study to understand how educational badges affect learning motivation of students with different achievement goal orientations and which badges to include and how to include them in the design of a course.