Article: Hakulinen, L., Auvinen, T., & Korhonen, A. (2015). The effect of achievement badges on students’ behavior: An empirical study in a university-level computer science course. International Journal of Emerging Technologies in Learning (IJET), 10(1), 18-29. doi:10.3991/ijet.v10i1.4221

Summary of the Article

Hakulinen, Auvinen and Korhonen (2015) investigated if achievement badges have an effect on students’ behaviors in terms of time management, carefulness and learning. And their second research question was regarding the attitudes of students towards badges. In the introduction section, the authors reported different definitions of achievement badges (such as “secondary reward systems with optional sub-goals” or “additional system which provides optional goals and challenges” [p. 19]) and provided a review of mixed findings of earlier studies about the effects of badges on engagement and motivation. Moreover, they talked about the Cognitive Evaluation Theory (CET) to mention about the possible risks of gamification. That is, according to CET external reward could have informational and controlling aspects and if a reward is perceived as a controlling, it may decrease motivation while a reward perceived as an informational may increase it. They also mentioned about the problems of automated assessments. Previous research found that students have an inclination of procrastination and trial-error attempts when multiple submissions are allowed. As this study used automated assessments as a measurement method, the author thought that conducting this study would be worthwhile to understand if achievement badges could have an effect of these bad study habits of students.  

The researchers randomly assigned students (N=281) into a treatment group with badges visible and a control group without badges visible. The experiment was conducted in the Data Structures and Algorithms course in Aolta University in Finland. The course has 56 homework assignments which were divided into 8 rounds of exercises with an achievement badge corresponding to each one and students had to take these exercises from the TRAKLA2 online learning environment. They designed eight different achievement badges and added to TRAKLA2 under three categories: time management badges, carefulness badges and learning badges. Thus, badges were awarded for promptly turning in exercises before deadline, solving exercises without error and completing exercise rounds with full points. They collected following data from TRAKLA2: final grades, log data for the time used per exercise, number of sessions, total time, and normalized total number of badges. They also collected a survey which composed of Likert-scale questions and open-ended questions at the end of the course.

The number of awarded badges was not normally distributed, so the data were analyzed by two-tailed Wilcoxon rank-sum test, a non-parametric test. The authors had to normalized the number of awarded badges data and they found that there was a significant difference between treatment and control groups in terms of earned badges. So treatment group was intentionally pursuing the badges and they earned more badges than the control group. In terms of the time management, although students in the treatment group started slightly earlier to the exercises, the difference between groups was not significant. As for the carefulness category, researchers measured this construct by looking at the mean time spend per submission and the number of the attempts per exercise. They found that students in the treatment group spent significantly more time for submissions and used less attempts, but this difference was not significant. As for the learning category, treatment group spent significantly more time in TRAKLA2 and had more number of sessions in TRAKLA2. Also, students in the treatment group got slightly better (but statistically insignificant) grades from the exercises. Finally, the survey revealed positive attitudes toward badges by most of the students and students reported that badges were motivating and badges affected their behaviors.

The authors concluded that achievement badges could be a promising way to increase motivation of students while decreasing their undesirable study behaviors. They also suggested that badges should not be a part of course grading and they should provide voluntary challenges. Otherwise, this may cause badges to be perceived as controlling by the learner and may decrease learners’ internal motivation as proposed by the Cognitive Evaluation Theory. These findings of the study could have important practical implications for the instructors and instructional designers about how to implement badges into course design and possible effects of badges on the student behaviors.

Evaluation of the Article

The introduction section of the article was well written to explain what achievement badges are and the authors summarized the earlier research about effects of badges on learner motivation and engagement. The authors mentioned only one theory in the article, which is the Cognitive Evaluation Theory, while explaining the possible disadvantages of badges on the internal motivation of learners. Hence, their research questions mostly followed from the earlier research instead of a theory.

The method section has some problems. They did not provide any information about participants, such as gender and age. The only information about the participants was that students were taking that course either as a major or minor student. So, it is not clear whether or not they collected demographic information of students. The demographic features of participants would be valuable to help us to understand if badges are equally effective on students’ behaviors for different genders and age groups. In the rest of the method section, they successfully explained in details how they designed different badge categories and how the study was conducted. 

The result section was well written and their representation of the findings with graphs and figures helped me to understand the findings easily. They used non-parametric test to analyze the results and they needed to normalized the data of awarded badges, however they explained the reasons behind them successfully. They reported cumulative results for all students without differentiating students as major and minor students, as they were interested in overall effects of badges on student behaviors. However, I think that this distinction could be important to understand the effects of badges on behaviors, because the exercise scores formed 20% of the final course grade for major students and 30% for minor students. Although awarded badges did not affect the course grade, the different weight of exercise points would affect students’ attitudes toward completing exercises. Furthermore, in order to calculate the total time spend per session for the carefulness badges, the authors had to estimate the used time from the intervals between submissions at log data of TRACKLA2. Hence, they could not estimate the time used for the first session. In addition, the authors could not confirm if students spent that amount of time at TRACKLA2 by working in the exercises or doing unrelated things. They pointed out this problem as a validity treat and I think that the uncertainty caused by estimation method was a big validity issue for that part of the data. The data of likert scale and open-ended questions of the final survey were included at the end of the result section. The authors provided detailed explanations for the answers of open-ended questions with typical student answers and this was especially beneficial for the readers to see real students’ reflections about the badges. An interesting point for the open-ended questions that some control group students also answered the questions about the badges, although those questions were addressed to the treatment group. It was not clear in the article how this happened but it arises some concerns about the contamination problems. I think issue of contamination could also be possible, because authors stated that they had to openly answer the questions of students about badges in the class, which may let control group students aware of and learn about badges in TRACKLA2. Control group students might have learned about badge criteria from the treatment group students and changed their behaviors accordingly, which might change the baseline.

The discussion section was well written. They connected their arguments in the introduction section with those in discussion section with adding their findings. They also elaborated on the importance of carefully creating badge criteria to award badges. The authors accepted that some badges in the time management and carefulness categories might have caused conflicting interest in this study, so that students might have chosen being fast or being careful over another. This point should be most important practical implication of this research; that is, designers and instructors should be cautious about the badge criteria to award badges. They should set criteria for each badge challenging enough to motivate behavior change and badges should be perceived as informational instead of controlling to increase intrinsic motivation, as it is suggested by the Cognitive Evaluation Theory. Another important implication could be potential use of badges to decrease bad study habits of students such as procrastination, as the authors claimed badges could be an easy and efficient way for educators to prevent these undesirable study practices.