Clustering to define interview participants for analyzing student feedback: a case of Legends of Learning
Ayaz Karimov
Faculty of Information Technology
University of Jyväskylä
akarimov@jyu.fi
Mirka Saarela
Faculty of Information Technology
University of Jyväskylä
mirka.saarela@jyu.fi
Tommi Kärkkäinen
Faculty of Information Technology
University of Jyväskylä
tommi.karkkainen@jyu.fi

ABSTRACT

Within the last decade, different educational data mining techniques, particularly quantitative methods such as clustering, and regression analysis are widely used to analyze the data from educational games. In this research, we implemented a quantitative data mining technique (clustering) to further investigate students’ feedback. Students played educational games within a week on the educational games platform, Legends of Learning and after a week, we asked them to fulfill the feedback survey about their feelings on the use of this platform. To analyze the collected data from students, firstly, we prepared clusters and selected one prototype student closest to the centroid of each cluster to interview. Interviews were held to explain the clusters more and due to time and resource limitations, we were unable to interview all (N=60) students, thus only the most representative students were interviewed. In addition to the students, we conducted an interview with the teacher as well to get her detailed feedback and observations on the usage of educational games. We also asked students to take an exam before and after the research to see the impact of games on their grades. Our results depict that though educational games can increase students’ motivation, they may negatively impact some students’ grades. And even though playing games made students feel interested and fun, they would not like to play them on a daily basis. Hence, using educational games for a certain duration such as subject revision weeks may positively influence students’ grades and motivation.

Keywords

clustering, educational games, educational technology, serious game analytics, legends of learning

1. INTRODUCTION

Serious games or educational games can be defined as games that have the primary purpose of learning and education rather than entertainment. Educational games are special kinds of games that particularly aim to reach another outcome in addition to entertaining players. These games are used as a tool to increase the motivation and attention of students. In addition to the positive impact of educational games on students’ attitudes, they can also directly help learners to increase their grades [31, 46]. However, it is important to mention that if not developed and implemented correctly, educational games can also negatively interfere with the learning outcome [22].

The implementation fields of educational games vary a lot. For example, while educational games were used in social science education such as English language education [1], history education [25], they were also widely used in more technical fields as well such as biology education [10], computer science education [30]. Particularly, within the teaching of complex subjects, educational games can help both teachers and students to ease the learning process. These games have the potential to get the attention of learners for a longer period compared to traditional lectures and by using different game elements, they encourage players to continue studying. [47] carried out the research about the systematic literature review on the use of educational games. Based on their analysis of published papers between 2009 and 2018, many factors can influence successful educational game usage. Gaming easiness, backstory, and production can be examples of these factors. Furthermore, in this research, we used the digital educational games platform called “Legends of Learning”. Legends of Learning 1 is an online educational game platform that offers over 2,000 math and science games. Inside the platform, there are various games for each subject and class. Teachers create playlists of games based on standards and students work through completing each one.

The aim of this research was twofold:

2. LITERATURE REVIEW

2.1 Educational games in science education

Much research has been conducted on the usage of physical educational games in science education [27, 39, 8]. [23] developed an educational card game, then measured its impact on students’ performance. They found that there is a significant difference between students’ test results before and after playing this card game. Students also provided very satisfactory feedback to the researchers (4.29 over 5.00). Moreover, [6] found that using physical educational card games can enhance the educational experience of pharmacy students. [6] asked students to answer 90 questions before starting to play games and they played each game for 1 hour, 3 times over a 6-week period. The main subject of the games was cardiology and infectious diseases and students improved their assessment scores significantly (19.2% vs. 5.1%, (p < 0.001) and 10.3% vs. 5.1% (p = 0.006). In addition to increasing their grades, students also mentioned that they would like to play these card games in the future. Furthermore, [12] and [32] investigated educational board games. [12] used a board game called “Gut Check" where players try to develop a healthy microbiome for themselves and they also disrupt opponents’ efforts. In the development of this game, the researchers worked with gamers and biologists to develop both educational and entertaining games. While [12] only focused on the educational board game, [32] both created a new educational board game whose structure of spatial relationships mirrored the structure of rational numbers and to measure the impact of the game, they implemented pretest-posttest assessments. [32] found that the correlation between posttest and pretest scores was not statistically significant, r = .11, t(36) = 0.63, p = .531. The baseline knowledge of their participants did not influence the estimated normalized knowledge gains. Even though these papers build a strong understanding and impact of using education in science education, they focus on physical educational games. In our research, we focused on the impact of digital educational games.

Some researchers investigated digital educational games and their impact on students’ learning [42, 40, 5, 4]. [3] investigated the impact of using video gaming technology on middle school students learning within the scope of basic electromagnetism. For this, they used the game called “Supercharged!" which is a 3D action/racing game. In this game, players try to maneuver through a set of obstacles to obtain a certain goal. Supercharged! is designed based on the laws of electrostatics and the game helps the players to build stronger intuitions about how charged particles interact with electric and magnetic fields. [3] divided participants into two categories: experiment and control. Experiment group members learned the given physics subjects by playing games on Supercharged!, however, control group members learned the same physics subject by using the traditional learning method. The researchers found that the experiment group outperformed the control group and there was no significant difference from the perspective of gender. The researchers also asked open-ended questions to students and students mentioned that talking in the classroom during the learning process is not familiar to them, thus in the beginning it was challenging for some students to adapt. Additionally, based on their findings, digital educational games do not replace instruction, but they can support teaching. Moreover, [44] and [10] focused on the utilization of digital educational games in chemistry and biology education respectively. Both of these authors implemented pretest and posttest research methods where they divided students into experiment and control groups and asked students to take a test before and after playing digital educational games. [44] found that compared to the traditional teaching approach, the game-based learning approaches depicted better effects. Additionally, they also found that students are prone to have higher self-efficacy than those in a traditional lecture class when learning science. [10] found similar positive results from their research that there was a significant improvement in the overall learning achievement of students after playing digital games. Hereby, there was detailed research about the implementation of digital educational games in science education, nonetheless, we could not find research carried out in Azerbaijan. In our research, we focused on the usage of digital educational games in a school located in Azerbaijan.

Furthermore, in some cases, the implementation of educational games is not successful [47]. For instance, [13] and [15] found that educational games have a negative influence on the relationship between mental workload and learning effect. Additionally, [14] also investigated the digital educational games and they found that there were no significant differences in in-depth learning among learners.

2.2 Research on Legends of Learning

In this research, we used Legends of Learning as a digital educational game platform and we also did a literature review of the research carried out about the platform. [18] used the Legends of Learning platform to see how the platform impacts students’ knowledge of the physics of light. 50 8th-grade students participated in the research and the pretest-posttest method was implemented. They found that there was a medium development for the concept mastery enhancement and student curiosity enhancement has shown a negative impact. Furthermore, the Legends of Learning team partnered up with the team from Vanderbilt University and they investigated the usage of educational games in the classroom [11]. They found that the students who used educational games as part of their regular curricula perform better than their peers on both factual knowledge and depth of knowledge. Based on our search, Legends of Learning was not utilized as a digital educational game platform in Azerbaijan, and in our research, we used this platform to see its impact on students’ learning and motivation in a school located in Azerbaijan.

2.3 Educational games in Azerbaijan

In Azerbaijan, educational games and gamified apps are not familiar to the local market and the market lacks localized or translated international educational games platforms [26]. In [26], the researchers designed and developed an educational game called “FunMath" in which players need to solve mathematical problems to advance their scores. The app is designed from scratch in collaboration with the teachers and students. And after doing the usability test and final interview with the school teacher, they found that even though there could be more improvements, FunMath can extensively be used within and outside of school to increase learners’ motivation.

[43] investigated how gamified environment can impact learners’ motivation and math abilities. For this, they utilized the platform called “Polyup". Polyup is a computational thinking playground where students can experiment with numbers and functions. After playing games, they asked students to provide feedback by fulfilling the survey. Students mentioned that the platform can be used to connect mathematics with real-life experiences as well as can also enhance their mathematical calculation skills. Furthermore, based on our research, apart from [26] and [43], we could not find research focusing on educational games or gamified platforms in Azerbaijan. Additionally, both of these papers were focused on researching educational games within the scope of math education, and in our research, we focus on the impact of educational games in science education.

2.4 Educational games analytics by using clustering algorithms

Educational or serious games analytics refers to analytics or insights converted from gameplay data within educational games for the aim of performance measurement, assessment, or improvement [28]. Different supervised and unsupervised data mining methods can be implemented to analyze data from educational games [19]. Clustering is one of the unsupervised techniques of data mining and it contains various algorithms such as K-means, hierarchical clustering, and expectation maximization [38]. [2] did the systematic literature review study on the applications of data science techniques to analyze data from educational games. In their research, [2] found that 16 of the total 87 academic papers utilized clustering as a data mining technique which was one of the mostly-used methods along with linear regression and correlation analysis. And in this research, we also used the K-means algorithm to develop clusters.

[29] proposed and validated the questionnaire which is an instrument to measure the game preferences and habits of an intended audience. This instrument is called “The Game Preferences Questionnaire (GPQ) which possesses 10 Likert-scale items and produces a classification of the participants into four discrete clusters. While creating these clusters, [29] utilized the K-means algorithm and they found four main clusters: casual players, no gamers, well-rounded gamers, and hardcore gamers. Furthermore, [33] implemented K-means clustering, K-means++ initialization, and CVIs algorithms to investigate the creation of the new clustering-based profiling method. For this, they used the platform called “GraphoLearn" where users can play games for learning to read. [33] found that by utilizing their clustering method it is possible to cluster various kinds of learners and the method can help to track students who have reading disabilities. Additionally, [9] also used the K-means algorithm to analyze the measure of educational games on students. They used the platform called “OMEGA+" and in that platform, learners can play games to enhance their knowledge of problem-solving, planning and organization, associative reasoning, and accuracy and evaluation. After the implementation of the k-means algorithm, they found four clusters based on their activity status. [9] also mentioned that female players do not benefit from educational games due to their low activity status. In our research, in addition to the clustering analysis and interpretation of clusters, we utilized clustering results to select the students for the qualitative data collection. From this perspective, our paper brings novelty since it is a natural combination of qualitative and quantitative data analysis rather than only utilizing quantitative clustering data for quantitative data analysis purposes.

In addition to the K-means, other clustering algorithms were also used to analyze data from educational games. For instance, [17] used the density-based spatial clustering of applications with noise (DBSCAN) algorithm to research students’ behavior in educational games. Based on the DBSCAN, [17] propose a new method called “SPRING" that helps student profile modeling in educational games. Furthermore, [24] focused on the analysis of player strategies in educational games, and for this, they utilized hierarchical clustering. Within this study, they used “GrAZE" which is a puzzle-based game where learners can improve their algorithmic thinking by playing this game. [24] highlighted problem areas that can be fixed in the early design phase.

3. METHODOLOGY

We implemented a mixed methodology: quantitative (surveys) and qualitative (interviews). To measure the change in students’ learning, we implemented the pretest-posttest method where we asked students to take a test before and after playing games. Additionally, we also asked students to fulfill a survey which helped us to analyze their feedback and define clusters according to their background and test results. Furthermore, we selected one prototype student closest to the centroid of each cluster to interview (Figure 1).

3.1 Research set up and participants

Before starting the research process, we prepared agreements to be signed by the school principal and students’ parents. All the permissions were collected from the principal and parents two weeks prior to the research. Students whose parents did not sign the agreement also played games but their data was not collected in any form. In parallel with the collection of permission forms, the teacher selected educational games from the platform for each class (Appendix C). On Legends of Learning, there are many games and each game has a very different learning goal, thus it was necessary to define the games for each class in advance. Moreover, we also communicated with all parents and school administrators to ensure that there is an internet connection in each classroom and students have their devices with them.

We recruited students who study in the 5th, 6th, or 7th grades(5th-grade students = 43.3%, 6th-grade students = 25%, 7th-grade students = 31.7%). We gathered data from 67 students, nevertheless, since 7 students could not take the test after playing educational games, we removed them from the dataset. Exactly 50% of students mentioned that they identify themselves as “male”, and the remaining 50% selected the “female” option.

Research process
Figure 1: Research process

3.2 Legends of Learning platform

Legends of Learning contain many various games inside the platform and the type of the game type can differ from memory type to matching games. Thus, it totally depends on the learning goal of the instructor and what kinds of knowledge they want to deliver. In our research, since the teacher had prior experience using the platform in the classroom, she selected games (Appendix C). The selected games can mainly be categorized under three sections: matching, memory, and video games. In the matching games, students were asked to match images with the given information, and in the memory games, students were asked to memorize some of the notions. And in the case of video games, students play a regular game while in the middle of the game, the game stops and it asks the question from the student in a quiz format. If students answer questions correctly, then they collect points. Furthermore, one of the interesting features of Legends of Learning is the “Awakening" section. After finishing their regular games, students can move to the Awakening part, a virtual world. During the Awakening section, students can walk, meet each other, and solve additional questions. There is no endpoint in the Awakening section, thus one can collect as many points as one can. However, since we selected games previously and we wanted all of the students to play all selected games, we put a time limit of five minutes for the Awakening section.

3.3 Data collection and analysis

In this research, we collected both quantitative and qualitative data 2. To collect quantitative data, we prepared the survey and arranged special dates for students to fulfill the survey at the school. To make sure that students understand how they need to fulfill the survey, the teacher explained each question to students while guiding them. In the development of the survey questions, we used closed-ended questions where participants choose one or more of the predetermined responses. The reason to select the closed-ended questions is that they are easier and faster to answer [45]. Considering the age of this study’s participants, for us, it was necessary to keep their attention while they were fulfilling the survey. The main aim of this survey was to collect students’ feedback. The feedback survey is designed in a five-item Likert scale ((1) Strongly disagree; (2) Disagree; (3) Neither agree nor disagree; (4) Agree; (5) Strongly agree). In the development of statements for the survey, we considered our main research focus such as what kind of information we would like students to provide, and additionally, we used [20] as a reference to select some of the statements. [20] developed these statements particularly to measure user feedback after using the e-learning games.

To measure the impact of educational games on the learning outcome, we implemented a pretest-posttest design where we asked students to take two tests: pregame and postgame tests [16]. The aim of asking students to take two tests was to investigate how students’ grades changed before and after playing educational games. The difficulty of questions in both tests was very similar and the teacher prepared all questions. Moreover, on the first day of research, all students were asked to take a test (pregame test) for 20 minutes, and on the last of the research, students were again asked to take another test (postgame test). For the postgame test, students were also given 20 minutes to finish it. Furthermore, we wanted to measure whether the difference between pregame and postgame scores is significant or not. Since we had only two variables (pregame and postgame score), we conducted a t-test.

To create clusters, we implemented the K-means algorithm and we preprocessed the data before starting the algorithm implementation phase. Initially, since we did not have any missing values, we moved to the data transformation stage where we converted categorical variables into binary or numerical variables. As a next step, we standardized data by using MinMaxScaler. Within the implementation of the K-means algorithm, one of the important phases is to define the optimal number of clusters (k) and for this, we used the “Elbow method". When we visualized the graph we observed that the graph rapidly changed at a point and this happened when the number of K was 4. Furthermore, for clustering data, we used the following variables: students’ responses to all questions in the feedback survey, pretest-posttest scores, gender, previous gaming experience, and grade that they study.

To collect qualitative data, we held semi-structured interviews. The main goal of holding these interviews was to collect detailed feedback from students and teachers about their experience using the platform. The reason why we selected semi-structured interviews was that we wanted to ask certain questions but we also wanted to investigate students’ additional thoughts. While holding semi-structured interviews, it is possible to direct the interview based on participants’ responses [35]. Furthermore, once we had the results from the clustering analysis, we selected one person from each cluster (the closest prototype to the centroid) because this student prototype would be the best representation of the cluster that they belong to. All interviews were held in the school based on the availability of the selected students and teachers. Interviews were recorded and transcribed in the Azerbaijani language, however, for this academic research, the main outputs were translated from Azerbaijani into English. The parents of the interview participants signed the agreement to give permission before we start interviewing. To analyze interview data, we implemented the coding scheme methodology [7].

Table 1: Overview of clusters
Cluster ID Cluster description NGender (female vs male)Gaming experience (yes vs no)
cluster_0 Lowly motivated and positive-grade-change students 8 37.5% vs 62.5% 85.7% vs 14.3%
cluster_1 Average motivated and negative-grade-change students28 50% vs 50% 85.7% vs 14.3%
cluster_2 Average motivated and no-grade-change students 16 43.8% vs 56.2% 75% vs 25%
cluster_3 Highly motivated and high-positive-grade students 9 67% vs 33% 100% vs 0%
Change in the grades of each cluster on average
Figure 2: Change in the grades of each cluster on average

Responses of each cluster to the survey questions
Figure 3: Responses of each cluster to the survey questions

4. RESULTS

4.1 Survey results

In the survey, students provided feedback by answering questions where the maximum score was 5 and the minimum score was 1. When we asked students about their feeling while they revise the subject by playing games, they mentioned that they felt more satisfied (4.07) and more motivated (3.67). Additionally, students gave a score of 4.22 for their experience from the perspective of enjoyment. It depicts that students did not feel bored or anxious while playing these games. Subsequently, students mentioned that they would like to revise the study subjects by playing educational games in the future as well (4.32). However, students gave a score of 3.65 to the statement about improvement in their knowledge. To sum up, even though students provided high scores to the statements about their feelings and willingness to use educational games in the future, their scores for knowledge did not improve significantly.

4.2 Student profiles based on clustering results

According to the clustering results, we made four different student profiles (Table 1) and they are as follows:

  1. Highly motivated and high-positive-grade students: Students in this cluster changed their grades by 4.71 points on average (Figure 2). They were satisfied with their experience on the platform and according to them, educational games helped them to increase their knowledge. Lastly, these students enjoyed the games most.
  2. Average motivated and no-grade-change students: These students felt less motivated and they felt bored more at some parts of the games compared to the previous cluster of students. Moreover, there was no change in the grades of these students after playing educational games.
  3. Average motivated and negative-grade-change students: The main characteristics of these cluster students look alike with the second cluster students from the perspective of how they feel using educational games. The main difference between the second and this cluster was their grade change. Students in this cluster decreased their grades by 4 points on average. This was the only cluster that depicted a negative change in their grades after playing educational games.
  4. Lowly motivated and positive-grade-change students: Students in this cluster felt less motivated and they were not satisfied in comparison with their peers. Even though they provided lower points for their feelings about using the platform, these students made a positive change in their grades by 1.3 points on average.

When we ran a t-test to measure the significance of the change, we found that there is no sufficient evidence to say that the average grade of students before and after using educational games is different (p-value = 0.7746, alpha = 0.05). Moreover, we also analyzed the average score of each cluster to the feedback survey where “5" means that they strongly agree with the statement and “1" signifies that they strongly disagree with the statement. Overall, cluster_3 provided very positive feedback from any perspective such as being motivated to use educational games in the future and feeling positive and joyful while playing games. Cluster_0 provided lower scores compared to other clusters on average and the scores from cluster_1 and cluster_2 were almost the same. Nevertheless, almost all clusters (except cluster_2) gave the lowest score to the statement about the increase in their knowledge (“The games increased my knowledge.").

4.3 Students’ and teacher’s feedback on the platform

To collect detailed feedback, we selected one prototype student closest to the centroid of each cluster. During the interviews, students provided their feedback about the platform and their educational games experience (Appendix A). Most of the students highlighted positive points about the usage of educational games and they also emphasized that Legends of Learning is a very user-friendly platform, thus it was a good experience to play games on this platform (cluster_3 and cluster_2). Moreover, all interviewed students mentioned that they own previous experience in using gamified educational tools or educational games. Students also provided feedback about the disadvantages of using educational games. Cluster_2 and cluster_3 prototype students mentioned that it is challenging for them to learn new knowledge on the platform. Even though they encountered new notions and terms, after playing games, they could not remember most of them. Furthermore, cluster_1 prototype student noted that in some parts of the game, answering questions correctly did not influence their game performance, so they were trying to click any buttons so that they can move to the next stage faster. Lastly, cluster_0 prototype students mentioned that some types of games such as memory games even decreased their motivation to continue.

In addition to students, we also conducted an interview with the teacher to understand students’ attitudes, her feedback on teaching through educational games, and the challenges that she encountered during the research week (Appendix B). Overall, the teacher mentioned positive feedback on the use of educational games since she observed a positive change in students’ behavior. She highlighted that students collaborated with one another to explain some problems and it increased the engagement inside the classroom. Particularly, after finishing the research, some inactive students started to participate in the classroom more actively and she thought that games helped them to see science from a different perspective. The teacher also mentioned some challenges and negative feedback that she observed and heard from students. According to her, some students did not have any prior knowledge of playing games or even using computers, thus adoption of the educational games took longer time compared to their peers. Moreover, even though playing educational games was fun, students were not interested in playing them on a daily basis.

5. DISCUSSION AND CONCLUSION

The objective of the present study was to investigate the impact of utilizing educational games as a tool to revise science subjects. In the first step, we asked students to take a test before starting to play educational games. After taking a pregame test, they played games on the platform called “Legends of Learning" for a week. On the last day of the week, students took a postgame test. Pregame and postgame tests helped us to measure the change in students’ grades. Subsequently, students fulfilled a feedback survey where we asked them to rate statements about their feeling about using educational games. Based on the data from the feedback survey and their pregame and postgame test scores, we created clusters by using the K-means clustering algorithm. Then, we selected one prototype student from each cluster to get detailed feedback on their experience of using educational games. As a last step, we did an interview with a teacher to understand her perspective on teaching through educational games and students’ behavior change based on her observation.

[34] and [21] highlight that players with prior gaming experience positively impact their performance in other games. In our research, we also found a similar relationship between students’ prior gaming experience and their attitudes. Most of the students had prior experience playing educational games or using gamified platforms and all students mentioned that playing games in the classroom was entertaining (cluster_0), exciting (cluster_1), fun (cluster_2), and interesting (cluster_3). [3] mentioned that educational games cannot replace traditional instructions, however, they can support learning. Based on the interviews with students, we can also see similar answers where they highlighted encountering difficulties to learn a new subject by only playing games. All students and the teacher mentioned that they would prefer playing games only during revision weeks since regular lectures help them to learn more effectively. Additionally, as [41] found in their papers, the teacher also mentioned that games motivated introverted and passive students and they started participating in classroom discussions.

There are limitations to this work that should be noted. In this research, we used one digital educational games platform and within this study, we focused on elementary school students which restricted the scope of the research. Due to our resources, it was only possible to hold this research with a group of students, and the implementation of one platform was possible. Moreover, there is a wide range of future work that we want to address based on the results presented in this paper. Firstly, the participant profile can be changed to see whether it affects their motivation and grades differently. For instance, holding this research with primary and secondary school students can result in different outcomes. Secondly, instead of utilizing Legends of Learning, another digital educational games platform can be used. Because each platform is different and may bring different results. Last not but least, the games inside the Legends of Learning can be further studied.

5.1 Ethical concerns

To maintain participants’ confidentiality they were assigned a number rather than their name, and data were stored and will be disposed of securely according to the agreement that parents signed. Students were also given the right to withdraw whenever they want to stop and leave at any point in the study.

6. ACKNOWLEDGEMENT

We would like to acknowledge the Science teacher, Sabina Mammadova, and the administration of the IB MYP Programme of Educational Complex 132-134 for their invaluable support in data collection and setting up the research. Their assistance was essential to the successful completion of this study, and we are grateful for their dedication to advancing the field of education.

7. REFERENCES

  1. J. M. Al-Jarrah, O. T. Waari, R. H. Talafhah, and T. M. Al-Jarrah. Improving english grammar achievement through educational games among eleventh grade students in east jerusalem. International Journal of Academic Research in Progressive Education and Development, 8(1):75–86, 2019.
  2. C. Alonso-Fernández, A. Calvo-Morata, M. Freire, I. Martínez-Ortiz, and B. Fernández-Manjón. Applications of data science to game learning analytics data: A systematic literature review. Computers & Education, 141:103612, 2019.
  3. J. L. Anderson and M. Barnett. Learning physics with digital game simulations in middle school science. Journal of science education and technology, 22(6):914–926, 2013.
  4. J. W. J. Ang, Y. N. A. Ng, and R. S. Liew. Physical and digital educational escape room for teaching chemical bonding. Journal of Chemical Education, 97(9):2849–2856, 2020.
  5. L. A. Annetta, M.-T. Cheng, and S. Holmes. Assessing twenty-first century skills through a teacher created video game for high school biology students. Research in Science & Technological Education, 28(2):101–114, 2010.
  6. S. M. Barclay, M. N. Jeffres, and R. Bhakta. Educational card games to teach pharmacotherapeutics in an advanced pharmacy practice experience. American Journal of Pharmaceutical Education, 75(2):33, 2011.
  7. W. Barendregt and M. M. Bekker. Developing a coding scheme for detecting usability and fun problems in computer games for young children. Behavior research methods, 38(3):382–389, 2006.
  8. Y. Bouzid, M. A. Khenissi, F. Essalmi, and M. Jemni. Using educational games for sign language learning-a signwriting learning game: Case study. Journal of Educational Technology & Society, 19(1):129–141, 2016.
  9. D. Chandrasekaran, M. Chang, and S. Graf. A learning analytics approach to build learner profiles within the educational game omega+. In International conference on intelligent tutoring systems, pages 139–147. Springer, 2022.
  10. M.-T. Cheng, T. Su, W.-Y. Huang, and J.-H. Chen. An educational game for learning human immunology: What do students learn and how do they perceive? British Journal of Educational Technology, 45(5):820–833, 2014.
  11. D. B. Clark, E. Tanner-Smith, A. Hostetler, A. Fradkin, and V. Polikov. Substantial integration of typical educational games into extended curricula. Journal of the Learning Sciences, 27(2):265–318, 2018.
  12. D. A. Coil, C. L. Ettinger, and J. A. Eisen. Gut check: The evolution of an educational board game. PLoS Biology, 15(4):e2001984, 2017.
  13. B. Cowley, M. Fantato, C. Jennett, M. Ruskov, and N. Ravaja. Learning when serious: Psychophysiological evaluation of a technology-enhanced learning game. Journal of Educational Technology & Society, 17(1):3–16, 2014.
  14. B. Cowley, T. Heikura, and N. Ravaja. Learning loops–interactions between guided reflection and experience-based learning in a serious game activity. Journal of Computer Assisted Learning, 29(4):348–370, 2013.
  15. B. Cowley, N. Ravaja, and T. Heikura. Cardiovascular physiology predicts learning effects in a serious game activity. Computers & Education, 60(1):299–309, 2013.
  16. D. M. Dimitrov and P. D. Rumrill Jr. Pretest-posttest designs and measurement of change. Work, 20(2):159–165, 2003.
  17. M. H. Falakmasir, J. P. González-Brenes, G. J. Gordon, and K. E. DiCerbo. A data-driven approach for inferring student proficiency from game activity logs. In Proceedings of the third (2016) acm conference on learning@ scale, pages 341–349, 2016.
  18. A. F. N. Fikriah. Addressing Concept Mastery and Curiosity about the Physics of Light in Middle School Students through Discovery Learning with “Legends of Learning” Educational Games. PhD thesis, Universitas Pendidikan Indonesia, 2021.
  19. M. Freire, A. Serrano-Laguna, B. Manero, I. Martinez-Ortiz, P. Moreno-Ger, and B. Fernandez-Manjon. Game Learning Analytics: Learning Analytics for Serious Games, pages 1–29. Springer Nature Switzerland AG, Switzerland, Apr. 2016.
  20. F.-L. Fu, R.-C. Su, and S.-C. Yu. Egameflow: A scale to measure learners’ enjoyment of e-learning games. Computers & Education, 52(1):101–112, 2009.
  21. V. Garneli, M. Giannakos, and K. Chorianopoulos. Serious games as a malleable learning medium: The effects of narrative, gameplay, and making on students’ performance and attitudes. British Journal of Educational Technology, 48(3):842–859, 2017.
  22. M. Gaydos. Seriously considering design in educational games. Educational Researcher, 44(9):478–483, 2015.
  23. A. F. Gutierrez. Development and effectiveness of an educational card game as supplementary material in understanding selected topics in biology. CBE—Life Sciences Education, 13(1):76–82, 2014.
  24. B. Horn, A. K. Hoover, J. Barnes, Y. Folajimi, G. Smith, and C. Harteveld. Opening the black box of play: Strategy analysis of an educational game. In Proceedings of the 2016 Annual Symposium on Computer-Human Interaction in Play, pages 142–153, 2016.
  25. J. Huizenga, W. Admiraal, G. Ten Dam, and J. Voogt. Mobile game-based learning in secondary education: Students’ immersion, game activities, team performance and learning outcomes. Computers in Human Behavior, 99:137–143, 2019.
  26. S. Huseynli. Application of games in teaching mathematics at primary and preschools. 2021.
  27. G. Jin, M. Tu, T.-H. Kim, J. Heffron, and J. White. Game based cybersecurity training for high school students. In Proceedings of the 49th ACM Technical Symposium on Computer Science Education, pages 68–73, 2018.
  28. J. Kang, M. Liu, and W. Qu. Using gameplay data to examine learning behavior patterns in a serious game. Computers in Human Behavior, 72:757–770, 2017.
  29. B. Manero, J. Torrente, M. Freire, and B. Fernández-Manjón. An instrument to build a gamer clustering framework according to gaming preferences and habits. Computers in Human Behavior, 62:353–363, 2016.
  30. R. Mathew, S. I. Malik, and R. M. Tawafak. Teaching problem solving skills using an educational game in a computer programming course. Informatics in education, 18(2):359–373, 2019.
  31. U. Munz, P. Schumm, A. Wiesebrock, and F. Allgower. Motivation and learning progress through educational games. IEEE Transactions on Industrial Electronics, 54(6):3141–3144, 2007.
  32. J. A. Navarrete-Ulloa and F. Munoz-Rubke. Playing board games to learn rational numbers: A proof-of-concept. Mind, Brain, and Education, 16(4):293–299, 2022.
  33. M. Niemelä, T. Kärkkäinen, S. Äyrämö, M. Ronimus, U. Richardson, and H. Lyytinen. Game learning analytics for understanding reading skills in transparent writing system. British Journal of Educational Technology, 51(6):2376–2390, 2020.
  34. C. O’Donovan and J. Hussey. Active video games as a form of exercise and the effect of gaming experience: a preliminary study in healthy young adults. Physiotherapy, 98(3):205–210, 2012.
  35. S. E. Rabionet. How i learned to design and conduct semi-structured interviews: an ongoing and continuous journey. Qualitative Report, 16(2):563–566, 2011.
  36. M. Saarela, V. Heilala, P. Jääskelä, A. Rantakaulio, and T. Kärkkäinen. Explainable student agency analytics. IEEE Access, 9:137444–137459, 2021.
  37. M. Saarela, J. Hämäläinen, and T. Kärkkäinen. Feature ranking of large, robust, and weighted clustering result. Lecture Notes in Computer Science, 10234:96–109, 2017.
  38. M. Saarela and T. Kärkkäinen. Knowledge discovery from the programme for international student assessment. In Learning Analytics: Fundaments, Applications, and Trends, Studies in systems, decision and control, pages 229–267. Springer International Publishing, Cham, 2017.
  39. N. R. Scalise, E. N. Daubert, and G. B. Ramani. Benefits of playing numerical card games on head start children’s mathematical skills. The Journal of Experimental Education, 88(2):200–220, 2020.
  40. P. Sengupta, K. D. Krinks, and D. B. Clark. Learning to deflect: Conceptual change in physics during digital game play. The Journal of the Learning Sciences, 24(4):638–674, 2015.
  41. R. Smiderle, S. J. Rigo, L. B. Marques, J. A. Peçanha de Miranda Coelho, and P. A. Jaques. The impact of gamification on students’ learning, engagement and behavior based on their personality traits. Smart Learning Environments, 7(1):1–11, 2020.
  42. K. Squire, M. Barnett, J. M. Grant, and T. Higginbotham. Electromagnetism supercharged! learning physics with digital simulation games. 2004.
  43. Y. Tabesh, S. Zarkesh, A. Zarkesh, and I. Fazilova. Computational thinking in k-12: Azerbaijan’s experience. Olympiads in Informatics, 13:217–224, 07 2019.
  44. M. Wang and X. Zheng. Using game-based learning to support learning science: A study with middle school students. The Asia-Pacific Education Researcher, 30(2):167–176, 2021.
  45. S. Yaddanapudi and L. Yaddanapudi. How to design a questionnaire. Indian journal of anaesthesia, 63(5):335–337, 2019.
  46. Z. Yu, M. Gao, and L. Wang. The effect of educational games on learning outcomes, student motivation, engagement and satisfaction. Journal of Educational Computing Research, 59(3):522–546, 2021.
  47. Y. Zhonggen. A meta-analysis of use of serious games in education over a decade. International Journal of Computer Games Technology, 2019:1–8, 2019.

APPENDIX

A. STUDENTS’ FEEDBACK

Cluster_0 student:

Since the teacher does not explain anything, it is very hard to actually understand anything from scratch. Because in the classroom, our teacher explains everything many times and very easily but in the games, they just provide information and the explanation is not detailed. So maybe I would not like to play games to learn something from scratch. And I was not sure whether I am learning new things or not and that was bad. I would not like to play games daily because I would feel bored after some time but for the revision week, it would be amazing to play games always. I liked playing games because I felt more entertained in the classroom. I play these kinds of games very often at my tutoring sessions and it always makes me want to play more. The games on Legends of Learning were okay and I did not feel that bored except for memory games. Because in the memory games, I struggled a lot and I could not find many hints to use. But in general, it was good and exciting to play these games in the classroom.

Cluster_1:

It was neither good nor bad, it was fun to play games but after a certain point, I felt a bit bored. At the beginning of each class day, I was excited to play games and I think some of the games are not as fun as others. But I think the platform is good because it does not have many bugs because most of the games that I play at home have bugs or they work very slowly. I especially liked the Awakening section of the platform. Because I liked that there are no limits and I could do many things. Actually, at that point, I understood that I can just keep playing even if I answer the question correctly or wrongly. And I also remember that I can skip some points in which I could learn something. So it was a bit bad that sometimes I did not feel I am learning anything I was only playing games and ignoring the learning part. However, in the classroom, our teacher generally asks random questions in the classroom and I try to follow her so that I can answer her question. I think we can use games only for the revision week and it would not be good to use them every day.

Cluster_2:

I liked playing games during the revision week, it was very fun. In some parts of the game, I was asked to answer some questions and in these parts, I memorized what I learned previously in the class so it was very good to revise previous subjects. I also learned new information on the platform. Maybe we learned this in the classroom previously, I learned some of them in the games. Normally, I struggle a lot to sit and try to read something but by playing games, it felt more like free-time activity rather than really studying something. Even I was very excited to be in Science class. I previously played some educational games such as Kahoot! and Bluekit but they were only in the French and Biology classes. In general, I like playing educational games. I think the biggest disadvantage is that it is very hard to learn something new by only playing games. For example, I learned some new things but I already forget most of them. Because they appeared only in one part of the game and I used them to move to the next sessions. But I think that we can play games two or three times a month about the subjects that we already learned at school.

Cluster_3:

It was so much fun and interesting to play these games. I try to find and play similar games at home as well and these ones were very entertaining. But at home, I start feeling bored after some time and the best thing was that I was playing games together with my friends at school. Honestly, it did not feel like a lesson at all. I think I also learned new things by playing games. It was also good to answer questions that I already learned at school. I think I collected many points because I was trying to remember and select the right answer to each question calmly. I liked games on Legends of Learning because they differ from one another a lot. For example, I did not like memory games and in these games, and I was trying to finish the memory games and move on to more interesting ones. I do not think it has a huge disadvantage because I really enjoyed but maybe it is only good for the revision week. Because I was mainly using my previous knowledge on the topic and that is why I managed to finish many games earlier than my friends. And I also want to say that I think some students were just clicking randomly just to continue and collect points in the knowledge-sharing parts.

B. NOTES FROM THE INTERVIEW WITH THE TEACHER

Teacher:

Students enjoyed the platform and the Awakening was the most interesting part of the section because they were able to see one another and interact. In general, they were more motivated when they were collaborating and talking to one another. Some students whom we can call “a gamer" were more prone to solve problems and finish problems very quickly. Once they finished their games, they were trying to help their friends by giving hints and explaining the platform. There was also one interesting situation with one of my students who was very introverted and silent during our casual classes. But during the research week, I saw that she was very active and a pioneer to finish games way earlier than their peers. So I think educational games directly impact gamers who are not much active in regular classes. One of the greatest things was that there were at least six students who changed their attitudes toward the Science class as well. Before this research, these six students were not active in the classroom, but even after the research week, I can clearly see that they have more interest in Science class. I think they saw the different perspectives of Science and they really liked it. Students understand that the games are fun but they prefer the teacher to explain something and learn before. Because in some games, they particularly said that they did not learn this topic in the classroom so they could not pass to the next level. So I think these kinds of educational games are only good to revise some subjects. I also saw some students that were trying to skip the instructions part and maybe they managed to get some time in the beginning, but once they moved to the games part, they struggled a lot. Even some needed to go back and read the instructions again.

C. SELECTED EDUCATIONAL GAMES

The games for the 5th-grade students:

The games for the 6th-grade students:

The games for the 7th-grade students:

1www.legendsoflearning.com

2The datasets generated during and/or analyzed during the current study are available from the corresponding author on request.

© 2023 Copyright is held by the author(s). This work is distributed under the Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International (CC BY-NC-ND 4.0) license.