ABSTRACT

Even though Azerbaijan is considered a highly educated country from the perspective of schooling years and completed education level, student learning outcomes are underperforming, according to the World Bank. Due to limited resources such as classroom size, access to world-class educational materials, and high-qualified teachers, particularly students from under-resourced communities encounter more challenges during their education life compared to other students who possess these resources. Moreover, online educational platforms play an important role in eliminating learning gaps, particularly in developing countries such as Azerbaijan. In this paper, we describe the implementation and impact of utilizing an online educational platform, the Khan Academy, in one of the under-resourced communities of a developing country. For this, we collaborated with a school in Azerbaijan located in a suburban area. After collecting data through surveys, we applied the association rule mining method. Results from association rule mining concluded that students who studied using the online platform improved their grades and the gamification features of the Khan Academy motivated them. Furthermore, even though it was the first time the school used an online educational platform, almost all students mentioned they would like to learn with these resources in the future. Our study, thus, contributes to how online educational technologies can positively impact the motivation and learning outcomes of students in under-resourced communities.

Keywords

1. INTRODUCTION

Online learning refers to learning and other supportive resources that are available through a computer. The digital spaces where online learning happens are called online educational platforms (OEPs). OEPs generally contain educational content in different formats such as videos and articles. In some cases, OEPs can also analyze the learning of students based on their interaction inside the platform and provide feedback to improve their learning outcomes [7]. During COVID-19, OEPs played an important role in softening the negative impact of the pandemic on educational activities [2]. The utilization of the OEPs can bring opportunities for students who do not have access to high-quality education, and this can positively alter students’ attitudes toward the schools as well as the learning process [6]. After using the OEPs, students enhance their learning performance and become more motivated in the learning process since it guides them to have more meaningful learning behaviors [28]. Students’ attitudes towards the OEPs positively change due to various reasons such as being able to track the progress over learning duration, and the possibility of viewing the educational content anytime [4]. One of the reasons why students’ attitudes and motivations changed positively is because OEPs started using the gamification elements in their platform to increase engagement [46].

Gamification is the application of game design elements (badges, points, digital coins, etc.) in non-game contexts [15]. [35] and [3] researched the effect of gamification on the motivation of students. They found that gamification motivates students to attempt harder tasks and develop the information literacy skills necessary for success. Moreover, previous studies also found that the utilization of gamification within the learning process can also bring cognitive outcomes. For instance, [30] found that gamification positively affects student retention. They also found that gamification contributes positively to the growth of learners’ attitudes and interests at schools. In the research of [19], they designed a gamification plugin to collect students’ data and they found that students who completed the assignments in the gamified environments got higher scores. While they made the statement that gamification can possess an emotional and social impact on students that motivate them, gamification may not be the best way to increase motivation for all students. [19] highlighted that gamification environments can also discourage students if they do not acquire the goals within the gamified learning process. In addition to students’ motivation and engagement, gamification can also positively impact students’ grades [22]. [25] researched the impact of gamification on the students’ carefulness. They found that students indicate a higher level of carefulness when they perform their educational activities in the gamified environment and being more careful towards the assignments increased their grades.

This paper aims to investigate the impact of using OEPs in under-resourced communities and we used Khan Academy as an online educational platform. Khan Academy is one of the largest online educational platforms and it gamifies the learning process by adding gamification elements. Furthermore, our study adds to previous research from different perspectives. Firstly, previous research has been limited to exploring the implementation and impact of OEPs in other countries, and to our knowledge, no prior study has been carried out in Azerbaijan on the topic of the impact of OEPs and their gamification features on students’ learning. Secondly, even though the research on the usage of gamification and OEPs in under-resourced communities has been carried out, the number of participants in these studies was limited [1, 21, 50]. However, in our study, 207 students participated within 6 months. Thirdly, recent articles [36, 9, 34] investigated the utilization of Khan Academy and the impact of its gamification features on students’ learning. Our study fills the gap by focusing on primary school students in under-resourced communities and we measure how the gamification features impact students’ both motivation and grades.

2. LITERATURE REVIEW

2.1 Online educational platforms

Both students and teachers benefit from the OEPs from different perspectives. For teachers, functionalities of OEPs can help in analyzing the learning process and students’ learning outcomes in a detailed way [48] and they use OEPs for assigning additional exercises for students who would like to eliminate the learning gaps [33]. For students, OEPs offer the chance to study individually chosen topics and one of the significant advantages is to be able to replay the videos as much as they need which may not be possible at school. Additionally, some OEPs possess a complementary learning experience where the learner can do the follow-up exercises after watching videos or reading articles. Some research showed that using OEPs can positively impact students’ learning. [5] mentioned that OEPs improve the students’ ability to learn outside the classroom, and if combined effectively, then online and offline learning platforms can help students to understand the subject better. Furthermore, some of the researchers conducted research on the implementation of the OEPs to measure their impact on students’ learning. [27] investigated the effect of using online educational content for a month for a math class. They found that there is a positive correlation between the number of studied online educational content and students’ achievement. [10] examined the causal impact of online education on the academic performance of students. They found that online educational activities have a positive impact on the exam performance of students. [10] highlight the importance of the content played a crucial role, the lectures that were recorded by higher- quality teachers produced better exam results. In addition to this, [21] and [29] also researched the effect of OEPs on students’ performance. Even though they both mentioned the positive impact of OEPs on the students’ learning, according to them, other factors should be taken into account within the learning process such as the quality of content and user experience of the platform. Moreover, Khan Academy is also one of these OEPs, and some research concentrated on investigating the utilization and implementation of Khan Academy in the classroom.

2.2 Khan Academy

Khan Academy¹ is a non-profit educational organization created in 2008 by Salman Khan. The organization aims to create a set of online tools that help in providing education to anyone everywhere. Inside the platform, students can watch videos, read articles, and do exercises to study the selected topic. Furthermore, Khan Academy is currently available in more than 50 languages and Azerbaijani is one of them. In our research, we collaborated with the team who leads the localization of the Khan Academy into the Azerbaijani language to measure the impact of the platform on students through a pilot project.

Khan Academy is also one of these OEPs that are used in the classrooms. [9] and [34] measured the impact of using Khan Academy on the grades of students. [9] measured the causal effects of Khan Academy by recruiting 103 students from the 6th and 7th grades. They mentioned that while the expected improvement was 10%, the students showed a 16% improvement in scores. From another perspective, [34] investigated the impact of using Khan Academy on 75 students from the 7th grade and they concluded their research with positive feedback from both students and teachers. According to [34], even though there are better sources to learn, Khan Academy motivates students more since it also includes some engagement features such as badges. Moreover, [23] predicted the effectiveness of Khan Academy’s MAP Accelerator which is a mathematics mastery learning platform. They collected data from 181000 students in grades 3-8 across the United States. [23] found that students from high ELL (English Language Learners) districts did not have the same benefits from the use of the MAP Accelerator as their peers. Additionally, according to them, students from these districts were prone to improve 5.3 skills on average per hour, while this number was 7.2 for mid-ELL and 8.9 for low-ELL. Khan Academy also utilizes advanced analytics tools to analyze the learning of students [14]. [42] describes the ALAS-KA provides an extension of the learning analytics support for the Khan Academy platform. ALAS-KA includes also visualized dashboards which allow teachers to analyze the students’ learning process. And it also helps students to reflect on their learning.

Gamification is one of the tools that Khan Academy utilizes to increase the learning outcomes of students. Within this framework, they implement various gamification features such as badging, collecting experience points, etc. [36] researched the gamification of computer science content on Khan Academy. According to them, Khan Academy addresses the short-term engagement in the platform successfully by using gamification and this motivates the learner to move further. However, they concluded their research by mentioning the lack of meaningful gamification because this gamification model is not user- centric. [36] mentioned this because in the platform learners collect points without matching them to the underlying activities. This does not make the gamification “playful”. From another perspective, [41] researched the gamification features of Khan Academy. [41] conducted research on the learning of freshmen students in the topics of physics, chemistry, and mathematics. In this research, they particularly focused on the badging gamification feature of Khan Academy. They found that gaining badges increased their motivation to study more and they felt more motivated by gaining repetition because they were easier to get.

2.3 Gamification

Starting from the early 2010s, gamification was started to be used in the education context to increase learning productivity. Some OEPs utilize such gamification elements to improve the user experience [44]. One example of this can be Duolingo where while learning new words, the user collects the badges and points, then is eventually promoted to the next league in Duolingo [39]. The most common gamification elements are badges, leaderboards, virtual goods, etc. [12], [18], and [17] investigated how one of the gamification elements, badges, impacts students’ learning. Badges are graphical symbols or icons given as a reward for certain accomplishments in class such as being an active student, doing all the homework, and getting the badges [20]. According to [12] and [17], using badges positively impact the learning outcomes of students and improves students’ grades. They observed that utilizing badges increased the students’ motivation in the classroom. Nonetheless, [1] found that utilizing badges can negatively impact the motivation of students. [1] mentioned that when the students earned fewer badges compared to other students, it decreased their motivation.

From another perspective, [38, 11] concentrated on how the leaderboards impact students’ motivation and learning productivity. Leaderboards are used to enhance engagement through social comparisons. In the leaderboards, all the participants try to collect points, and based on their points, they are sorted from the most to the least points [31]. Even though the research results of [38], [11] showed leaderboards increase the engagement, motivation, and grades of students, [8] suggests that the implications of leaderboards can also lead to failure. Because, for example, in the case of the research by [32] and [37] leaderboards did not positively impact the motivation of certain groups in the classroom. The reason for this was that students who were at the bottom part of the leaderboard mentioned that it was impossible for them to catch up with the leaders. Thus, at a certain point, they decided to drop out.

Gamification is not limited only to badges, and leaderboards. For instance, [43] used avatars while [13] used virtual goods to increase the motivation of the students. Furthermore, [26] researched the impact of gamification by creating two different groups of students. The first group received a gamified curriculum and the second group received the same curriculum but without gamification elements. Their research resulted with showing the negative impact of gamification on students. They found that students who studied without a gamification-based curriculum scored higher in the final exams. [16] conducted a systematic literature review on the implementation of gamification in education. In this systematic review, [16] found that gamification has a great potential to improve learning if it is designed well and coordinated correctly. They also found that the majority of papers report positive feedback on using gamification in education since it increases the engagement of students, their attendance, and participation in voluntary activities. [24] researched a case study where a student dropped out the school and demonstrated why gamification could change this. [24] discusses that the student was very engaged and motivated about his classes once his teacher was using gamification elements such as rewards. However, with the new academy year, his teacher and their teaching methods changed, then his grades also drastically decreased. [24] hypothesizes that “gamification if conducted globally and interconnected within multiple subjects, can act as a protective factor against early school living.”

3. METHODOLOGY

3.1 Context of the study

This research analyzes the impact of using Khan Academy on the learning of students who were part of the pilot project supported by the Ministry of Education of the Republic of Azerbaijan. The project took place in one of the suburban areas of Baku, Azerbaijan, where the graduation level and student participation are lower (around 70%) than in other parts of the country. The project continued from the beginning of October 2021 until the beginning of April 2022. In this pilot project, students were introduced to using the math content of Khan Academy for the primary school levels. The focus group was the 3rd and 4th-grade students. The pilot project was designed in a way that each month around 50 students joined the project. Within this month, they were supposed to learn how to use the platform and then commence studying the topic that they were eager to. Since some students wanted to revise or study the topics that they could not learn in the previous years, they started to study the 1st-grade topic N the platform. Furthermore, at the end of the month, students were offboarded and instead of them, new around 50 students are onboarded. This happened 6 times and in total, 207 participated in the project. Even though Khan Academy holds different gamification features, we collected data about badging and collected experience points of students on the platform. We asked them to highlight three points from their Khan Academy (Appendix C).

3.2 Participants

Students were recruited by sending an information letter to the teachers 15 days in advance. All parents and teachers signed the consent forms to participate in the research. In addition to that, special research permission got also received from the school administration. We provided the tablets to all students and they answered the survey questions with the guidance of teachers. We held an additional online session with teachers to explain to them how the survey should be fulfilled. At last, all the students who participated in the pilot project fulfilled the survey. We gathered data from 207 students who studied in the 3rd or 4th grades (3rd grade students=53.6%, 4th grade students=46.4%). 53.6% of students mentioned that they identify themselves as “male”, and the remaining 46.4% selected the “female” option. The vast majority of students (81.2%) participated on all days of the project and only 2.6% of students mentioned that they missed the classes more than 5 times. On average, the students spent 255.0641 minutes (SD=189.5876 minutes) on Khan Academy, and the average experience points that the students collected were 43448.83 (SD=26807.62 points).

3.3 Data collection and analysis

We collected the data ² through surveys. While preparing the questions for the survey, we aimed to collect information about the profiles of students and their performance on Khan Academy (Appendix A). To understand the profiles of students, we asked them to mention their gender, grade, how they evaluate the research project, the number of participation days, whether using Khan Academy changed their grades or not, and their thoughts about using the platform in the future. To collect the responses, we defined several dates (16.04.2022, 18.04.2022, 20.04.2022) with the principal of the school. Because based on the feedback from the principal, the survey was complicated for the students to fulfill by themselves, and they needed the support of teachers. Due to ethical issues, parents had to confirm the participation of the students in the research, and consent was already collected at the beginning of the project when the students joined. Moreover, the main teacher of each class contacted the students to participate in the research. All the students who participated in the pilot project fulfilled the survey in the agreed sessions in the school together with the support of teachers and Khan Academy representatives. To facilitate the process, we also conducted one introductory session for the teachers so that they can answer any upcoming questions from the students. Based on the feedback from the teachers, no problems emerged within the survey fulfilling sessions. Moreover, to analyze the data, we implemented an association rule mining technique where we included variables collected through the survey.

4. RESULTS

4.1 The impact of the platform on the motivation and grades of students

We asked the students to evaluate their experience at Khan Academy. Table 1 demonstrates the responses of students to that question. The students mentioned their thoughts about the platform by giving points from 1 to 5 (1: Very bad, 2: Bad, 3: Normal, 4: Good, 5: Very good). While none of the students mentioned that their experience was very bad, 92.36% of students evaluated Khan Academy as “very good” and the average evaluation score was 4.87. Furthermore, students spent 255.06 minutes and they collected 26807.62 points on the platform on average. Secondly, we asked the students to mention whether they will use Khan Academy in the future. Almost all of the students (96.8%) mentioned that they will use Khan Academy as an additional source to improve their learning outcomes. Subsequently, we measured whether after using Khan Academy, their grades changed. Students could select one of these three options: 1) grade increased; 2) grade remained stable; 3) grade decreased. 68.6% of students answered that their grades increased after using Khan Academy and 29.5% mentioned that their grades did not change.

4.2 Association rules

After the application of the Apriori algorithm, we found three main associations that improved students’ motivation and learning. Table 2 indicates the association rules that we found after holding data analysis. The explanation of each variable is explained in Appendix B. The minimum support was 0.5 and the highest support was 0.81 among future_yes and participation_fully variables (confidence=0.98). In Table 2, all rules were generated when the minimum support was 0.5. The confidence in the rules (minimum support=0.5) varied from 51% to 99%. Table 2 also shows the generated association rules, their support, and confidence. From Table 2, we can see that 80.7% of the students, who fully participated in the pilot project, said that they will use Khan Academy in the future. And 67.3% of students, who mentioned that they plan to use Khan Academy in the future, increased their grades. Moreover, we found three major associations that confirmed the positive impact of the pilot project. Firstly, students who earned the Meteorite badge mentioned that they plan to use Khan Academy in the future (support=0.58, confidence=0.97). Meteorite badges are earned in the initial parts of Khan Academy and it is used to motivate the learner. The association that we found shows that earning the Meteorite badge motivated students and they increased their grades. Secondly, the students who fully participated in the classes increased their grades (support=0.62, confidence=0.74). Last but not least, the male students and 3rd- grade students are more prone to utilize Khan Academy in the future (support rate=0.5, confidence=0.98 and support rate=0.53, confidence=0.54 respectively).

When we decreased the minimum support to 0.4, then we also found that students who fully participated in the sessions and increased their grades are more prone to use Khan Academy in the future. Furthermore, the students who received Meteorite badges and fully participated in the classes mentioned that they will utilize the platform in the future. Lastly, based on the generated rules mentioned in Table 2, we can conclude that both full participation in the classes and increasing grades after using Khan Academy motivated students more to use Khan Academy in the future. Moreover, based on the generated association rules, we can see that getting Meteorite badges motivated students to participate in the classes and continue using Khan Academy further. Last but not least, students who identify themselves as "male" and students from the 3rd grade are more motivated to utilize Khan Academy in the future.

5. DISCUSSION

This paper presents the analysis of implementing online educational platforms to increase the motivation and learning outcomes of students. In this research, we investigated the case of using particularly Khan Academy as a tool to improve students’ grades and engagement. [49] and [51] also researched the Khan Academy’s impact. [49] mentions that it is very important that the teacher supports the students while using Khan Academy and this research found that Khan Academy can motivate students to do more exercises that directly affect their learning positively. Moreover, [51] highlighted the flipped classroom which included the Khan Academy promoted retention and enhanced students’ understanding. In our research, we can also mention that teacher assists students and it brings an extra engagement factor. Students mainly utilized Khan Academy in the school with the guidance and support of the teachers in our research and as [49] mentioned, it helped students not to deal with technical problems rather than focus on the learning process. Furthermore, similar to [49] and [51], we also found that Khan Academy increases the students’ motivation. Motivation is one of the most factors to increase learning and while the mentioned authors measured the students’ motivation by asking them directly, we analyzed their motivation by asking whether they wanted to use the platform in the future or not [45].

Gamification was also one of the important features that make Khan Academy more engaging from the perspective of students [36]. In our research, we found that students who fully participated in sessions and earned badges (Meteorite badge on Khan Academy) are the ones who also increased their grades. Here, we observe the positive impact of gamification on students’ grades. Even though [47] and [40] conducted their research in different regions (Brazil and Spain respectively), they also found that the gamification features of Khan Academy increase the engagement and motivation of students and this directly affects the students’ grades.

6. CONCLUSION AND FUTURE WORK

Online educational platforms enhanced students’ motivation and learning productivity in different cases. And gamification is also mentioned as one of the most impactful tools to increase engagement inside these online educational platforms. We found that in under-resourced communities, online educational platforms, particularly Khan Academy, positively affect students’ grades and motivation. Moreover, gamification increased the willingness of students to spend more time on the platform and use Khan Academy in the future as part of their education. Furthermore, in this research, we focused on primary school students in Azerbaijan, and as an extension of this study, the following points can be investigated. The first potential extension of this paper can be conducting research with secondary school students. Khan Academy helps students to go back to the subjects that students did not understand and study that particular topic. Hereby, secondary school students possess more subjects studied previously. Thus, conducting this research by focusing on secondary school students may indicate the impact of Khan Academy from a different perspective. The second potential extension of this research can be implementing the pilot project within the scope of other social sciences subjects. In our research, math was selected as the subject to implement within the pilot project. Nevertheless, the studying patterns of each subject are various.

Although all the students, who were part of the pilot project that was supported by the Ministry of Education of the Republic of Azerbaijan, fulfilled the survey, these students live in the same community. Thus, the research would provide more detailed results if we were able to collect data from other parts of the country. However, since it was the first time that this project was implemented, only one school was selected. Moreover, due to the scope of the project, we could not collect various information from students. And it affected the minimum support value that we defined. Initially, we defined the support value as 0.7, however, this value did not provide enough association rules to analyze.

6.1 Ethical concerns

Before starting the data collection, we informed both students and parents about the aim of the research and we mentioned that at any phase of the research, they can opt-out to participate and withdraw. Additionally, in the survey, we did not ask any questions that can identify participants. Some parents and students did not want the learners’ data to be collected and these students did the same activities with their peers, however, their data were not collected in any form. Moreover, anonymized data were stored in a secure database. Last but not least, there were not any kinds of potential legal, physical, or social harm to students.

7. REFERENCES

1. S. Abramovich, C. Schunn, and R. M. Higashi. Are badges useful in education?: It depends upon the type of badge and expertise of learner. Educational Technology Research and Development, 61(2):217–232, 2013.
2. O. B. Adedoyin and E. Soykan. Covid-19 pandemic and online learning: the challenges and opportunities. Interactive Learning Environments, 31:1–13, 2020.
3. L. Aguiar-Castillo, L. Hernández-López, P. De Saá-Pérez, and R. Pérez-JIménez. Gamification as a motivation strategy for higher education students in tourism face-to-face learning. Journal of Hospitality, Leisure, Sport & Tourism Education, 27:100267, 2020.
4. N. Ameen, R. Willis, M. N. Abdullah, and M. Shah. Towards the successful integration of e-learning systems in higher education in iraq: A student perspective. British Journal of Educational Technology, 50(3):1434–1446, 2019.
5. W. Bao. Covid-19 and online teaching in higher education: A case study of peking university. Human behavior and emerging technologies, 2(2):113–115, 2020.
6. T. Brahimi and A. Sarirete. Learning outside the classroom through moocs. Computers in Human Behavior, 51:604–609, 2015.
7. S. Carliner. An overview of online learning. 2004.
8. Y.-k. Chou. Actionable gamification: Beyond points, badges, and leaderboards. Packt Publishing Ltd, 2019.
9. L. Chu, A. Nautiyal, S. Rais, and H. Yamtich. Does khan academy work, 2018.
10. A. E. Clark, H. Nong, H. Zhu, and R. Zhu. Compensating for academic loss: Online learning and student performance during the covid-19 pandemic. China Economic Review, 68:101629, 2021.
11. M. Ćwil. Leaderboards–a motivational tool in the process of business education. In Joint International Conference on Serious Games, pages 193–203. Springer, 2020.
12. L. da Rocha Seixas, A. S. Gomes, and I. J. de Melo Filho. Effectiveness of gamification in the engagement of students. Computers in Human Behavior, 58:48–63, 2016.
13. L. de Marcos Ortega, A. García-Cabo, and E. G. López. Towards the social gamification of e-learning: A practical experiment. The International journal of engineering education, 33(1):66–73, 2017.
14. Á. Del Blanco, Á. Serrano, M. Freire, I. Martínez-Ortiz, and B. Fernández-Manjón. E-learning standards and learning analytics. can data collection be improved by using standard data models? In 2013 IEEE Global Engineering Education Conference (EDUCON), pages 1255–1261. IEEE, 2013.
15. S. Deterding, D. Dixon, R. Khaled, and L. Nacke. From game design elements to gamefulness: defining" gamification". In Proceedings of the 15th international academic MindTrek conference: Envisioning future media environments, pages 9–15, 2011.
16. D. Dicheva, C. Dichev, G. Agre, and G. Angelova. Gamification in education: A systematic mapping study. Journal of educational technology & society, 18(3):75–88, 2015.
17. M. Dindar, L. Ren, and H. Järvenoja. An experimental study on the effects of gamified cooperation and competition on english vocabulary learning. British Journal of Educational Technology, 52(1):142–159, 2021.
18. L. Ding. Applying gamifications to asynchronous online discussions: A mixed methods study. Computers in Human Behavior, 91:1–11, 2019.
19. A. Domínguez, J. Saenz-de Navarrete, L. De-Marcos, L. Fernández-Sanz, C. Pagés, and J.-J. Martínez-Herráiz. Gamifying learning experiences: Practical implications and outcomes. Computers & education, 63:380–392, 2013.
20. D. Gibson, N. Ostashewski, K. Flintoff, S. Grant, and E. Knight. Digital badges in education. Education and Information Technologies, 20(2):403–410, 2015.
21. R. Gopal, V. Singh, and A. Aggarwal. Impact of online classes on the satisfaction and performance of students during the pandemic period of covid 19. Education and Information Technologies, 26(6):6923–6947, 2021.
22. S. Grant and B. Betts. Encouraging user behaviour with achievements: an empirical study. In 2013 10th Working Conference on Mining Software Repositories (MSR), pages 65–68. IEEE, 2013.
23. P. Grimaldi, K. Weatherholtz, and K. M. Hill. Estimating the causal effects of Khan Academy Map Accelerator across demographic subgroups. In A. Mitrovic and N. Bosch, editors, Proceedings of the 15th International Conference on Educational Data Mining, pages 839–846, Durham, United Kingdom, July 2022. International Educational Data Mining Society.
24. L. Guerrero-Puerta and M. A. Guerrero. Could gamification be a protective factor regarding early school leaving? a life story. Sustainability, 13(5):2569, 2021.
25. L. Hakulinen, T. Auvinen, and A. Korhonen. The effect of achievement badges on students’ behavior: An empirical study in a university-level computer science course. International Journal of Emerging Technologies in Learning, 10(1):18–29, 2015.
26. M. D. Hanus and J. Fox. Assessing the effects of gamification in the classroom: A longitudinal study on intrinsic motivation, social comparison, satisfaction, effort, and academic performance. Computers & education, 80:152–161, 2015.
27. H. J. Heo and M. R. Choi. Flipped learning in the middle school math class. Advanced Science and Technology Letters, 71:94–97, 2014.
28. G.-J. Hwang, S.-Y. Wang, and C.-L. Lai. Effects of a social regulation-based online learning framework on students’ learning achievements and behaviors in mathematics. Computers & Education, 160:104031, 2021.
29. S. S. Jaggars and D. Xu. How do online course design features influence student performance? Computers & Education, 95:270–284, 2016.
30. N. F. Jamaludin, T. S. M. T. Wook, S. F. M. Noor, and F. Qamar. Gamification design elements to enhance adolescent motivation in diagnosing depression. International Journal of Interactive Mobile Technologies, 15(10):154–172, 2021.
31. Y. Jia, Y. Liu, X. Yu, and S. Voida. Designing leaderboards for gamification: Perceived differences based on user ranking, application domain, and personality traits. In Proceedings of the 2017 CHI conference on human factors in computing systems, pages 1949–1960, 2017.
32. R. N. Landers, K. N. Bauer, and R. C. Callan. Gamification of task performance with leaderboards: A goal setting experiment. Computers in Human Behavior, 71:508–515, 2017.
33. P. Magalhães, D. Ferreira, J. Cunha, and P. Rosário. Online vs traditional homework: A systematic review on the benefits to students’ performance. Computers & Education, 152:103869, 2020.
34. S. Marple, K. Jaquet, A. Laudone, J. Sewell, and K. Liepmann. Khan academy in 7th grade math classes: A case study. WestEd. org, 2019.
35. J. Martí-Parreño, A. Galbis-Córdova, and R. Currás-Pérez. Teachers’ beliefs about gamification and competencies development: A concept mapping approach. Innovations in education and teaching international, 58(1):84–94, 2021.
36. B. B. Morrison and B. DiSalvo. Khan academy gamifies computer science. In Proceedings of the 45th ACM technical symposium on Computer science education, pages 39–44, 2014.
37. M. Ninaus, S. D. Freitas, and K. Kiili. Motivational potential of leaderboards in a team-based math game competition. In International Conference on Games and Learning Alliance, pages 242–252. Springer, 2020.
38. M. Ortiz-Rojas, K. Chiluiza, and M. Valcke. Gamification through leaderboards: An empirical study in engineering education. Computer Applications in Engineering Education, 27(4):777–788, 2019.
39. D. Persico, M. Passarelli, F. Pozzi, J. Earp, F. M. Dagnino, and F. Manganello. Meeting players where they are: Digital games and learning ecologies. British Journal of Educational Technology, 50(4):1687–1712, 2019.
40. J. A. Ruipérez-Valiente, P. J. Muñoz-Merino, C. Delgado Kloos, et al. Detecting and clustering students by their gamification behavior with badges: A case study in engineering education. International Journal of Engineering Education, 33(2-B):816–830, 2017.
41. J. A. Ruipérez-Valiente, P. J. Muñoz-Merino, and C. D. Kloos. Analyzing students’ intentionality towards badges within a case study using khan academy. In Proceedings of the sixth international conference on learning analytics & knowledge, pages 536–537, 2016.
42. J. A. Ruipérez-Valiente, P. J. Muñoz-Merino, D. Leony, and C. D. Kloos. Alas-ka: A learning analytics extension for better understanding the learning process in the khan academy platform. Computers in Human Behavior, 47:139–148, 2015.
43. M. Sailer, J. U. Hense, S. K. Mayr, and H. Mandl. How gamification motivates: An experimental study of the effects of specific game design elements on psychological need satisfaction. Computers in human behavior, 69:371–380, 2017.
44. M. Sailer and M. Sailer. Gamification of in-class activities in flipped classroom lectures. British Journal of Educational Technology, 52(1):75–90, 2021.
45. U. Schiefele. Interest, learning, and motivation. Educational psychologist, 26(3-4):299–323, 1991.
46. J. Simões, R. D. Redondo, and A. F. Vilas. A social gamification framework for a k-6 learning platform. Computers in Human Behavior, 29(2):345–353, 2013.
47. M. M. Tenório, R. P. Lopes, L. A. d. Góis, and G. d. S. Junior. Influence of gamification on khan academy in brazilian high school. PUPIL: International Journal of Teaching, Education and Learning, 2(2):51–65, 2018.
48. O. Viberg, M. Khalil, and M. Baars. Self-regulated learning and learning analytics in online learning environments: A review of empirical research. In Proceedings of the tenth international conference on learning analytics & knowledge, pages 524–533, 2020.
49. H. E. Vidergor and P. Ben-Amram. Khan academy effectiveness: The case of math secondary students’ perceptions. Computers & Education, 157:103985, 2020.
50. B. Yusuf and J. Ahmad. Are we prepared enough? a case study of challenges in online learning in a private higher learning institution during the covid-19 outbreaks. Advances in Social Sciences Research Journal, 7(5):205–212, 2020.
51. Y. Zengin. Investigating the use of the khan academy and mathematics software with a flipped classroom approach in mathematics teaching. Journal of Educational Technology & Society, 20(2):89–100, 2017.

APPENDIX

A. SURVEY QUESTIONS

1. What grade are you in? a. 3rd grade b. 4th grade
2. What is your gender? a. male b. female c. other (please specify)
3. How would you evaluate the platform? 1 (Very bad) - 2 (Bad) - 3 (Normal) - 4 (Good) - 5 (Very good)
4. How many days did you participate in the project? a. I participated all days. b. I missed 1-2 days. c. I missed 3-4 days. d. I missed more than 5 days.
5. What badges did you earn on the platform? a. Meteorite b. Moon c. Earth d. Sun e. Black Hole f. Challenge patches
6. How many minutes of study time did you have during the project?
7. How many practice points (XP) did you collect on the platform?
8. How has your math grade changed since using the platform? a. My grades increased. b. My grades decreased. c. My grades remained stable.
9. Do you want to use the platform in the future? a. Yes b. No

B. ACRONYM OF THE VARIABLES

• future_yes: the students who mentioned that they would use Khan Academy in the future
• badge_meteorite: the students who received meteorite badges on Khan Academy
• gradechangegrades_increased: the students who increased their grades during the project
• gendermale: the students who identify themselves as male
• grade3rd: the students who study in the 3rd grade
• participationfully: the students who participated in whole days of the project

C. COLLECTED DATA FROM STUDENTS’ PROFILE ON KHAN ACADEMY

• Badges that they earned on the platform. Badges are one of the gamification tools to increase engagement on Khan Academy and active users are awarded badges based on different accomplishments. On Khan Academy, users can earn six various types of badges (Challenge badges - special awards for completing topic challenges on Computing courses; Meteorite badges - common and easy to earn when just getting started; Moon badges - uncommon and represent an investment in learning; Earth badges - require a significant amount of learning; Black Hole badges - the rarest Khan Academy awards; Sun badges - require impressive dedication).
• Experience points (XPs) that they earned. By watching videos, reading articles, and doing exercises, the user can earn points on Khan Academy and we asked the students to highlight how many XPs they earned.
• Learning duration. Khan Academy counts the number of minutes spent on the platform while doing learning activities such as watching videos and solving problems and students mentioned this in the survey.

© 2023 Copyright is held by the author(s). This work is distributed under the Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International (CC BY-NC-ND 4.0) license.