ABSTRACT

Research on tailored gamification has shifted from analysing single students’ characteristics (e.g. gender or behavioural profiles) to multiple characteristics and how they are influenced by gamification (e.g. context and system log usage). Yet, few studies have been conducted that are concerned with culture, which influences many of the students’ characteristics and, consequently, their learning performance. In order to provide a better gamified experience, it is important to understand culture and how it impacts on students’ perceived importance of gamification elements, so these elements can be adapted to specific cultures. To this end, we conducted an exploratory study using Association Rule Mining to explore how the cultural landscape (country) influences the students’ behaviour and perceived importance of gamification elements in educational environments. We collected and analysed data (N = 1296) from two different countries, Brazil (N = 428) and the United States (N = 868) and identified significant differences between the perceived importance of some gamification elements.

Keywords

1. INTRODUCTION

Personalised gamification in learning environments has become a trend in the past few years [7, 6]. This field of study focuses on enhancing existing gamification approaches with user-centred and personalised design, tailored to the specific characteristics of the user, which will have an impact on their perceived satisfaction, engagement, and motivation, when using a gamified system [4]. Tailored gamification engaged the interest of researchers and educational professionals, due to facilitating positive effects associated with it, such as motivation and performance increase [12].

Recent studies focused on the use of personalised approaches using student demographics (e.g. gender) [21], gamer profiles (e.g. using HEXAD [22] model), and gaming preferences (e.g. their favourite game genres or elements) [14] as part of the gamification design.

Whilst these recent studies demonstrated that personalised gamification tends to have a positive influence on students’ motivation and performance, there is still no consensus on what kind of personalisation and which attributes should be considered, since most studies focus only on single factors [16]. Literature on personalised gamification points to a lack of studies that deal with other learners’ characteristics that can also influence their perception and interaction with gamification, one such characteristic is that of culture [6, 13].

Culture is not a trivial concept to define; according to [18], culture is an evolving cognitive structure, composed of schema that influence the behaviour of members of a given group. One such set of schema are those that influence and are influenced by the geographical location of an individual and educational contexts they are exposed to [17]. It is this aspect of ’culture’ that we will seek to investigate within the rest of this paper. Recently, culture’s importance to gamification has come to the fore [2, 24, 19]. In spite of this surge in interest, according to [19], most existing gamification studies that address education and culture are focused on language learning, rather than on understanding how culture can impact on gamification design.

Based on what has been exposed, this paper aims to answer the following research question ”Does culture impact on the importance of gamification elements?”. We answer this research question, by conducting a relatively large and representative¹ quantitative study, analysing the perceived importance of gamification elements in two different countries (United States and Brazil). Our results provide empirical evidence on the differences between the perceived importance of gamification elements for these two different countries, which can guide designers when considering national culture as a set of variables² in gamification design for learning environments, and also guide researchers to conduct new research in this field. In summary, our main contributions are new insights on which game elements are more preferred by different countries.

2. METHODS

Considering our main research question, we designed an initial sub-research question to identify of a Country consider a certain gamification element more important than other country (in this case, Brazil and US). In this sense the can be any element from the Taxonomy of Gamification Elements for Education (TGEE) proposed in [20], while the country can be any country that has data available to be analysed.

We opted for the TGEE proposed in [20] due to its being the first classification created and evaluated for educational purposes. According to the authors, the TGEE is composed of 21 gamification elements, alongside its synonyms, descriptions, and examples of use. These elements are classified in 5 categories that deal with students’ performance, sociability, personal information, and experience, and the environment ecosystem.

Through analysing the perceived importance of gamification elements by different countries, we can infer how the culture can influence the design of personalised gamified applications for education. To conduct this research, first we used inferential statistics, to test our assumption that the elements’ perceived importance differs, depending on the country of residence. To do so, we used the Mann-Whitney U test (due to the non-parametric and independent nature of our data [11]) and Cliff’s delta, to understand to what extent perceptions differ (i.e., the effect size) [10]. Following this we conduct an exploratory analysis, using unsupervised learning algorithms and descriptive/inferential statistics to provide inferences that can be explored deeply in future studies. The data-driven pipeline consists of data collection, filtering, and analysis, further explained in the respective sections.

To collect our data, we relied on the survey method, due to its low cost and other benefits (e.g. speed of obtaining answers) [8]. The survey used in this study was designed in Google forms and consists of two parts.

The first collects self-reported demographic information about the respondents, such as gender, country (that they reside) and also some information regarding their ’gamer’ status, such as: years playing games, hours per week spent playing, favourite game genres and favourite setting (single or multiplayer games). Whilst it is known that games and gamification are not the same, however, recent studies [15] demonstrated that gaming characteristics of users do influence their gamification acceptance. Previous literature also showed that students’ player profile might additionally influence their performance when using gamified learning environments. Based on these premises, we collect the participants’ gaming culture information, to establish how it influences the perceived importance of the gamification elements. The second part of the survey consists of 21 gamification elements that were proposed for educational contexts, where we asked what the respondents’ perceived importance for each of the 21 gamification elements was, on a Likert scale [9] from 1 to 5, where 1 meant “not important at all” and 5 “extremely important”.

The recruitment of respondents occurred via (1) Amazon Mechanical Turk, since recent studies [3] demonstrated that this is an effective platform to obtain a representative sample of answers (with some limitations) especially in the US, and (2) social networks, which allow us to reach a broad audience and can also be useful tools for survey recruitment [3], social networks were used mainly to collect data from Brazil. We left the survey open for answers for 3 months. The first page of the survey contained the informed consent form³, which the participant needed to agree to, in order to participate in the study, as well as the information regarding the study’s objectives, as suggested in [5].

After preparing our dataset, we used descriptive statistics and ARM to explore the data. Descriptive statistics allow us to understand the significant differences between countries, which in addition can support us in answering. ARM consists of verifying the associations between the data, by presenting if-then-else clauses that are explained using a given set of metrics. It is mainly used in market-basket analysis, to identify possible combinations of elements [1]. In this work, we will consider support (related to the frequency of a given item), confidence (related to the strength of the clause based on its frequency), and lift (which measures the independence of the items in the clause) as the main metrics for ARM.

3. RESULTS AND DISCUSSIONS

In this section, we present the results of our analyses using the given dataset. Initially, our raw data consisted of 1952 answers; after applying our filters, as described in the previous section, we obtained a total of 1296 respondents who live in Brazil (n = 428; 33%) or the US (n = 868; 67%). Considering the country distribution and genders (Table 1), 428 are from Brazil (33%) and 868 from the US (67%). Among the respondents, 447 identify themselves as female (34.5%), 839 as male (64.7%), nine as non-binary (0.7%) and one person preferred not to disclose (0.1%).

Considering their age and time spent on games distribution: the minimum age (in years) reported (after applying our filters) is 16, while the maximum age was 77. The medium age of the population is 29.7 with a standard deviation of 10. Considering the experience and gaming preferences of our population, they had an average of 18.7 years of playing games (SD = 8.2), and an average of 14 hours per week (SD = 13.8). The top 3 favourite game genres were RPG (499 answers, 38.5%), adventure (259 answers, 20%) and strategy (186 answers, 14.4%).

The favourite setting (Table 2) of the respondents was single player games (898 answers, 69.3%), followed by multiplayer games (398, 30.7%). In Brazil, the top 3 favourite game genres were RPG (N = 182, 42.5%), followed by Adventure (N = 66, 15.4%), and Strategy (N = 65, 15.2%). As for the US, they considered RPG (N = 317, 36.5%), followed again by Adventure (N = 193, 22.24%), and Strategy (N = 121, 13.94%).

When applying our sub-research question formula to each of the elements, we can see in Table 3 that only 6 elements showed a significant difference ($p<0.05$, after corrected with False discovery rate (FDR)). We also applied Cliff’s Delta to measure the effect size (since this delta can be used to measure the effect size of different size populations) and obtained small effect size results on the same 6 elements [23].

Considering the results seen in Table 3, we can observe that 6 elements (Cooperation, Novelty, Stats, Sensation, Narrative, and Storytelling) had a significant difference in the hypothesis tests ($p<0.05$), and a small effect-size (up to 0.3). In practice, this means that each of the analysed countries consider these elements to a different degree of importance. The low effect-size indicates that these element differences may pose a significant impact when tied with other elements considering the respondents’ country; this must be confirmed with empirical studies using these elements.

As for the Association Rules, when mining all the rules that contain the variable country, we found 937 rules. In this work, we considered only rules that contained a gamification element and that satisfies the following conditions: (A) $\mathit{confidence}>0.8$; and (B) $\mathit{lift}>1.1$, similar as seen in [21].

Considering our conditions, we found 44 rules for condition (A); and 295 for condition (B). When considering the intersection between condition A and B, we found 40 rules ($A\cap B=40$). The strongest rule (rule 3275, confidence = 0,93; lift = 1,99) that was mined was associating Brazil with Objective, Storytelling, and Narrative. The following rules also associated Brazil with different gamification elements as Progression (rule 3279), Sensation (rule 3270) and characteristics as favourite setting as single player (rule 3278). The same set of gamification elements (Narrative, Storytelling, Progression, and Objectives) were also associated with the United States. The major difference was in the Sensation element that was found in the strongest rules associated with Brazil (Table 4).

4. CONCLUSIONS AND FUTURE WORKS

This paper presented a data-driven work focusing at exploring cultural differences between Brazil and the US regarding their perceived preferences on gamification elements. We found significant and interesting associations and groups of gamification elements that can be used in educational environments (both virtual and non-virtual). These can also be used as input for adaptive gamification environments as a new set of variables that can be grouped by the country of origin of the student.

In summary, we provide as the main contributions of our work to be provide the first empirical evidence on the impact of culture in the perceived importance of gamification elements, comparing Brazil and the US, as well as providing new strategies based on respondents’ country of origin and others easy-to-obtain characteristics that can be used in adaptive gamified learning environments.

In future works, we intend to expand the research by applying other unsupervised algorithms, and investigating other sub-cultures within these countries and identify different and similar patterns. We also intend to expand the concept of culture and not associate it only with a country, but also including the region the respondent was raised and live nowadays, as well as other variables that can be related to their education level, or gaming preferences that might have an influence in their perceived importance of gamification elements. Another future work is related to the implementation of these strategies in a real educational environment. Finally, it would be interesting to also analyse other countries and see the differences and similarities between patterns.

5. ACKNOWLEDGMENTS

This research was partially funded by the following organizations: Brazilian National Council for Scientifc and Technological Development (CNPq)—processes 141859/2019-9, 163932/2020-4, 308458/2020-6, and 308513/2020-7; Coordination for the Improvement of Higher Education Personnel (CAPES)—Finance Code 001; and São Paulo State Research Support Foundation (FAPESP)—processes 2018/15917- 0 and 2013/07375-0. Additionally, this research was carried out within the scope of the Samsung-UFAM Project for Education and Research (SUPER), according to Article 48 of Decree n◦ 6.008/2006 (SUFRAMA), and partially funded by Samsung Electronics of Amazonia Ltda., under the terms of Federal Law n◦ 8.387/1991, through agreements 001/2020 and 003/2019, signed with Federal University of Amazonas and FAEPI, Brazil.

6. ADDITIONAL AUTHORS

Additional authors: Craig D. Stewart from Durham University, Elaine Harada Teixeira de Oliveira from Federal University of Amazonas, Isabela Gasparini from Santa Catarina State University, Seiji Isotani from University of Sao Paulo, and Alexandra I. Cristea from Durham University.

7. REFERENCES

1. R. Agrawal, T. Imieliński, and A. Swami. Mining association rules between sets of items in large databases, 1993.
2. A. AlMarshedi, V. Wanick, and G. B. Wills. Gamification and Behaviour. In Gamification, pages 3–18. 2016.
3. F. Bentley, K. O. Neill, K. Quehl, and D. Lottridge. Exploring the Quality , Efficiency , and Representative Nature of Responses Across Multiple Survey Panels. In Proceedings of the 2020 CHI Conference on Human Factors in Computing Systems, pages 1–12, New York, NY, USA, apr 2020. ACM.
4. M. Böckle, J. Novak, and M. Bick. Towards Adaptive Gamification: A Synthesis of Current Developments. In Twenty-Fifth European Conference on Information Systems (ECIS), jun 2017.
5. L. Gelinas, R. Pierce, S. Winkler, I. G. Cohen, H. F. Lynch, and B. E. Bierer. Using Social Media as a Research Recruitment Tool: Ethical Issues and Recommendations. https://doi.org/10.1080/15265161.2016.1276644, 17(3):3–14, mar 2017.
6. A. C. T. Klock, I. Gasparini, M. S. Pimenta, and J. Hamari. Tailored gamification: A review of literature. International Journal of Human-Computer Studies, page 102495, jun 2020.
7. E. Lavoue, B. Monterrat, M. Desmarais, S. S. George, É. Lavoué, B. Monterrat, M. Desmarais, and S. S. George. Adaptive Gamification for Learning Environments. IEEE Transactions on Learning Technologies, 12(1):16–28, jan 2018.
8. J. Lazar, J. H. Feng, and H. Hochheiser. Research methods in human-computer interaction. Morgan Kaufmann, 2nd edition, 2017.
9. R. Likert. A technique for the measurement of attitudes. Archives of Psychology, 140(140):44–53, 1932.
10. G. Macbeth, E. Razumiejczyk, and R. D. Ledesma. Cliff’s delta calculator: A non-parametric effect size program for two groups of observations. Universitas Psychologica, 10(2):545–555, 2011.
11. H. B. Mann and D. R. Whitney. On a Test of Whether one of Two Random Variables is Stochastically Larger than the Other. The Annals of Mathematical Statistics, 18(1):50–60, jun 1947.
12. W. Oliveira and I. I. Bittencourt. Tailored Gamification to Educational Technologies. Springer Singapore, Singapore, 2019.
13. P. T. Palomino, A. M. Toda, L. Rodrigues, W. Oliveira, and S. Isotani. From the Lack of Engagement to Motivation: Gamification Strategies to Enhance Users Learning Experiences. SBC – Proceedings of SBGames 2020, pages 1127–1130, 2020.
14. L. Rodrigues, P. T. Palomino, A. M. Toda, A. C. T. Klock, W. Oliveira, A. P. Avila-Santos, I. Gasparini, and S. Isotani. Personalization Improves Gamification: Evidence from a Mixed-Methods Study. Proc. ACM Hum.-Comput. Interact., 5(CHI PLAY), oct 2021.
15. L. Rodrigues, F. Pereira, A. Toda, P. Palomino, W. Oliveira, M. Pessoa, L. Carvalho, D. Oliveira, E. Oliveira, A. Cristea, and S. Isotani. Are they learning or playing? moderator conditions of gamification’s success in programming classrooms. ACM Trans. Comput. Educ., nov 2021. Just Accepted.
16. L. Rodrigues, A. M. Toda, P. T. Palomino, W. Oliveira, and S. Isotani. Personalized gamification: A literature review of outcomes, experiments, and approaches. ACM International Conference Proceeding Series, pages 699–706, oct 2020.
17. I. Savard, J. Bourdeau, and G. Paquette. Considering cultural variables in the instructional design process: A knowledge-based advisor system. Computers and Education, 145:103722, feb 2020.
18. I. Savard and R. Mizoguchi. Context or culture: what is the difference?, dec 2019.
19. A. Toda, A. C. T. Klock, P. T. Palomino, L. Rodrigues, W. Oliveira, C. Stewart, A. I. Cristea, I. Gasparini, and S. Isotani. GamiCSM: Relating education, culture and gamification - a link between worlds. In XIX Brazilian Symposium on Human Factors in Computing Systems, page In press., Diamantina, 2020.
20. A. M. Toda, W. Oliveira, A. C. Klock, P. T. Palomino, M. Pimenta, I. Gasparini, L. Shi, I. Bittencourt, S. Isotani, A. I. Cristea, L. Shi, I. Gasparini, S. Isotani, A. I. Cristea, L. Shi, I. Bittencourt, S. Isotani, and A. I. Cristea. A Taxonomy of Game Elements for Gamification in Educational Contexts: Proposal and Evaluation. In IEEE 19th International Conference on Advanced Learning Technologies (ICALT), pages 84–88, 2019.
21. A. M. Toda, W. Oliveira, L. Shi, I. Bittencourt, S. Isotani, and A. Cristea. Planning Gamification Strategies based on User Characteristics and DM : A Gender-based Case Study. In Proceedings of the Educational Data Mining 2019 conference, number i, pages 438 – 443, Montréal, may 2019.
22. G. F. Tondello, R. R. Wehbe, L. Diamond, M. Busch, A. Marczewski, and L. E. Nacke. The Gamification User Types Hexad Scale. In Proceedings of the 2016 Annual Symposium on Computer-Human Interaction in Play - CHI PLAY ’16, pages 229–243, New York, New York, USA, 2016. ACM Press.
23. A. Vargha and H. D. Delaney. A Critique and Improvement of the CL Common Language Effect Size Statistics of McGraw and Wong:. http://dx.doi.org/10.3102/10769986025002101, 25(2):101–132, nov 2016.
24. R. Wellington. Context to Culture for Gamification HCI Requirements: Familiarity and Enculturement. In Gamification in Education and Business, pages 151–163. Springer International Publishing, Cham, 2015.

© 2022 Copyright is held by the author(s). This work is distributed under the Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International (CC BY-NC-ND 4.0) license.