ABSTRACT

To better understand the needs and support the development of computational thinking (CT) skills and persistence for young learners, we implement BarrelBots, a puzzle-based block coding activity in Minecraft, with middle school and high school learners. BarrelBots leverage an appealing and emotionally engaging agent that is controlled by a custom programming language. Learners solve a range of pre-designed puzzles of increasing difficulty. We present a comparison between two summer camps with identical BarrelBots curricula, a middle school and high school camp. To explore whether age impacts problem-solving behaviors, we ran a comparison of within-sequence Shannon entropy between the camps and found that younger students experienced more diverse states on a more difficult puzzle task. We then ran cSPADE, descriptive statistics, and hierarchical clustering of pairwise distances to understand how the problem-solving behaviors differed between the camps and between clusters within camps on the more challenging puzzle. The results demonstrated that high performing learners solved the task iteratively, behaviors like removing all blocks or rearranging blocks in subsequent attempts may indicate unproductive struggling or unproductive problem-solving behaviors, and younger students may benefit more from productive struggling while older learners should focus on fewer more deliberate attempts. The findings have implications for the development of wheel-spinning detectors and supports around persistence.

Keywords

1. INTRODUCTION

Computational thinking (CT) is fundamentally grounded in problem solving. Wing [27] characterizes CT as a way of approaching problems, system design, and human behavior through the application of core concepts from computer science. This framing positions CT as a cognitive process that extends beyond programming syntax, and encompasses how learners reason and approach complex tasks. Understanding CT problem solving therefore requires attention to how learners engage during CT activities. CT problem solving is inherently sequential and iterative, requiring learners to coordinate planning, execution, and evaluation across multiple attempts [23]. Productive persistence, sometimes framed as productive struggle, refers to sustained engagement that ultimately contributes to learning through iterative problem solving, strategy revision, and recovery from failure [16]. In contrast, non-productive persistence occurs when students continue working without making meaningful progress, often due to ineffective or unchanging strategies [14]. One well-studied manifestation of non-productive persistence is wheel-spinning, defined as situations in which learners persist for extended periods of time without achieving mastery or demonstrating durable learning [3, 14].

The importance of examining different learner characteristics and CT problem-solving skills has been emphasized in recent work. Research on the development of CT skills and learner characteristics like age, gender, prior experience, and other various learner characteristics are underexplored [2, 12, 25, 20]. Findings also suggest a need for deeper exploration into learners’ CT problem-solving strategies. Common surface indicators such as time on task or number of attempts are often treated as straightforward proxies for engagement or effort. However, evidence suggests these measures can be unstable and difficult to interpret [17]. Therefore, relying on aggregate counts or durations alone risks obscuring the processes that produce CT learning. These concerns motivate analytic approaches that prioritize sequences of action. Examining the order and timing of behaviors provides greater leverage for distinguishing intentional revision from repetition, exploration from stagnation, and persistence from wheel-spinning. A method that has been used to understand persistence in education research is sequential pattern mining (SPM) [6]. SPM is a technique in educational data mining that discovers productive and unproductive behavioral patterns in time-series data [15].

Thus, the aim of this study is to examine how age shapes both CT skill development and problem-solving behaviors associated with persistence among students engaged in Minecraft CT-focused, puzzle-based activities. Specifically, we investigate whether learners at different developmental stages (middle versus high school) demonstrate distinct problem-solving strategies when interacting with the same learning environment (BarrelBots). While prior work has used SPM to characterize problem-solving behaviors in game-based CT activities [26], little research has examined its use in Minecraft or whether the effectiveness of these behaviors varies by age. Addressing these gaps, our study explores age-related differences in puzzle-solving patterns, with a broader goal of informing the design of age-appropriate, just-in-time supports to promote persistence. Guided by these aims, we address the following research question: how do problem-solving behaviors, particularly those related to persistence, vary by participant age and computational thinking skill level during engagement with the same CT puzzle-based activity in Minecraft?

2. BARRELBOTS

Minecraft has become one of the most popular games in the world, with over 200 million monthly users, and 21.21% of daily user traffic originating from the U.S. [18]. Researchers have shown that even with little experience, learners master controls quickly and can effectively engage content when Minecraft is used in STEM learning environments [19]. Although some researchers caution about possible biases in Minecraft research [24], given the ubiquity of the game, ample opportunities to conduct research, and effectiveness to promote learning outcomes, Minecraft deserves attention for its potential in helping users in both formal and informal learning environments.

BarrelBots were modeled after the Minecraft barrel block with an approachable non-robotic appearance (see Figure 1). This is a step away from traditional computer science nomenclature like “agent” (programmable entities) and the association with robotics. They feature arrows on top indicating the direction that they are facing and will move, if instructed to go forward, in that direction (see Figure 1). Following suit with classic approaches to teaching sequential programming [21], BarrelBots provide a basic set of commands to the learner and three ways to move: move forward one block, rotate clockwise 90 degrees, and rotate counterclockwise 90 degrees. In addition, learners can unlock additional blocks including loops, conditionals, functions, and several specialized action blocks (e.g. to place a block in Minecraft, or “build”).

The BarrelBots activity contains researcher-designed puzzle modules that were previously tested with students from another pilot study. The puzzle modules are separated in “lanes”. Each module is organized by a given CT construct and difficulty level. Participants are first introduced to three introductory puzzles that incorporate movement and loops (see Figure 2) , then two intermediate puzzles that incorporate conditionals, and lastly four difficult puzzles that incorporate sequencing and functions. Participants are given a set number of specific blocks per puzzle and are tasked with moving the BarrelBots from their starting position to the green block, representing the end state, while staying within the puzzle boundaries (see Figure 2).

3. METHODS

3.1 Participants

BarrelBots was deployed as part of a large week-long summer camp series in 2024 and 2025 that reaches over 100 middle and high school learners. Participants came from partner public schools that operate in some of the most under-served regions of a large midwestern city in the United States. Our study included two subsections (single week long classes) with a total of 18 learners in 2024 and 24 learners in 2025. The demographic breakdown of the 2024 camp is as follows: 72% African American, 17% White/Caucasian, 11% other ethnicity, 11% identified as female, and average age of 12.38. The demographic breakdown of the 2025 camp is as follows: 77% African American, 8% other ethnicity, 15% prefer not to answer, 23% identified as female, and average age of 13.45. Learners in both camps chose the BarrelBots activity out of several options. Many learners elected to participate in the BarrelBots camp but not all of them were able to be included. Most importantly, the majority of learners came from schools where there is limited computer science curriculum and many were beginner programmers.

3.2 Materials

Participants were all provided with a laptop, mouse, and an individual loaner (anonymous) account to play Minecraft: Java Edition on a server with the BarrelBots data pack described above. They used the same account for each session. The code blocks in BarrelBots are tracked every time players are finished editing their code and they are stored in a remote, secure database.

For pre- and post-tests, we used the 25-question competent Computational Thinking test (cCTt), a CT puzzle assessment developed by experts through use of focus groups and validated with substantial learner samples [7]. The measure showed strong model fit for 6 latent factors representing CT concepts (e.g. while statements, loops) across all participants (ages 7 - 9; \(\chi ^2\)(260) = 551, p < .001, CFI = .978, TLI = .974, RMSEA = .027, SRMR = .052). Overall internal reliability for the measure was high for the full sample (\(\alpha \) = .85). The cCTt has also been shown to be valid and reliable with middle school students [8]. Although a measure with more complex items may be more useful for older students, we posit that the cCTt can be used with our participants. See Figure 3 for an example question from the measure.

3.3 Procedure

Both camps consisted of five days based in a classroom, with the first three being dedicated to BarrelBots activities. On the first day, campers tried activities related to all of the camps offered. They ranked their first and second choices. On the second day of camp, the participants that ranked the BarrelBots activity highest (or only) were selected to take part in the sessions. They were introduced to rules for the camp and for playing on the Minecraft server. The participants had 60 minutes to finish the puzzles in module 1 and create their own puzzle using loops. They then had 30 minutes to share the puzzle they created with their partners and the instructors. After a recess break, participants had additional time to work on module 2 (conditionals) and module 3 (sequencing and functions) puzzles and create their own puzzles for each module. The third day, participants had an hour to work on continuing puzzles from the previous day. After recess, they had time to continue the sequencing and function puzzles and had collaborative open build time if they finished early.

3.4 Data Analysis

We conducted SPM and descriptive statistics on participant game play behaviors to determine whether and how do the problem-solving behaviors differ by camp and CT competency. Borrowing from the methodology of Wang et al. [26], we used SPM to analyze the log data from two BarrelBots puzzles, an introductory and a more advanced puzzle, to mitigate bias. The introductory problem (see the middle puzzle in Figure 2) was intended to be solvable without prerequisite understanding of computing structures such as loops or conditionals. The more advanced puzzle was the final puzzle that was reached by participants in both camps. It contained an enemy bot that shot lasers that disabled BarrelBots and challenged participants to creatively figure out solutions to get around the obstacle using knowledge from previous modules.

To compare puzzle-solving strategies between the drastically different puzzles, we used BarrelBots inventory states [11] instead of the exact blocks used in the attempts (see Table 1 for definitions and examples of each inventory state).

We used the TraMineR package in R to compare and visualize the sequential player behavioral patterns from our participants [10]. To determine how the diversity of problem solving behaviors is related to the participant age we conducted a linear regression of within-sequence Shannon entropy on the camp type. Shannon entropy is a measure of diversity of states at a given position in the sequence [10], and is a methodology that has been used in social science research [4, 9]. To understand how the puzzle-solving patterns differ between groups we conducted cSPADE to compare the most frequent strategies used by participants within each camp. We specified a minimum support (minimum frequency that a sequence must occur in order to be considered) of .25 and a maximum gap (maximum time difference between to be considered consecutive elements) of 2 [15]. Lastly, to compare how puzzle-solving behaviors differed within camps based on CT competency level, we used hierarchical clustering of pairwise distances between sequences with the TraMineR package to visualize player behavior patterns [26]. To calculate the pairwise distances between the sequences of inventory states for our participants we use the optimal matching method [1], a common sequence alignment algorithm that enables us to categorize patterns [1, 10]. We then used Ward’s criterion to perform hierarchical clustering and group similar sequences [22]. To determine the number of clusters that should be used for each camp, we used the dendrograms for each dataset to visually identify the approximate number of groupings.

4. RESULTS

4.1 Pattern Differences Between Camp

A linear regression for the camp type on the Shannon entropy values of the player sequences on the introductory puzzle was insignificant, F(1,53) = .89, p = .35. The linear regression for the camp type on the Shannon entropy values of the player sequences on the more challenging puzzle, however, was significant, F(1,34) = 7.40, p = .01, R\(^2\) = .18, adjusted R\(^2\) = .15. Middle school students had significantly more diversity of states in their sequences, t(34) = 2.72, p = .01. Since the main construct of interest of this study is persistence, which is typically exhibited on more advanced tasks [14], and we did not find significant differences in state diversity on the introductory puzzle, follow-up analysis was only ran on the more challenging puzzle.

4.2 cSPADE Patterns Between Camps

The results of cSPADE for the frequent patterns on the more advanced puzzle showed that students in both camps used similar puzzle-solving behaviors. The support, percentage of learners in the group that used the sequence of inventory states, for patterns in the middle school camp was typically higher than the support for the same patterns found for the high school camp. High school learners, however, had higher support for iterative problem-solving patterns (Add \(\rightarrow \) Add and Add \(\rightarrow \) Add \(\rightarrow \) Add). One behavior that was found with the middle school students that was not present with the high school students was Remove All.

4.3 Cluster Analysis of Puzzle-Solving Behaviors Within Camps

Following the methodology of Wang et al. [26], since we are primarily interested in developing personalized supports for persistence early in the puzzle-solving task, we truncated each participants’ inventory sequences to the first 10 states (half of the mean number of attempts per participant used on the task). From visual inspections of the dendrograms that visualize the hierarchical relationships of the clusters, we concluded that three clusters were optimal for both camps.

For the middle school camp, participants who scored the highest on the cCTt post-test most frequently tackled the problem iteratively by focusing their early attempts on adding new blocks. Players in this cluster later removed blocks then added new ones. Participants in the lowest scoring clusters frequently added blocks then removed all of them perhaps when their initial strategy did not work.

For the high school camp, participants who scored the highest on the cCTt post-test most frequently tackled the problem iteratively by focusing their early attempts on adding new blocks. The players in the lowest scoring cluster added blocks but unlike other clusters rearranged them in multiple subsequent attempts.

4.4 Number of attempts

On the more challenging puzzle, the middle school learners used almost double the amount of attempts than the high school learners. On average, the younger students took 25.36 attempts (SD = 16.19), ranging from 3 to 59, when working on the challenging puzzle task. The participants in the highest scoring middle school camp cluster for the cCTt post-test used the most attempts, M = 31.80, SD = 21.43, ranging from 15 to 59 attempts. The participants in lowest scoring clusters used about the same amount of attempts. On average, the learners in the second lowest scoring cluster used 21 attempts (SD = 15.35) ranging from 3 to 37, and the lowest scoring cluster used 23.33 attempts (SD = 5.51) ranging from 18 to 29. These findings suggest that for our middle school participants that more attempts may be linked to higher CT knowledge.

The high school learners used less attempts than the middle school learners, M = 14.56, SD = 10.83, ranging from 4 to 50. Unlike the middle school camp, the highest scoring cluster for the cCTt post-test in the high school camp used the least attempts, M = 8.88, SD = 3.72, ranging from 4 to 16. The second highest scoring cluster used the second least amount of attempts, M = 13.67, SD = 6.92, ranging from 8 to 27. The lowest scoring cluster used the most attempts, M = 27.25, SD = 15.87, ranging from 16 to 50. These findings suggest that for our high school participants that less attempts use may be linked to higher CT knowledge.

5. DISCUSSION

Insights from the game play behaviors of low- and high-performing students can assist the development of wheel-spinning detectors and supports around persistence for students from similar backgrounds [28]. The results from SPM demonstrate that high performing learners at both camps, participants that scored the highest on the cCTt post-test, most frequently used strategies that align with characteristics of productive struggle and deliberate problem solving such as iteratively adding blocks to their BarrelBots inventory in subsequent steps [16], strategy revision [16], and increased time intervals between behaviors [5]. From the between-camp cSPADE analysis, despite the middle school participants having relatively higher supports for their frequent puzzle-solving patterns, they had lower supports for iterative puzzle solving patterns (e.g., Add \(\rightarrow \) Add and Add \(\rightarrow \) Add \(\rightarrow \) Add). The within camp clustering also found that the high performing learners at both camps most frequently used strategies that incrementally added blocks to their BarrelBots inventory in subsequent steps. These finding align with CT practices like being incremental and iterative [23]. The highest performing middle school cluster demonstrated in their most frequent problem-solving pattern that after iteratively adding blocks in early attempts they removed some blocks and added more, which may be an indication of a strategy revision.

The descriptive statistics for the average number of attempts and informal analysis of average player latency between subsequent inventory states after removing outliers for each cluster shows that middle school learners benefit from using more inventory states with less time in between attempts, while high performing clusters in the high school group used less states with more time in between attempts. Middle school participants in the highest performing cluster spent less time between attempts compared to the lowest scoring middle school cluster, 57.28 seconds versus 96.85. The high school learners in the highest performing cluster spent more time between attempts and used fewer attempts compared to the high school learners in the other clusters. Together with the SPM results, the relatively large amount of time between attempts and fewer attempts could indicate that high-performing high school participants were more deliberate with their attempts.

The low performing learners at both camps, participants that scored the lowest on the cCTt post-test, most frequently used problem-solving patterns that align with characteristics of wheel-spinning [16]. The presence of removing all blocks may be an indication of wheel-spinning. The participants from the middle school camp in the cSPADE analysis and lowest performing clusters from that camp both had removing all blocks as a state in their most frequent patterns. Middle school learners also used almost double the amount of attempts compared to participants in the high school camp. Together these findings indicate that the low-performing middle school learners may have experienced wheel-spinning, they persisted during the task but were more prone to giving up on their ideas causing them to spend extended time on the puzzle without meaningful progress. Although remove all was not a state present in the most frequent patterns for the high school learners, rearranging blocks in subsequent attempts was only used by the lowest-scoring cluster and could be indicative of non-productive game play behaviors. High school participants in the lowest performing cluster used more attempts and seemingly more non-deliberate actions compared to the highest scoring cluster when solving the difficult puzzle.

5.1 Limitations and Future Work

A limitation for our study is our relatively small sample size and choice to conduct our analyses on camps that took place in subsequent years, which could potentially alter our results. For our analyses we also solely rely on log data. Triangulation with other data sources like video and audio of students working on the task could enable us to find more nuanced results and could be a beneficial direction for the field.

Prior work has demonstrated the potential for large language models (LLMs) to provide meaningful, adaptive feedback to students within Minecraft-based learning environments [13]. Building on this foundation, our future work will leverage the indicators of persistence identified in this study, along with the strategies students employ during wheel-spinning, to inform the design of system prompts that enable LLMs to deliver targeted, conversational feedback. Overall, we are optimistic that these future supports, grounded by the findings from this study, will enhance student persistence, reduce wheel-spinning, and improve learning outcomes for students.

6. ACKNOWLEDGMENTS

The materials used in this study are based upon work supported by the National Science Foundation, Directorate for Education and Human Resources, and Institute of Education Sciences under Grants 1713609, 1906873, and 2229612.

7. REFERENCES

1. A. Abbott and A. Tsay. Sequence analysis and optimal matching methods in sociology: Review and prospect. Sociological methods & research, 29(1):3–33, 2000.
2. C. Angeli and M. Giannakos. Computational thinking education: Issues and challenges, 2020.
3. J. E. Beck and Y. Gong. Wheel-spinning: Students who fail to master a skill. In International conference on artificial intelligence in education, pages 431–440. Springer, 2013.
4. F. C. Billari. The analysis of early life courses: Complex descriptions of the transition to adulthood. Journal of population research, 18(2):119–142, 2001.
5. E. Bumbacher, S. Salehi, C. Wieman, and P. Blikstein. Tools for science inquiry learning: Tool affordances, experimentation strategies, and conceptual understanding. Journal of Science Education and Technology, 27(3):215–235, 2018.
6. B. Eichmann, S. Greiff, J. Naumann, L. Brandhuber, and F. Goldhammer. Exploring behavioural patterns during complex problem-solving. Journal of computer assisted learning, 36(6):933–956, 2020.
7. L. El-Hamamsy, M. Zapata-Cáceres, E. M. Barroso, F. Mondada, J. D. Zufferey, and B. Bruno. The competent computational thinking test: Development and validation of an unplugged computational thinking test for upper primary school. Journal of Educational Computing Research, 60(7):1818–1866, 2022.
8. L. El-Hamamsy, M. Zapata-Cáceres, E. Martín-Barroso, F. Mondada, J. D. Zufferey, B. Bruno, and M. Román-González. The competent computational thinking test (cctt): A valid, reliable and gender-fair test for longitudinal ct studies in grades 3–6: L. el-hamamsy et al. Technology, Knowledge and Learning, 30(3):1607–1661, 2025.
9. E. Fussell. Measuring the early adult life course in mexico: An application of the entropy index. Advances in Life Course Research, 9:91–122, 2005.
10. A. Gabadinho, G. Ritschard, N. S. Müller, and M. Studer. Analyzing and visualizing state sequences in r with traminer. Journal of statistical software, 40:1–37, 2011.
11. B. Hagan. Exploring persistence in computing education: An ordered network analysis of student engagement and persistence in a minecraft learning environment. Unpublished manuscript, 2024.
12. T.-C. Hsu, S.-C. Chang, and Y.-T. Hung. How to learn and how to teach computational thinking: Suggestions based on a review of the literature. Computers & Education, 126:296–310, 2018.
13. S. Hum, M. Gadbury, J. Ginger, K. Duda, and H. C. Lane. Barrelbots: Chatgpt feedback for middle school student creative minecraft artifacts. In International Conference on Artificial Intelligence in Education, pages 3–15. Springer, 2025.
14. S. Kai, M. V. Almeda, R. S. Baker, C. Heffernan, N. Heffernan, et al. Decision tree modeling of wheel-spinning and productive persistence in skill builders. Journal of Educational Data Mining, 10(1):36–71, 2018.
15. J. Kang, M. Liu, and W. Qu. Using gameplay data to examine learning behavior patterns in a serious game. Computers in Human Behavior, 72:757–770, 2017.
16. M. Kapur. Productive failure. Cognition and instruction, 26(3):379–424, 2008.
17. V. Kovanovic, D. Gašević, S. Dawson, S. Joksimovic, and R. Baker. Does time-on-task estimation matter? implications on validity of learning analytics findings. Journal of Learning Analytics, 2(3):81–110, 2015.
18. N. Kumar. How many people play minecraft in 2026 (player count). https://www.demandsage.com/minecraft-statistics/, 2026.
19. H. C. Lane, M. Gadbury, J. Ginger, S. Yi, N. Comins, J. Henhapl, and A. Rivera-Rogers. Triggering stem interest with minecraft in a hybrid summer camp. 2022.
20. W. J. M. Computational thinking’s influence on research and education for all. Italian Journal of Educational Technology, 25(2):7–14, 2017.
21. T. S. McNerney. From turtles to tangible programming bricks: explorations in physical language design. Personal and Ubiquitous Computing, 8:326–337, 2004.
22. F. Murtagh and P. Legendre. Ward’s hierarchical agglomerative clustering method: which algorithms implement ward’s criterion? Journal of classification, 31(3):274–295, 2014.
23. V. J. Shute, C. Sun, and J. Asbell-Clarke. Demystifying computational thinking. Educational research review, 22:142–158, 2017.
24. E. J. Slattery, P. Lehane, D. Butler, M. O’Leary, and K. Marshall. Assessing the benefits of digital game-based learning with minecraft in children, adolescents and young adults: A broad systematic review. Review of Education, 13(1):e70035, 2025.
25. J. Voogt, P. Fisser, J. Good, P. Mishra, and A. Yadav. Computational thinking in compulsory education: Towards an agenda for research and practice. Education and information technologies, 20(4):715–728, 2015.
26. K. D. Wang, H. Liu, D. DeLiema, N. Haber, and S. Salehi. Discovering players’ problem-solving behavioral characteristics in a puzzle game through sequence mining. In Proceedings of the 14th Learning Analytics and Knowledge Conference, pages 498–506, 2024.
27. J. M. Wing. Computational thinking. Communications of the ACM, 49(3):33–35, 2006.
28. G. K. Wong, S. Jian, and H.-Y. Cheung. Engaging children in developing algorithmic thinking and debugging skills in primary schools: A mixed-methods multiple case study. Education and Information Technologies, 29(13):16205–16254, 2024.

© 2026 Copyright is held by the author(s). This work is distributed under the Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International (CC BY-NC-ND 4.0) license.