Introduction to the Proceedings

The Georgia Institute of Technology is proud to host the seventeenth International Conference on Educational Data Mining (EDM) in Atlanta, Georgia, July 14-July 17, 2024. EDM is the annual flagship conference of the International Educational Data Mining Society. This year’s theme is “New tools, new prospects, new risks – educational data mining in the age of generative AI.” The theme focuses on the movement from descriptive and predictive models to generative artificial intelligence (AI) and what that means for learning environments and processes. While the new methods unlock exciting new potentials for educational data mining, they also foreground many ethical considerations and risks that are associated with all types of machine learning and artificial intelligence. This year, we additionally welcomed research in the following areas: mitigating biases and harms that may result from model use, accounting for the stereotypes that are inherent to the large models that drive generative AI, separating the hype surrounding these new technologies from their potential in educational settings, and finding ways to use these models to better understand learning processes and support learning.

The scientific programming for EDM2024 includes

• Two keynote talks by outstanding researchers in the field, Tanja Käser (EPFL, Switzerland) and Carolyn Rosé (CMU, USA)

• A plenary Test of Time award talk by Gautam Biswas (Vanderbilt University, USA)

• A plenary Data Set award talk by Jakub Kužílek (Humboldt-Universität zu Berlin & German Research Center for Artificial Intelligence, Germany) and Martin Hlosta (Institute for Distance Learning and eLearning Research & Swiss Distance University of Applied Sciences, Switzerland)

• Five tutorials (foundational as well as advanced)

• Five workshops

• Twelve paper presentation sessions on the topics of large language models in education (two sessions), affective computing, learning analytics and recommender systems, reinforcement learning and pedagogical agents, knowledge tracing and curricula, collaborative learning, prediction and supervised learning, research practices (two sessions), and computer science education (two sessions)

• Two poster presentation sessions

• An industry track and industry panel

• A doctoral student consortium

The tutorials are: (1) Logistic Knowledge Tracing Tutorial, (2) Promoting Open Science in Educational Data Mining, (3) Thinking Causally in EDM, (4) Beyond Tutor Logs: Utilizing Sensor Data for Measuring Student Behavior, and (5) Tools for Planning and Analyzing Randomized Controlled Trials and A/B Tests. The workshops are: (1) 8th Educational Data Mining in Computer Science Education (CSEDM) Workshop, (2) Human-Centric eXplainable AI in Education (HEXED) Workshop, (3) Causal Inference in Educational Data Mining, (4) Leveraging Large Language Models for Next-Generation Educational Technologies, and (5) Educational Data Mining in Writing and Literacy Instruction.

The venue is the Global Learning Center of Georgia Tech, home of Georgia Tech’s Division of Lifetime Learning. The Global Learning Center is nestled in Tech Square, a burgeoning tech hub in Midtown Atlanta attached to the Georgia Tech Hotel & Conference Center and across the street from the Tech Square Research Building and the Coda Building, home of many of the faculty of the institute’s College of Computing. EDM 2024 received 81 submissions to the full papers track (10 pages), 68 to the short papers track (6 pages), and 29 to the poster and demo track (4 pages). The program committee accepted 21 full papers (26 The EDM 2024 industry track and the industry panel fostered exchange between industry application research and basic research. Four papers and two posters were included in the industry track. Panelists included Kristen DiCerbo (Khan Academy), Diego Zapata-Rivera (ETS Research Institute), Lewis Johnson (Alelo), and Bibi Groot (EeDI). EDM 2024 also continued its tradition of providing opportunities for young researchers to present their work and receive feedback from their peers and senior researchers. The doctoral consortium this year features 12 such participants. EDM 2024 is especially proud to offer travel sponsorships to 15 students who will attend EDM thanks to this support. We thank the sponsors of EDM 2024 for their generous support. We thank all the authors who submitted their work and the program committee members for their expert inputs. We thank the members of the organization committee for their leadership that made this conference possible. And, a big Thank You to the local organizing committee who made this event memorable.

July 14th, 2024
Atlanta, GA, USA

General Chairs

• David Joyner – Georgia Institute of Technology, USA

Program Chairs

• Benjamin Paaßen (they/them) – Bielefeld University, Germany

• Carrie Demmans Epp (she/her) – University of Alberta, Canada

Equity, Diversity, and Inclusion Chairs

• Anna Rafferty (she/her) – Carleton College, USA

• Jie Tang (he/him) – Tsinghua University, China

Accessibility Chairs

• Nigel Bosch (he/him) – University of Illinois Urbana-Champaign, USA

Industry Track Chairs

• Carol M. Forsyth (she/her) – Educational Testing Service, USA

• Avi Segal (he/him) – Ben-Gurion University of the Negev, Israel

Poster & Demo Track Chairs

• Heeryung Choi (she/her) – University of Michigan, USA

• Irena Koprinska (she/her) – The University of Sydney, Australia

Doctoral Consortium Chairs

• Neil Heffernan – Worcester Polytechnic Institute, USA

• Luc Paquette (he/him) – University of Illinois Urbana-Champaign, USA

JEDM Track Chairs

• Maria Mercedes T. Rodrigo (she/her) – Ateneo de Manila University, Philippines

• Agathe Merceron (she/her) – Berlin Hochschule für Technik, Germany

• Jaclyn Ocumpaugh – University of Pennsylvania, USA

Workshop & Tutorials Chairs

• Bita Akram (she/her) – North Carolina State University, USA

• Sergey Sosnovsky – Utrecht University, Netherlands

Awards Chairs

• Danielle McNamara – Arizona State University, USA

• Cristóbal Romero – University of Córdoba, Spain

• Marianne Winslett – University of Illinois Urbana-Champaign, USA

Scholarship Chairs

• Luc Paquette (he/him) – University of Illinois Urbana-Champaign, USA

• Ryan S. Baker – University of Pennsylvania, USA

Student Volunteer Chair

• Sherry Sahebi (she/her) – University at Albany – SUNY, USA

Social Media & Publicity Chairs

• Oleksandra Poquet – Technical University of Munich, Germany

• Jill-Jênn Vie (he/him) – Inria Saclay, France

Web Chairs

• Paul Salvador Inventado (he/him) – California State University Fullerton, USA

• Ramkumar Rajendran (he/him) – Indian Institute of Technology Bombay, India

• Rafael D. Araújo (he/him) – Federal University of Uberlândia, Brazil

Proceedings Chairs

• Mirko Marras (he/him) – University of Cagliari, Italy

• Maomi Ueno (he/him) – The University of Electro-Communications, Japan

Sponsorship Chairs

• Hannah Moon (she/her) – Georgia Institute of Technology, USA

Online Experience Chair

• Nick Lytle (he/him) – Georgia Institute of Technology, USA

Local Organizing Team

• David Joyner – Georgia Institute of Technology, USA

• Hannah Moon (she/her) – Georgia Institute of Technology, USA

• Nick Lytle (he/him) – Georgia Institute of Technology, USA

• Alex Duncan (he/him) – Georgia Institute of Technology, USA

IEDMS Officers

IEDMS Board of Directors

Senior Program Committee Members

Program Committee Members

Generalizable and Interpretable Models of Learning

Tanja Käser, EPFL School of Computer and Communication Sciences

Modeling learners’ knowledge, behavior, and strategies is at the heart of educational technology. Learner models serve as a basis for adapting the learning experience to students’ needs and supporting teachers in classroom orchestration. Consequently, a large body of research has focused on creating accurate models of student knowledge and behaviors. However, current modeling approaches are still limited: they are either defined for specific and well-structured domains (e.g., algebra, vocabulary learning) requiring substantial work from experts and limiting generalizability, or they lack interpretability. Recent advances in generative AI, in particular large language models (LLMs), have the potential to address these constraints. However, LLMs lack alignment with educational goals and a grounded knowledge.

In this talk, I will discuss the key challenges in developing generalizable and explainable models, and our solutions to address them, including models tracking learning in open-ended environments and generalizing between different environments and populations. I will present our work on explainable AI, including a rigorous evaluation of existing approaches, the development of inherently interpretable models, as well as studies on effectively communicating model explanations. Finally, I will show some of our recent results combining “traditional” modeling approaches and LLMs to provide interpretable feedback and explanations while not compromising on model trustworthiness.

Opportunities and Challenges for LLM Agent-Based Support for Collaborative Design

Carolyn Rosé, Professor of Language Technologies and Human-Computer Interaction, School of Computer Science, Carnegie Mellon, USA

Supporting collaborative design is an ideal context for exploring the capabilities and limitations of LLM-based conversational agents. The ability to extract information in context and produce a coherent sounding text can be used to generate reflection triggers. In two recent studies, we have employed LLM-based conversational agents with the goal of triggering human reflection and learning during collaborative software design. As humans engage in collaborative design, they employ their own abilities to reason abstractly, to decompose problems, and apply principles productively. Reflection is a valuable activity for promoting human learning in these settings. However, what humans are able to do in terms of abstraction and reasoning as part of their creative problem solving is precisely what is most difficult for LLM agents to do. In contrast to claims of “super-human performance” in the media, in this talk we will explore the complementarity of human intelligence and Artificial Intelligence. We will begin with results of a classroom study where LLM-based conversational agent support for collaborative software development was successful in increasing student learning. From there we will move on to argue in favor of a research agenda for exploiting the complementarity both in terms of applying AI capabilities to the betterment of human learning as well as inspiring further extension of technical capabilities from insights derived from observation of human reflection and learning in collaborative design.

Test of Time Award Talk

Assessing Student Learning in Open Ended Learning Environments From Sequential to Multimodal Data Analysis

Gautam Biswas, Professor of Computer Science, Vanderbilt University, Nashville, TN. USA

From my early days as an AIED and EDM researcher, I have focused on understanding how students learn, especially in scenarios where they have to construct and apply their knowledge to problem-solving tasks. Collaborating with peers, we developed open-ended learning environments (OELEs) where K-12 students build scientific models and apply them to solve real-world problems. Challenges arise, as students have to navigate with multiple tools in the computer-environments. Some students overcome these challenges to become effective learners while others struggle to progress often applying suboptimal learning strategies. John Kinnebrew and I began analyzing learners’ activity logs to study these differences, resulting in the Differential Sequence Mining algorithm, which earned us the best paper award at EDM 2012. Expanding on this work, we developed the Contextualized Difference Mining method for understanding students’ learning behaviors, for which we are receiving this Test of Time award.

In my talk, I will review our work on Differential sequence mining, and explore its applications in understanding students’ cognitive and metacognitive learning behaviors. We have leveraged these insights to provide adaptive support, helping students’ progress in our Open-Ended Learning Environments (OELEs). Beyond this, we have employed other sequential representations, such as Markov Chains and Hidden Markov Models, to analyze students’ activities and behaviors in the context of their learning and problem-solving tasks. Beyond this, our OELEs have advanced to facilitate students’ integrated learning of science, computing, and engineering problem-solving, including collaborative efforts in computer-based and embodied learning scenarios. From these richer learning environments, I will share insights into our latest efforts involving multimodal data analysis, incorporating video, speech, and activity logs. Using vision-based deep learning models and large language models (LLMs), we integrate analyses across modalities, offering a comprehensive understanding of students’ collaborative learning and problem-solving activities. In conclusion, I will discuss the potential implications of our work on shaping future of learning in classrooms.

EDM data set award

Jakub Kužílek

Senior Researcher, Computer Science Education / Computer Science and Society research group, Humboldt-Universität zu Berlin & German Research Center for Artificial Intelligence, Berlin, Germany.

Bio. Jakub Kužílek is affiliated with the Computer Science Education / Computer Science and Society research group at Humboldt-Universität zu Berlin and the Educational Technology Lab at the German Research Center for Artificial Intelligence (DFKI) as a senior researcher. His research investigates student self-regulated learning within online learning environments, collaborative group work, adaptive assessments, and feedback within the context of digital education. In the past, he developed (together with Martin Hlosta and Zdenek Zdrahal) an OU Analyse system used to support 200.000 students of the Open University (United Kingdom) during their studies and founded learning analytics research at Czech Technical University. He has led (and currently is doing research within) a project on AI use in assessment feedback at Humbold-Universität. In parallel, he is leading the project on AI-driven recommendation systems in vocational education (KIPerWeb at DFKI).

Martin Hlosta

Anand Deshpande, Founder and Chairman, Persistent Systems, IN

Bio. Martin Hlosta is a Senior Researcher at the Institute for Distance Learning and eLearning Research (IFeL) at Swiss distance university of applied sciences (FFHS). Before joining FFHS, he led research and development of OUAnalyse at The Open University (OU) – a Predictive Learning Analytics project deployed in all undergraduate courses, improving student retention and teachers practice. It is one of the world-largest deployment of analytics systems in education and in 2020 it was selected by UNESCO as one of the four best projects using AI in education. His following work focused on identifying factors contributing to large gaps of disadvantaged students in the UK, and in another study presented how using predictive analytics by teachers in an online course can lower these gaps for students coming from low Socio-Economic areas. Currently, he is leading research and teaching in Learning Analytics at FFHS and works on various strands how learning analytics can improve feedback. His most recent project funded by Unity and Meta to target inequalities in education explores how immersive Virtual Reality and enhanced analytics for reflection can help future teachers in South Africa.

The Knowledge Component Attribution Problem for Programming: Methods and Tradeoffs with Limited Labeled Data

Understanding students’ learning of knowledge components (KCs) is an important educational data mining task and enables many educational applications. However, in the domain of computing education, where program exercises require students to practice many KCs simultaneously, it is a challenge to attribute their errors to specific KCs and, therefore, to model student knowledge of these KCs. In this paper, we define this task as the KC attribution problem. We first demonstrate a novel approach to addressing this task using deep neural networks and explore its performance in identifying expert-defined KCs (RQ1). Because the labeling process takes costly expert resources, we further evaluate the effectiveness of transfer learning for KC attribution, using more easily acquired labels, such as problem correctness (RQ2). Finally, because prior research indicates the incorporation of educational theory in deep learning models could potentially enhance model performance, we investigated how to incorporate learning curves in the model design and evaluated their performance (RQ3). Our results show that in a supervised learning scenario, we can use a deep learning model, code2vec, to attribute KCs with a relatively high performance (AUC > 75% in two of the three examined KCs). Further using transfer learning, we achieve reasonable performance on the task without any costly expert labeling. However, the incorporation of learning curves shows limited effectiveness in this task. Our research lays important groundwork for personalized feedback for students based on which KCs they applied correctly, as well as more interpretable and accurate student models.

Automated Evaluation of Classroom Instructional Support with LLMs and BoWs: Connecting Global Predictions to Specific Feedback

With the aim to provide teachers with more specific, frequent, and actionable feedback about their teaching, we explore how Large Language Models (LLMs) can be used to estimate “Instructional Support” domain scores of the CLassroom Assessment Scoring System (CLASS), a widely used observation protocol. We design a machine learning architecture that uses either zero-shot prompting of Meta’s Llama2, and/or a classic Bag of Words (BoW) model, to classify individual utterances of teachers’ speech (transcribed automatically using OpenAI’s Whisper) for the presence of Instructional Support. Then, these utterance-level judgments are aggregated over a 15-min observation session to estimate a global CLASS score. Experiments on two CLASS-coded datasets of toddler and pre-kindergarten classrooms indicate that (1) automatic CLASS Instructional Support estimation accuracy using the proposed method (Pearson R up to 0.48) approaches human inter-rater reliability (up to R = 0.55); (2) LLMs generally yield slightly greater accuracy than BoW for this task, though the best models often combined features extracted from both LLM and BoW; and (3) for classifying individual utterances, there is still room for improvement of automated methods compared to human-level judgments. Finally, (4) we illustrate how the model’s outputs can be visualized at the utterance level to provide teachers with explainable feedback on which utterances were most positively or negatively correlated with specific CLASS dimensions.

An Approach to Improve k-Anonymization Practices in Educational Data Mining

With the aim to provide teachers with more specific, frequent, and actionable feedback about their teaching, Educational data mining has allowed for large improvements in educational outcomes and understanding of educational processes. However, there remains a constant tension between educational data mining advances and protecting student privacy while using educational datasets. Publicly available datasets have facilitated numerous research projects while striving to preserve student privacy via strict anonymization protocols (e.g., k-anonymity); however, little is known about the relationship between anonymization and utility of educational datasets for downstream educational data mining tasks, nor how anonymization processes might be improved for such tasks. We provide a framework for strictly anonymizing educational datasets with a focus on improving downstream performance in common tasks such as student outcome prediction. We evaluate our anonymization framework on five diverse educational datasets with machine learning-based downstream task examples to demonstrate both the effect of anonymization and our means to improve it. Our method improves downstream machine learning accuracy versus baseline data anonymization by 30.59%, on average, by guiding the anonymization process toward strategies that anonymize the least important information while leaving the most valuable information intact.

Exploring the Impact of Symbol Spacing and Problem Sequencing on Arithmetic Performance: An Educational Data Mining Approach

Experimental research on perception and cognition has shown that inherent and manipulated visual features of mathematics problems impact individuals’ problem-solving behavior and performance. In a recent study, we manipulated the spacing between symbols in arithmetic expressions to examine its effect on 174 undergraduate students’ arithmetic performance but found results that were contradictory to most of the literature (Closser et al., 2023). Here, we applied educational data mining (EDM) methods to that dataset at the problem level to investigate whether inherent features of the 32 experimental problems (i.e., problem composition, problem order) may have caused unintended effects on students’ performance. We found that students were consistently faster to correctly simplify expressions with the higher-order operator on the left, rather than right, side of the expression. Furthermore, average response times varied based on the symbol spacing of the current and preceding problem, suggesting that problem sequencing matters. However, including or excluding problem identifiers in analyses changed the interpretation of results, suggesting that the effect of sequencing may be impacted by other, undefined problem-level factors. These results advance cognitive theories on perceptual learning and provide implications for educational researchers: online experiments designed to investigate students’ performance on mathematics problems should include a variety of problems, systematically examine the effects of problem order, and consider applying different data analysis approaches to detect effects of inherent problem features. Moreover, EDM methods can be a tool to identify nuanced effects on behavior and performance in the context of data from online platforms.

Effect of Gamification on Gamers: Evaluating Interventions for Students Who Game the System

Gaming the system is a persistent problem in Computer-Based Learning Platforms. While substantial progress has been made in identifying and understanding such behaviors, effective interventions remain scarce. This study explores the impact of two types of interventions – gamification and manipulation of assistance access – on the learning outcomes of students who tend to game the system using a method of causal moderation known as Fully Latent Principal Stratification. The results indicate that gamification does not consistently mitigate these negative behaviors. One gamified condition had a consistently positive effect on learning regardless of students’ propensity to game the system, whereas the other had a negative effect on such students. However, delaying access to hints and feedback may have a positive effect on the learning outcomes of those gaming the system. This paper also illustrates the potential of integrating detection and causal methodologies within education data mining for understanding how to respond to behaviors effectively after they are detected.

LearnSphere: A Learning Data and Analytics Cyberinfrastructure

LearnSphere is a web-based data infrastructure designed to transform scientific discovery and innovation in education. It supports learning researchers in addressing a broad range of issues including cognitive, social, and motivational factors in learning, educational content analysis, and educational technology innovation. LearnSphere integrates previously separate educational data and analytic resources developed by participating institutions. The web-based workflow authoring tool, Tigris, allows technical users to contribute sophisticated analytic methods, and learning researchers can adapt and apply those methods using graphical user interfaces, importantly, without additional programming. As part of our use-driven design of LearnSphere, we built a community through workshops and summer schools on educational data mining. Researchers interested in particular student levels or content domains can find student data from elementary through higher-education and across a wide variety of course content such as math, science, computing, and language learning. LearnSphere has facilitated many discoveries about learning, including the importance of active over passive learning activities and the positive association of quality discussion board posts with learning outcomes. LearnSphere also supports research reproducibility, replicability, traceability, and transparency as researchers can share their data and analytic methods along with links to research papers. We demonstrate the capabilities of LearnSphere through a series of case studies that illustrate how analytic components can be combined into research workflow combinations that can be developed and shared. We also show how open web-accessible analytics drive the creation of common formats to streamline repeated analytics and facilitate wider and more flexible dissemination of analytic tool kits.

Session-based Methods for Course Recommendation

In higher education, academic advising is crucial to students’ decision-making. Data-driven models can benefit students in making informed decisions by providing insightful recommendations for completing their degrees. To suggest courses for the upcoming semester, various course recommendation models have been proposed in the literature using different data mining techniques and machine learning algorithms utilizing different data types. One important aspect of the data is that usually, courses taken together in a semester fit well with each other. If there is no correlation between the co-taken courses, students may find it more difficult to handle the workload. Based on this insight, we propose using session-based approaches to recommend a set of well-suited courses for the upcoming semester. We test three session-based course recommendation models, two based on neural networks (CourseBEACON and CourseDREAM) and one on tensor factorization (TF-CoC). Additionally, we propose a postprocessing approach to adjust the recommendation scores of any base course recommender to promote related courses. Using metrics capturing different aspects of the recommendation quality, our experimental evaluation shows that session-based methods outperform existing popularity-based, association-based, similarity-based, factorization-based, neural networks-based, and Markov chain-based recommendation approaches. Effective course recommendations can result in improved student advising, which, in turn, can improve student performance, decrease dropout rates, and a more positive overall student experience and satisfaction.

Confusion, Conflict, Consensus: Modeling Dialogue Processes During Collaborative Learning with Hidden Markov Models

There is growing recognition that AI technologies can, and should, support collaborative learning. To provide this support, we need models of collaborative talk that reflect the ways in which learners interact. Great progress has been made in modeling dialogue for high school and college-age learners, but the dialogue processes that characterize collaborative talk between elementary learner dyads are not currently well understood. This paper reports on a study with elementary school learners (4th and 5th grade, ages 9–11 years old) coded collaboratively in dyads. We recorded dialogue from 22 elementary school learner dyads, covering 7594 total utterances. We labeled this corpus manually with dialogue acts and then induced a hidden Markov model to identify the underlying dialogue states and the transitions between these states. The model identified six distinct hidden states which we interpret as Social Dialogue, Confusion, Frustrated Coordination, Exploratory Talk, Directive & Disagreement, and Disagreement & Self-Explanation. The HMM revealed that when students entered into a productive exploratory talk state, the primary way they transitioned out of this state is when they became confused or reached an impasse. When this occurred, the learners then moved into states of disputation and conflict before re-entering the Exploratory Talk state. These findings can inform the design of AI agents who support young learners’ collaborative talk and help agents determine when students are conflicting rather than collaborating.