The effects of a personalized recommendation system on students’ high-stakes achievement scores: A field experiment
Abstract: This study examines data from a field experiment examining the effects of a personalized recommendation algorithm for suggesting videos after students complete mini-assessments available on an intelligent virtual learning environment (IVLE) for Algebra. The end users of this system are students enrolled in an Algebra 1 course in middle and high schools, and the system is used both during and out of school time. The objective of the recommendation algorithm is to increase student preparation to take the state-mandated End-of-Course (EOC) Algebra 1 assessment at the end of the school year. The algorithm is based on a Markov Decision Process informed by the students’ responses to a series of mini-assessment tests. The current study randomly assigned 16,406 students to either treatment or control conditions, which were blind to both student and teachers. The results indicate that the effects of the recommendation algorithm depend on the level of usage of students, showing significant improvements on EOC test scores of students who have a moderate level of usage. However, there was no effect for low usage students. Also, the study shows that students’ practice with the mini-assessment tests, available on the IVLE, exhibit a small but significant effect on improving students’ End-of-Course test scores, irrespective of the usage level. Finally, the study provides insights on challenges posed for implementing personalized recommendation algorithms at a large scale related to student self-regulation and teacher orchestration of technology use in the classroom.