Using Student Trace Logs To Determine Meaningful Progress and Struggle During Programming Problem Solving
Abstract: Over the years, researchers have studied novice programming behaviors when doing assignments and projects to identify struggling students. Much of these efforts focused on using student programming and interaction features to predict student success at a course level. While these methods are effective at early detection of struggling students in the long run, there is also a need to identify struggling students during an assignment so that we can provide proactive intervention to prevent unproductive struggle and frustration. This work proposes a data-driven method that uses student trace logs to identify struggling moments during a programming assignment and determine the appropriate time for an intervention. We define a struggling moment as not achieving significant progress within a certain amount of time, relative to the amount of progress made and time taken in a sample student dataset. The paper describes how we determine significant progress and a time threshold for struggling students. We validated our algorithm’s classification of struggling and progressing moments with experts rating whether they believe an intervention is needed for a sample of 20% of the dataset. The result shows that our automatic struggle detection method can accurately detect struggling students with less than 2 minutes of work with over 77% estimated accuracy. Our work contributes significantly to building proactive immediate support features for intelligent programming environments.