Toward Improving Student Model Estimates through Assistance Scores in Principle and in Practice
Abstract: Student modeling is especially useful in educational research and instructional development, as well as intelligent tutoring systems due to a capability to estimate latent student attributes. While the widely used approaches, such as AFM or BKT, have shown satisfactory results, they can only handle binary outcomes, which makes the models inflexible and results in potential information loss. In this work, we proposed a new modeling approach, PC-AFM, to support polytomous outcomes, namely the amount of assistance a student needs. Because student errors and hint requests may not only derive from their ability, but also from non-cognitive factors (e.g., they may game the system), we wanted to first test PC-AFM on synthetic data where this source of noise is not present. We confirm that PC-AFM is indeed better than AFM in recovering the true student and knowledge component (KC) parameters and even predicts student error rates better even though estimates are based on assistance score. Applying the approach to six real-world datasets, we found that PC-AFM consistently outperforms AFM in reliable estimation of KC parameters and produces better generalization to new students (which requires better KC estimates). However, consistent with the hypothesis that student assistance behavior is driven by motivational or meta-cognitive factors beyond their ability, we found that PC-AFM was not better in reliable estimation of student parameters nor in generalization across items (which requires accurate student estimates). This paper also illustrates a set of strategies for addressing the general problem of how to compare alternative measurement models for the same desired latent outcome.