Move your lamp post: Recent data reflects learner knowledge better than older data

April Galyardt, Ilya Goldin

Abstract


In educational technology and learning sciences, there are multiple uses for a predictive model of whether a student will perform a task correctly or not. For example, an intelligent tutoring system may use such a model to estimate whether or not a student has mastered a skill. We analyze the significance of data recency in making such predictions, i.e., asking whether relatively more recent observations of a student's performance matter more than relatively older observations. We investigate several representations of recency, such as the count of prior practice in the AFM model, and the proportion of recent successes with exponential and box kernels. We find that an exponential decay of a proportion of successes provides the summary of recent practice with the highest predictive accuracy over alternative models. As a secondary contribution, we develop a new logistic regression model, Recent-Performance Factors Analysis, that leverages this representation of recent performance, and has higher predictive accuracy than existing logistic regression models.


Full Text:

PDF

References


ADAMS , R. J., WILSON , M., and WANG , W.- C . 1997. The multidimensional random coefficients multinomial logit model. Applied psychological measurement 21, 1, 1–23.

AKAIKE , H. 1973. Information theory and an extension of the maximum likelihood principle. In Proceedings of Second International Symposium on Information Theory, B. Petrov and F. Caski, Eds. Akademiai Kiado, Budapest, 267–281.

AKAIKE , H. 1985. Prediction and entropy. In A Celebration of Statistics, A. Atkinson and S. Fienberg, Eds. Springer: New York, 1–24.

ALEVEN , V. and KOEDINGER , K. R. 2000. Limitations of student control: Do students know when they need help? In Intelligent Tutoring Systems, G. Gauthier, C. Frasson, and K. VanLehn, Eds. Vol. 1839. Springer Berlin Heidelberg, Berlin, Heidelberg, 292–303.

BAKER , R. S., CORBETT, A. T., KOEDINGER , K. R., and WAGNER , A. Z. 2004. Off-task behavior in the cognitive tutor classroom: when students game the system. In Proceedings of SIGCHI conference on Human factors in computing systems. ACM, 383–390.

BAKER , R. S., GOLDSTEIN , A. B., and HEFFERNAN , N. T. 2011. Detecting learning moment-bymoment. In IJAIED. Vol. 21. 5–25.

BATES , D., MAECHLER , M., BOLKER , B., and WALKER , S. 2013. lme4: Linear mixed-effects models using eigen and s4. Computer Program.

BECK , J. E. and CHANG , K.- M . 2007. Identifiability: A fundamental problem of student modeling. In User Modeling 2007, C. Conati, K. McCoy, and G. Paliouras, Eds. Number 4511 in Lecture Notes in Computer Science. Springer Berlin Heidelberg, 137–146.

BECK , J. E., CHANG , K.- M ., MOSTOW, J., and CORBETT, A. 2008. Does help help? Introducing the Bayesian evaluation and assessment methodology. In Intelligent Tutoring Systems, B. P. Woolf, E. A¨ımeur, R. Nkambou, and S. Lajoie, Eds. Vol. 5091. Springer, Berlin, 383–394.

BOOTS , B., SIDDIQI , S. M., and GORDON , G. J. 2011. Closing the learning-planning loop with predictive state representations. The International Journal of Robotics Research 30, 7 (June), 954–966.

CEN , H., KOEDINGER , K., and JUNKER , B. 2006. Learning Factors Analysis – a general method for cognitive model evaluation and improvement. In Proc of 8th ITS Conf, M. Ikeda, K. D. Ashley, and T.-W. Chan, Eds. Vol. 4053. Springer Berlin Heidelberg, Berlin, Heidelberg, 164–175.

CHI , M., KOEDINGER , K., GORDON , G., JORDAN , P., and VAN LEHN , K. 2011. Instructional factors analysis: A cognitive model for multiple instructional interventions. In Proceedings of 4th International Conference on Educational Data Mining.

CORBETT, A. T. and ANDERSON , J. R. 1995. Knowledge tracing: Modeling the acquisition of procedural knowledge. User Modeling and User-Adapted Interaction 4, 4, 253–278. DE BOECK , P. and WILSON , M., Eds. 2004. Explanatory item response models: a generalized linear and nonlinear approach. Springer, New York.

FALAKMASIR , M. H., PARDOS , Z. A., GORDON , G. J., and BRUSILOVSKY, P. 2013. A spectral learning approach to knowledge tracing. In Proceedings of 6th International Conference on Educational Data Mining., S. K. D’Mello, R. A. Calvo, and A. Olney, Eds. Memphis, TN, 28–34.

FALAKMASIR , M. H., YUDELSON , M., RITTER , S., and KOEDINGER , K. 2015. Spectral bayesian knowledge tracing. In Proceedings of the 8th International Conference on Educational Data Mining., O. C. Santos, J. G. Boticario, C. Romero, M. Pechenizkiy, A. Merceron, P. Mitros, J. M. Luna, C. Mihaescu, P. Moreno, A. Hershkovitz, S. Ventura, and M. Desmarais, Eds. Madrid, Spain, 360– 364.

FISCHER , G. H. 1973. The linear logistic test model as an instrument in educational research. Acta Psychologica 37, 359–374.

GALYARDT, A. and GOLDIN , I. 2014. Recent-Performance Factors Analysis. In Proceedings of 7th International Conference on Educational Data Mining, J. Stamper, Z. Pardos, M. Mavrikis, and B. McLaren, Eds. 411–412. (Poster paper).

GOLDIN , I. and GALYARDT, A. 2015a. Viz-R: Using Recency to Improve Student and Domain Models. In Proceedings of Second (2015) ACM Conference on Learning @ Scale. L@S ’15. ACM, New York, NY, USA, 417–420.

GOLDIN , I. M. and GALYARDT, A. 2015b. Convergent validity of a student model: Recent-Performance Factors Analysis. In Proceedings of 8th International Conference on Educational Data Mining. Madrid, Spain.

GOLDIN , I. M., KOEDINGER , K. R., and ALEVEN , V. A. W. M. M. 2012. Learner Differences in Hint Processing. In Proceedings of 5th International Conference on Educational Data Mining, K. Yacef, O. Zaiane, A. Hershkovitz, M. Yudelson, and J. Stamper, Eds. Chania, Greece, 73–80.

GONG , Y., BECK , J. E., and HEFFERNAN , N. T. 2011. How to construct more accurate student models: Comparing and optimizing knowledge tracing and performance factor analysis. International Journal of Artificial Intelligence in Education 21, 1, 27–46.

GONZALEZ-BRENES , J., HUANG , Y., and BRUSILOVSKY, P. 2014. General features in knowledge tracing: Applications to multiple subskills, temporal item response theory, and expert knowledge. In Proceedings of 7th International Conference on Educational Data Mining, J. Stamper, Z. Pardos, M. Mavrikis, and B. McLaren, Eds. London, England, 84–91.

HEFFERNAN , N. T. and HEFFERNAN , C. L. 2014. The ASSISTments ecosystem. IJAIE 24, 4 (Dec.), 470–497.

JUNKER , B. W. 2011. Modeling hierarchy and dependence among task responses in educational data mining. In Handbook of Educational Data Mining. Chapman & Hall/CRC.

KASER , T., KOEDINGER , K., and GROSS , M. 2014. Different parameters-same prediction: An analysis of learning curves. In Proceedings of 7th International Conference on Educational Data Mining. London, UK.

KHAJAH , M. M., HUANG , Y., GONZALEZ-BRENES , J., MOZER , M. C., and BRUSILOVSKY, P. 2014. Integrating knowledge traching and item response theory: A tale of two frameworks. In Fourth International Workshop on Personalization Approaches in Learning Environments (PALE 2014), M. Kravcik, O. C. Santos, and J. G. Boticario, Eds. 7–15.

KHAJAH , M. M., WING , R. M., LINDSEY, R. V., and MOZER , M. C. 2014. Integrating latent-factor and knowledge-tracing models to predict individual differences in learning. In Proceedings of 7th International Conference on Educational Data Mining, J. Stamper, Z. Pardos, M. Mavrikis, and B. McLaren, Eds. London, England.

PAVLIK , P. I., CEN , H., and KOEDINGER , K. 2009. Performance Factors Analysis–a new alternative to Knowledge Tracing. In Proceedings of 14th International Conference on Artificial Intelligence in Education. IOS Press, 531–538.

PAVLIK , P. I., YUDELSON , M., and KOEDINGER , K. R. 2015. A measurement model of microgenetic transfer for improving instructional outcomes. International Journal of Artificial Intelligence in Education, 1–34.

RUPP, A., TEMPLIN , J., and HENSON , R. 2010. Diagnostic Measurement: Theory, Methods and Applications. Guilford Press, New York, NY.

STONE , M. 1977. An asymptotic equivalence of choice of model by cross-validation and Akaike’s criterion. Journal of the Royal Statistical Society. Series B (Methodological) 39, 1, 44–47.

WASSERMAN , L. 2006. All of Nonparametric Statistics. Springer, New York, NY.

YUDELSON , M., HOSSEINI , R., VIHAVAINEN , A., and BRUSILOVSKY, P. 2014. Investigating automated student modeling in a Java MOOC. In Proceedings of 7th International Conference on Educational Data Mining. London, UK.


Refbacks

  • There are currently no refbacks.