Abstract: High-stakes digital-first assessments are assessments that can be taken anytime and anywhere in the world and their scores impact test takers' lives. Computational psychometrics, a blend of theory-driven psychometrics and data-driven algorithms, provides the theoretical underpinnings for these data-rich assessments. The unprecedented flexibility, complexity, and high-stakes nature of these digital-first assessments poses enormous quality assurance challenges. In order to ensure these assessments meet both “the contest and the measurement” requirements of high-stakes tests, it is necessary to conduct continuous pattern monitoring and be able to promptly react when needed. In this paper, we illustrate the development of a quality assurance system, Analytics for Quality Assurance in Assessment (AQuAA), for a high-stakes and digital-first assessment. To build the system, educational data from continuous administrations of the assessments are mined, modeled and monitored via an interactive dashboard.