ABSTRACT

Educational AI systems often assume that behavioral failure patterns in student data reveal latent semantic structures useful for tutoring. We rigorously audit this assumption using 15.8 million student interactions from UK secondary mathematics. By correlating a 50-dimensional SVD-based behavioral manifold with neural semantic embeddings across 1,869 questions, we identify a critical Semantic Gap: a near-zero correlation (\(r=0.000\)) demonstrating that behavioral patterns are functionally independent of semantic content. This gap causes systematic Topical Drift in naive RAG systems, where behavioral-only retrieval performs below random chance (12.3% pairwise win rate). To bridge this gap, we introduce the principle of Semantic Primacy and a hybrid Manifold-Augmented Generation (MAG) architecture. Our results show that MAG achieves superior diagnostic specificity (\(4.78/5\)) compared to zero-shot baselines by enforcing semantic filtering before behavioral personalization. Finally, we characterize the Semantic Opacity of behavioral clusters through expert audit, proving the necessity of architectural integration over manual curation. This work establishes a new baseline for understanding behavioral-semantic orthogonality in educational data mining.

Keywords

1. INTRODUCTION

Educational AI systems currently face a profound grounding problem. While foundation models generate pedagogically plausible responses, they often lack diagnostic specificity, leaving them unable to distinguish surface-level errors from deep, structural misconceptions [8]. This blindness leads to a reliance on linguistic patterns rather than pedagogical ones. Recent work has turned to behavioral manifolds: clustering questions by how students fail to reveal diagnostic structures invisible to semantic analysis [17]. The hypothesis is compelling: if students make identical errors across different problems, those problems likely share a behavioral attractor, a latent misconception that transcends the wording of the prompt in question.

Recent systematic evaluations [8] expose the depth of this diagnostic blindness and demonstrate that foundation models consistently conflate procedural fluency with conceptual understanding, praising student responses that exhibit surface-level correctness while harboring fundamental misconceptions. A model might validate a student who correctly applies the quadratic formula yet cannot explain why the discriminant determines the number of solutions. This creates a pedagogical hazard: students receive affirmation for brittle knowledge that crumbles under transfer. The issue is not linguistic coherence but diagnostic precision. Without grounding in actual student failure patterns, even the most eloquent model remains pedagogically shallow.

However, this behavioral manifold hypothesis rests on an untested assumption of alignment. Do interaction patterns from student responses preserve the semantic coherence necessary for retrieval in agentic teaming? If behavioral clustering merely captures difficulty levels or generic test-taking patterns rather than conceptual relationships, systems using these patterns will retrieve topically irrelevant context. This Topical Drift undermines the agent’s reliability, creating a mismatch where a student struggling with negative numbers receives hints about fractions simply because both show similar failure rates.

In this work, we rigorously audit the prevailing architectural standard of linear embeddings. While non-linear techniques are emerging, the vast majority of scalable educational platforms default to linear methods like SVD-based matrix factorization for real-time recommendation due to their computational frugality and interpretability. This focus on linear methods is not arbitrary. Singular Value Decomposition (SVD) based approaches dominate real-time educational platforms precisely because they provide the computational speed required for responsive tutoring at scale. Platforms serving millions of students require sub-second response times for hint generation and content sequencing. SVD decomposes sparse interaction matrices into
low-dimensional representations that enable rapid
nearest-neighbor search without expensive recomputation.

Intelligent tutoring systems, adaptive learning platforms, and recommendation engines from Khan Academy to Carnegie Learning rely on variants of this approach. Given this
widespread deployment, auditing the semantic fidelity of linear behavioral embeddings directly impacts the pedagogical quality experienced by students daily. If these methods systematically fail to preserve topic coherence, millions of learners receive contextually inappropriate feedback through architectures designed for efficiency rather than semantic grounding. We treat these methods as the empirical baseline to investigate the following research questions (RQs):

• RQ1: Do linear projections of binary correctness preserve semantic similarity when measured against modern neural embeddings?
• RQ2: What are the practical consequences of semantic-behavioral orthogonality for retrieval-based tutoring systems?
• RQ3: Can architectural integration, such as Manifold-Augmented Generation (MAG), leverage orthogonal signals to outperform high compute zero-shot models?
• RQ4: Do behavioral patterns exhibit Semantic Opacity, rendering them unreadable to human experts and necessitating computational stabilization?

Using a massive dataset of 15.8 million student interactions from UK secondary mathematics, we construct a 50-dimensional behavioral manifold via SVD. This finding implies that Behavioral Sightedness and Semantic Understanding are orthogonal dimensions of the learning experience. We demonstrate that this gap leads to systematic failure in naive retrieval systems, where behavioral-only retrieval frequently retrieves topically irrelevant hints. However, we propose that this independence is a design opportunity rather than a defect.

By enforcing Semantic Primacy, a hybrid architecture that filters by topic before personalizing by behavior, we can leverage these orthogonal signals to catch invisible errors that linguistic models miss. Finally, we bridge the gap between data mining and the learning sciences through a qualitative Expert Tagging study to evaluate whether these manifolds represent human-readable pedagogical traps. This work advances what we term the Sighted-Frugal paradigm.

Rather than pursuing diagnostic precision through ever larger foundation models, we demonstrate that architectural constraint can achieve superior specificity at a fraction of the computational cost. The prevailing assumption in educational AI has been that pedagogical failures stem from insufficient model capacity. This fuels an arms race toward parameter scaling, where each generation of models grows more expensive to train and deploy. We challenge this assumption. Our results show that a modest model like GPT-4o-mini, when augmented with properly structured behavioral context, outperforms far larger zero-shot alternatives in diagnostic tasks. The key is not scale but grounding.

By respecting the orthogonality between behavioral and semantic dimensions, we can build systems that are simultaneously more accurate and more accessible. This matters for equity. Computationally intensive models concentrate educational innovation in well resourced institutions. Sighted-Frugal architectures democratize access to diagnostic AI.

Our primary contributions are as follows:

1. Quantification of the Semantic Gap: We provide the first large-scale empirical evidence that linear behavioral embeddings common in deployed systems are functionally independent of semantic content, providing a theoretical ceiling for naive RAG systems.
2. The Principle of Semantic Primacy: We establish a core design requirement for educational RAG: systems must strictly filter by semantic content before applying behavioral personalization to avoid pedagogical drift.
3. Demonstration of Sighted-Frugal AI: We validate a hybrid Manifold-Augmented Generation (MAG) architecture that outperforms high compute zero-shot models in diagnostic specificity, offering a path toward more equitable and sustainable educational AI.

2. RELATED WORK

2.1 Behavioral Data for Misconception Detection

Mathematics education research has long argued that effective instruction requires identifying systematic error patterns rather than mere correctness [9]. Cognitive evidence suggests that these behavioral signatures reflect underlying conceptual structures [2]. Recent large-scale efforts, such as the Eedi dataset [3] and diagnostic benchmarks from Wang et al. [17], have enabled the computational mining of these misconceptions.

The Gap: While these works provide the raw material for diagnostic AI, they often assume that behavioral clusters are inherently semantically meaningful. Our work explores and tests this assumption by auditing whether the clusters discovered in these datasets actually align with the conceptual categories recognized by expert educators and semantic models.

2.2 Knowledge Tracing and Interaction Modeling

Knowledge Tracing (KT) has evolved from Bayesian frameworks [4] to deep recurrent architectures such as RNNs [11] capable of predicting student mastery from high-dimensional interaction logs.

The Gap: These models are traditionally optimized for predictive performance (i.e., will a student get the next problem right?). However, being able to predict a failure does not necessarily mean the model understands the semantic nature of that failure. Thus, we aim to move beyond prediction to investigate whether the latent spaces created by KT-like interaction modeling are sufficiently grounded in content to support retrieval and generation in tutoring applications, specifically for mathematics-based coursework.

2.3 Retrieval-Augmented Generation (RAG)in Education

RAG systems improve generation by retrieving external context based on semantic similarity [7, 13]. In education, this has roots in retrieving worked examples for Cognitive Tutors [14] and early automated hint generation from historical data [1].

The Gap: Standard educational RAG assumes that semantic similarity (how a problem is worded) is the best proxy for retrieval. Conversely, behavioral systems assume that failure patterns are the best proxy. To our knowledge, no prior study has measured the alignment between these two signals. We identify the Semantic Gap between these dimensions, necessitating the shift toward our proposed Semantic Primacy architecture.

2.4 Model Alignment and Diagnostic Precision

General-purpose foundation models are aligned for helpfulness and safety using techniques like RLHF [10] and DPO [12]. However, recent evaluations show that this alignment often lacks diagnostic specificity, causing models to provide polite but pedagogically vague feedback [8].

The Gap: Effective feedback requires precision [5], yet models often fall into a Black Box of Competence. We investigate whether incorporating behavioral manifolds as retrieval context can provide the Sightedness necessary to restore diagnostic precision without the heavy compute costs of model retraining.

2.5 Triangulated Evaluation Methodologies

The rise of LLM-as-a-Judge [18] has enabled scalable evaluation, but educational contexts require more rigorous, multi-dimensional validation to ensure replicability and improved metacognition [15].

The Pivot: We avoid reliance on a single metric by employing a triangulated methodology. We combine automated neural scoring with a qualitative Expert Tagging component to test for Semantic Opacity. This ensures that our findings are grounded not just in statistical correlation, but in pedagogical reality.

3. METHODOLOGY

3.1 Data Sources and Problem Formulation

We utilize the NeurIPS 2020 Education Challenge dataset [17], comprising 15.8 million interactions from the Eedi platform (UK secondary mathematics, ages 11-18). To ground our behavioral manifold in expert pedagogy, we cross-reference this with the 2024 Mining Misconceptions dataset [3], which provides expert-verified labels for common student errors.

From an initial pool of 27,613 questions, we isolated a core set of 1,869 questions with expert labels. For our comparative agentic experiments, we focused on 242 questions that met the threshold for high diagnostic utility: a minimum of 500 student interactions and a failure rate of at least 40%. This subset was divided into a development set (\(n=120\)) and a held-out evaluation set (\(n=122\)).

3.2 Auditing the Behavioral Manifold (SVD)

To represent the industry standard for scalable educational AI, we constructed a behavioral manifold using (SVD). We generated a sparse binary interaction matrix \(M\), where \(M_{i,j} = 1\) denotes a correct response from student \(i\) to question \(j\).

We reduced this matrix to a 50-dimensional space, capturing the latent patterns of student success and failure. While modern deep learning techniques exist, we intentionally selected SVD to audit the specific linear assumptions prevalent in real-time production systems. To identify structural attractors within this space, we applied spectral clustering [16] with a nearest-neighbors affinity graph (\(n\)=10), identifying \(k = 50\) clusters. This choice was based on domain considerations rather than optimization, though silhouette analysis yielded a score of \(-\)0.050, reflecting the pedagogically realistic property that trap clusters are not geometrically isolated from the broader manifold. Behavioral coherence was instead validated through downstream cluster statistics: identified trap clusters exhibited consistent failure rates and substantial interaction volumes, confirming that spectral clustering captured psychometric structure rather than geometric separation.

We prioritized pedagogical interpretability over statistical optimization: clusters with clean separation (high silhouette scores) but mixed difficulty profiles would be computationally elegant yet pedagogically meaningless. Our resulting clusters exhibited behavioral coherence, questions within each cluster shared similar failure rates and discrimination parameters, validating that spectral clustering captured psychometric structure rather than random noise. Critically, this behavioral coherence did not translate to semantic coherence, as subsequent correlation analysis would reveal.

While alternative cluster counts were not systematically explored, the resulting structure proved pedagogically interpretable: identified trap clusters exhibited consistent failure rates (40 to 51 percent) and substantial interaction volumes (1000 plus interactions per question), while mastery clusters demonstrated clear success patterns (success rates exceeding 70 percent). This behavioral coherence, despite the lack of formal hyperparameter tuning, suggests the clustering captured meaningful pedagogical structure rather than statistical noise.

3.3 Quantifying the Semantic Gap

To measure the alignment between behavior and content, we computed semantic embeddings for all questions using Sentence-BERT (S-BERT) [13]. We validated that these 384-dimensional vectors captured pedagogical relationships (e.g., questions with identical misconception labels showed significantly higher cosine similarity, \(p < 0.001\)).

The Semantic Gap was measured by calculating the correlation between behavioral similarity (computed as \(1/(1+d)\) where \(d\) is the Euclidean distance in the 50D SVD space) and semantic similarity (cosine similarity in the 384D Sentence-BERT space). We employed Pearson’s \(r\), Spearman’s \(\rho \), and permutation testing to ensure that the observed independence was not a result of difficulty-based confounds.

This operationalization requires explicit justification. Euclidean distance in the SVD-reduced space represents the standard metric for behavioral nearest neighbor search in production recommendation systems: it directly measures the dissimilarity used by retrieval algorithms when selecting contextually similar problems. Cosine similarity in neural embedding space serves as our semantic proxy because S-BERT explicitly optimizes for sentence-level semantic similarity, and we validated that it captures pedagogical relationships (questions sharing misconception labels show significantly higher similarity). The correlation between these distance measures tests whether behavioral neighborhoods preserve semantic structure: if they did, questions close in behavioral space would also be close semantically, yielding positive correlation. Our near-zero result indicates behavioral proximity provides no semantic information.

3.4 Agentic Architectures and Semantic Primacy

We compared four tutoring architectures to evaluate how different retrieval strategies impact the Sightedness of the agent. All systems utilized GPT-4o-mini as the generation backbone throughout, ensuring that differences in performance were attributable solely to the retrieval logic. GPT-4o was used exclusively as the evaluation judge for System D (see Section 3.5), not as a generator in any system.

• System A (Zero-Shot): A high compute baseline using GPT-4o-mini without external context, representing the ungrounded generation.
• System B (Semantic RAG): A standard RAG approach that retrieves the five most topically similar questions based on S-BERT embeddings.
• System C (Behavioral RAG): A behavior-only approach that retrieves the nearest neighbor in the 50D SVD space, testing if failure patterns alone provide sufficient diagnostic context. This single-neighbor retrieval reflects how behavioral-only systems operate in practice, and tests whether even minimal behavioral context provides diagnostic value.

• System D (MAG: Semantic Primacy): Our proposed hybrid architecture. It operates via a two-stage Filter-then-Personalize process:

  1. Semantic Filter: The system identifies the top 20 most topically relevant questions across the dataset.
  2. Behavioral Personalization: Within this semantically safe subset, the system selects the top 5 questions nearest in distance within the behavioral manifold.

3.5 Evaluation and Pedagogical Constraint

Ensuring diagnostic precision requires pedagogical guardrails that prevent common failure modes such as false validation, shallow feedback, or premature solution revelation. We employed carefully engineered prompts that emphasize conceptual scaffolding and diagnostic specificity. All systems used identical prompt templates, ensuring that performance differences could be attributed solely to retrieval strategy rather than linguistic variation in generation.

Beyond prompt engineering, we investigated whether preference learning could offer complementary benefits. We fine-tuned Llama-3-8B-Instruct using Direct Preference Optimization (DPO) [12] with 600 contrastive pairs constructed from 120 anchor questions. Each anchor generated five preference pairs corresponding to distinct pedagogical boundaries: politeness bias (empty affirmation masking diagnostic gaps), shallow diagnosis (symptom labeling without mechanistic explanation), geometric imprecision (conflating structurally adjacent misconceptions), premature answer-giving (violating the Zone of Proximal Development), and technical opacity (exposing retrieval mechanics to learners). Training employed LoRA adapters [6] (r=16, \(\alpha \)=32) over three epochs, explicitly teaching rejection boundaries through contrastive supervision. Rejected responses were systematically constructed to violate each pedagogical boundary: for politeness bias, rejected hints provided affirmation without diagnostic reasoning; for shallow diagnosis, they identified error symptoms without mechanistic explanation; for answer-giving, they revealed solutions rather than scaffolding discovery. Chosen responses demonstrated diagnostic precision aligned with each boundary constraint.

Comparative evaluation across all 122 test questions yielded a decisive result: the preference-aligned model achieved 2.96/5 quality, significantly trailing GPT-4o-mini within the MAG architecture (4.75/5¹, \(\Delta \) = -1.79, p \(<\) 0.001). Across 122 pairwise comparisons, the DPO model never outperformed the baseline (0% win rate). This empirical outcome validates a central tenet of the Sighted-Frugal paradigm: architectural constraint applied to capable foundation models dominates explicit alignment of smaller alternatives. The performance gap suggests that model capacity and structured retrieval provide greater pedagogical leverage than parameter-efficient supervision signals, at least within the scale regime tested here.

Final evaluation employed triangulated measurement:

1. Automated Scoring: Hint quality was evaluated on a five-point diagnostic scale across 122 questions using GPT-4o as judge for System D and GPT-4o-mini as judge for baseline systems A, B, and C. Statistical robustness was ensured through 10,000 bootstrap resamples, yielding confidence intervals for all effect sizes.
2. False Validation Detection: A rule-based classifier flagged instances of inappropriate praise, where systems validated incorrect student reasoning. This provided an objective safety check independent of neural evaluation.
3. Expert Audit: Two mathematics educators with secondary teaching experience evaluated cluster interpretability (Semantic Opacity) and pedagogical utility of generated hints. Inter-rater reliability metrics (Cohen’s \(\kappa \), ICC) quantified alignment between computational and human assessments.

3.6 Reproducibility

This study is designed for full replication. We used publicly available datasets: the NeurIPS 2020 Education Challenge (15.8 million interactions) and Eedi Mining Misconceptions (expert-labeled questions). Our pipeline implements standard techniques with documented parameters: 50-component SVD for the behavioral manifold, spectral clustering with nearest-neighbors affinity (\(n\)=10, \(k\)=50), and Sentence-BERT embeddings (all-MiniLM-L6-v2, 384 dimensions). The MAG architecture performs two-stage retrieval (top-20 semantic, top-5 behavioral) with clearly specified similarity thresholds. We will release complete code, including SVD construction, clustering, RAG implementation, and evaluation protocols, via GitHub at https://github.com/Rickygole/semantic-gap-educational-rag.

4. RESULTS

4.1 The Semantic Gap: Quantifying Orthogonality

Our primary analysis across 1,869 expert-labeled questions confirms the existence of the Semantic Gap: a state of functional independence between how students fail and what a problem says. As shown in Figure 2, we observed a near-zero correlation between behavioral similarity and semantic similarity (Pearson r \(\approx \) 0.000, 95% CI [\(-\)0.002, 0.001], p = 0.861; Spearman \(\rho \) = \(-\)0.002, p = 0.004).

Permutation testing (\(1000\) iterations) confirmed that this correlation falls entirely within the null distribution (\(p = 0.872\)). This near-zero correlation has direct retrieval consequences. Even weak positive correlation (r = 0.2 to 0.3) would allow behavioral signals to improve precision: most neighbors would fall within the correct semantic domain. Near-zero correlation means behavioral proximity provides no information about semantic relevance. A retrieved question is no more likely to be topically related than random selection. Worse, behavioral retrieval operates deterministically rather than randomly, systematically selecting off-topic problems with similar difficulty profiles. This transforms neutral noise into structured misdirection: students receive confident, detailed hints about the wrong mathematical domain.

We ruled out difficulty confounds through stratified analysis on 433 questions meeting high utility criteria (500+ interactions, 40%+ failure rate). Correlations remained near zero within difficulty tertiles: easy items (r = \(-\)0.02), moderate items (r = \(-\)0.01), and challenging items (r = 0.01). The Gap persists independent of psychometric difficulty. Stratified analysis confirmed that the Semantic Gap persists independently of psychometric item difficulty across all tertiles. Even within tightly constrained problem families such as triangle area calculations, behavioral clustering scattered questions across the manifold based on difficulty rather than geometric properties.

This suggests the Gap reflects a fundamental limitation of binary correctness signals rather than an artifact of aggregating heterogeneous content.

Interpretation: This visualization demonstrates that behavioral neighborhoods are semantically heterogeneous. A student’s failure pattern on a Negative Numbers question may be behaviorally identical to their failure on a Fractions question. Knowing the behavioral attractor (how they failed) provides zero information regarding the semantic content (what they were failing at). This confirms our hypothesis that LLMs are behaviorally blind and SVD manifolds are semantically opaque.

4.2 Systemic Deficits and the Semantic Primacy Rescue

The practical consequences of this Gap are substantial for naive retrieval systems. As detailed in Table 1, System C (Behavioral Retrieval) achieved a pairwise win rate of only 12.3% falling below the 33.3% random baseline for a four-system pairwise tournament (p < 0.001). This failure is driven by Topical Drift: retrieving behaviorally similar but topically irrelevant context. Consider a diagnostic example from our evaluation set. When presented with a student struggling to subtract negative integers, System C retrieved a question about simplifying algebraic fractions. The only shared attribute was a 52 percent failure rate. The behavioral manifold correctly identified both as difficult items requiring procedural precision, yet this psychometric similarity provided zero pedagogical value.

The student received worked examples about fraction manipulation when they needed conceptual grounding in the number line. System D, by contrast, first filtered to arithmetic operations involving integers, then selected behaviorally similar problems within that semantic boundary. The resulting hint addressed sign confusion in subtraction, a structurally analogous trap within the correct mathematical domain. This contrast illustrates how Topical Drift transforms behavioral patterns from diagnostic signals into pedagogical noise.

However, our Semantic Primacy architecture (System D) successfully bridged this gap. By first filtering for topical relevance and then personalizing via the behavioral manifold, MAG achieved the highest quality (\(4.78/5\)) with the lowest variance (\(\sigma = 0.49\)). MAG demonstrates substantial improvements with effect sizes of d = 0.560 versus zero-shot (p < 0.001), d = 0.935 versus semantic RAG (p < 0.001), and d = 1.421 versus behavioral RAG (p < 0.001). These medium to large effects confirm that architectural integration of orthogonal signals yields measurable gains in diagnostic precision. Prompt sensitivity analysis across 30 test cases verified that behavioral RAG’s deficit persists regardless of prompt engineering, isolating topical drift as the causal mechanism rather than suboptimal instruction design.

4.3 Semantic Opacity: The Human-in-the-Loop Barrier

Qualitative audit by two mathematics educators revealed that the behavioral manifold exhibits high Semantic Opacity. While the manifold identifies statistically robust clusters, human experts struggled to assign them consistent pedagogical labels (Cohen’s \(\kappa = 0.40\)). Furthermore, inter-rater reliability on the quality of behaviorally-derived hints was poor (\(ICC = -0.25\)).

Consider Question 514 as a concrete illustration. Students solving a triangle area problem systematically substituted slant height for perpendicular height, a dimensional confusion sharing behavioral signatures with other problems involving oblique versus orthogonal measurements. Semantic RAG retrieved topically proximate geometry questions, yet failed to surface this specific error pattern. The behavioral manifold, however, identified structurally analogous traps across disparate topics. When constrained by semantic filtering, this behavioral signal elevated hint quality by 2.0 points on our diagnostic scale, revealing misconceptions that pure linguistic similarity missed entirely. The structural trap here is subtle. Students treating slant height as equivalent to perpendicular height are not making a calculation error but a dimensional category error. They fail to recognize that area formulas require measurements orthogonal to the base, not measurements along an oblique edge.

This confusion appears across multiple geometric contexts: using hypotenuse length in right triangle area calculations, applying diagonal measurements in rectangle problems, or substituting arc length for radius in circular sectors. Semantic RAG cannot detect this pattern because the questions use different vocabulary and geometric shapes. The behavioral manifold, however, identifies the shared failure signature: students systematically choose the most visually salient measurement rather than the geometrically appropriate one. This is precisely the kind of latent misconception that benefits from computational detection paired with semantic constraint.

This finding suggests that student failure patterns contain latent traps that are mathematically detectable but pedagogically non-obvious. This necessitates the computational stabilization provided by our MAG architecture: rather than asking teachers to curate these clusters, we use semantic embeddings to ground the behavioral signal in human-readable content.

5. DISCUSSION AND LIMITATIONS

5.1 The Semantic Gap as a Structural Property

Our central finding identifies a fundamental structural reality: linear behavioral embeddings derived from binary correctness are functionally independent of semantic content. We argue that this Semantic Gap is not a temporary engineering hurdle that can be solved with more data or refined clustering; rather, it is an intrinsic property of what these representations capture.

Linear methods like SVD extract variance from correctness matrices that primarily reflect psychometric properties: item difficulty, student discrimination, and test-taking behaviors (e.g., fatigue or guessing). While these patterns are robust, they do not encode semantic relationships. A question on negative numbers and a question on fractions may share a behavioral attractor due to identical failure rates, but they possess zero conceptual overlap. This explains the substantial deficit of behavioral-only retrieval (12.3% pairwise win rate): the system lacks a mechanism to distinguish between a shared difficulty level and a shared concept. The 50-dimensional SVD space can be understood through the lens of classical test theory.

Each dimension likely corresponds to latent factors governing item response patterns: the first few dimensions capture global difficulty (analogous to the b-parameter in Item Response Theory), subsequent dimensions encode discrimination power (the a-parameter, measuring how sharply performance separates high and low ability students), and later dimensions reflect guessing patterns and test-taking strategies. However, these psychometric constructs are domain-agnostic. A dimension encoding requires multi-step reasoning applies equally to fraction division and quadratic factoring, collapsing conceptually distinct topics into a single behavioral signature. This is why behavioral distance preserves psychometric structure but destroys semantic boundaries.

The manifold encodes how hard questions are, not what they are about. This psychometric-semantic decoupling has profound implications for educational data mining. Traditional approaches to content recommendation implicitly assume that behavioral signals contain topic information that students who struggle similarly are struggling with similar concepts. Our findings falsify this assumption. The behavioral manifold captures a parallel dimension of the learning experience: not conceptual understanding, but cognitive load, procedural complexity, and metacognitive demand. These are pedagogically valuable signals, but only when grounded within semantic boundaries.

5.2 Semantic Primacy: A Restorative Design Principle

The success of our hybrid MAG model (\(4.78/5\)) establishes Semantic Primacy as a core design principle for educational RAG. Our results prove that systems must strictly filter by semantic content before applying behavioral personalization. Attempting to learn semantics from interaction patterns alone is a category error that leads to Topical Drift.

This principle offers immediate practical value for Frugal AI deployment. It allows developers to leverage behavioral data where it is strongest identifying specific failure modes within a topic while relying on semantic embeddings for what they do best: grounding the conversation in the correct mathematical domain. The independence we discovered is thus not a bug, but a feature that can be exploited through deliberate architectural constraint. The computational efficiency of this approach has direct equity implications. GPT-4o-mini operates at roughly one-tenth the inference cost of GPT-4o while achieving superior diagnostic performance when augmented with behavioral context. This cost differential is not academic.

In under-resourced educational settings where per-student budgets constrain technology adoption, the difference between 10 cents and 1 cent per interaction determines whether AI tutoring is feasible at scale. Sighted-Frugal architectures that combine modest models with structured data enable equitable access to diagnostic AI. The alternative, pursuing diagnostic precision through ever-larger foundation models, concentrates educational innovation in wealthy institutions while leaving resource-constrained schools reliant on generic chatbots. Our results demonstrate that architectural intelligence can substitute for parameter scale, democratizing access to pedagogically grounded AI.

5.3 Semantic Opacity and the Necessity of Computation

The observed Semantic Opacity (\(ICC = -0.25\)) indicates that the patterns captured by the behavioral manifold are human-unreadable. Because these patterns reflect psychometric variance rather than pedagogical categories, human educators cannot reliably recognize them through traditional pedagogical intuition.

This finding necessitates a shift in how we approach a Human-in-the-Loop design. Manual curation of behavioral clusters is likely infeasible. Instead, we must use computational stabilization like the MAG architecture to translate these opaque behavioral signals into semantically grounded, pedagogically useful hints. This bridges the gap between high-dimensional data mining and the practical needs of the classroom. This Semantic Opacity finding reframes the role of AI in educational systems. Rather than replacing teacher judgment, computational methods should function as translators between statistical patterns and pedagogical action.

Teachers cannot intuitively recognize that a triangle area problem shares behavioral signatures with cylinder volume calculations, yet the manifold detects this structural analogy through shared error patterns. The AI does not make pedagogical decisions but surfaces hidden relationships for human interpretation. This positions the system as an augmentation tool rather than an autonomous agent, a distinction critical for teacher trust and adoption. The design implication is clear: behavioral insights must always be presented within semantic containers that educators can evaluate and override.

5.4 Boundaries of the Linear Baseline

As a rigorous audit, this study is bounded by its focus on Linear Methods and Binary Correctness. While SVD is the industry standard for scalable systems, it represents the empirical baseline rather than the theoretical ceiling. It remains an open and vital question whether nonlinear techniques such as Variational Autoencoders (VAEs), Graph Neural Networks (GNNs), or Attention Mechanisms could bridge the Semantic Gap that linear methods cannot.

Furthermore, our use of binary correctness provides a “maximally impoverished” input. Future work should investigate if richer behavioral signals, such as response latencies, multi-attempt sequences, or option-choice analysis, might preserve semantic structure even under linear projection. Finally, while our dataset of 15.8 million interactions provides significant power, it is situated within the UK secondary mathematics curriculum. Cross-context replication is essential to determine if the Semantic Gap fluctuates across different cultures, age groups, or domains.

5.5 Methodological Reflection and Triangulation

We acknowledge the limitations inherent in using "LLM-as-a-Judge" to evaluate hint quality. System D was evaluated using GPT-4o while baseline systems were evaluated using GPT-4o-mini. We further acknowledge that the behavioral space (50D SVD) and semantic space (384D Sentence-BERT) use different underlying geometries, though both are transformed into comparable similarity measures before correlation; this choice does not affect our near-zero correlation finding, which holds across Pearson, Spearman, and permutation testing. To mitigate evaluation circularity, we employed a triangulated evaluation strategy, incorporating educator audits and rule-based classifiers to detect false validation. While these measures provide a robust signal for diagnostic specificity, definitive claims regarding student learning outcomes will require future longitudinal deployment studies in live classrooms.

By identifying the Semantic Gap, we provide a theoretical and practical framework for the next generation of educational AI. We have shown that the Sightedness required for effective tutoring does not come from larger models, but from architectures that respect the independence of behavior and semantics. By enforcing Semantic Primacy, we can move toward a Sighted-Frugal AI that is pedagogically precise, computationally efficient, and ethically grounded.

6. CONCLUSION

This paper identifies and quantifies the Semantic Gap: a state of empirical independence between student interaction patterns and semantic content. Through an audit of 15.8 million interactions, we demonstrated that while linear behavioral embeddings are functionally blind to topic, they capture robust psychometric properties that define how students struggle.

We proved that this Gap leads to systematic failure in naive RAG systems, which succumb to Topical Drift when retrieving based on behavioral signals alone. However, we have established a restorative design principle: Semantic Primacy. By enforcing a hybrid architecture that filters by topic before personalizing by behavior, our Manifold-Augmented Generation (MAG) system outperforms even high-compute zero-shot models in diagnostic specificity.

Our discovery of Semantic Opacity further suggests that behavioral data contains latent traps invisible to human pedagogical intuition, necessitating the computational stabilization provided by our MAG architecture. Ultimately, this work provides the first large-scale baseline for understanding the relationship between student behavior and content semantics. By respecting the independence structure of educational data, we can move toward a new generation of AI tutors that are not just linguistically fluent, but diagnostically sighted, computationally frugal, and pedagogically just.

7. FUTURE DIRECTIONS

While this study establishes a rigorous linear baseline, it opens several critical avenues for the evolution of Sighted-Frugal AI. We highlight four directions that we believe warrant systematic investigation by the research community.

Future research should investigate whether nonlinear architectures such as Variational Autoencoders (VAEs), Graph Neural Networks (GNNs), or Attention Mechanisms can bridge the Semantic Gap that linear methods cannot. Graph Neural Networks appear particularly promising given the relational structure of educational data. A message-passing architecture where nodes represent questions, edges encode co-occurrence in student trajectories, and node features combine behavioral statistics with semantic embeddings could learn joint representations. Specifically, a Graph Attention Network (GAT) with multi-head attention could weight connections by both behavioral similarity (questions students fail together) and semantic proximity (questions sharing misconception labels), potentially discovering hybrid distances that respect both dimensions. The question is whether such architectures can learn the Semantic Primacy constraint implicitly or whether architectural enforcement remains necessary.

We utilized binary correctness as a “maximally impoverished” input. Testing whether the inclusion of response latencies, multi-attempt sequences, and process traces can close the Gap even under linear projection remains a vital open question. Specific process signals merit investigation: response latency distributions (whether students pause before errors, suggesting confusion versus rapid incorrect attempts suggesting procedural errors), edit patterns (whether students erase and rewrite, indicating metacognitive awareness), attempt sequences (whether retry strategies differ across misconceptions), and navigation paths (whether students seek help resources differentially by trap type). These temporal and behavioral traces may encode semantic information absent from binary outcomes. For instance, students who pause extensively before answering geometry questions incorrectly may be struggling with visualization rather than calculation, a distinction invisible to correctness alone but potentially detectable through interaction timing.

Mapping the Semantic Gap across diverse domains, age groups, and cultural curricula will determine if this orthogonality is a universal property of educational data or a curriculum-specific phenomenon. Cross-context replication is essential to establish the generality of our findings beyond UK secondary mathematics.

Transitioning from automated quality metrics to longitudinal classroom studies is essential to measure how Semantic Primacy impacts actual skill acquisition and misconception resolution. A feasible deployment protocol would involve a randomized controlled trial in middle school algebra classrooms over one academic semester. Students in the treatment condition would receive MAG-generated hints when struggling with linear equations and systems of equations, while control students would receive standard computer-based practice with generic feedback. Pre and post assessments measuring both procedural fluency and conceptual understanding would quantify learning gains. Critically, the study would track misconception persistence: whether students who receive geometrically grounded hints demonstrate faster resolution of specific error patterns (e.g., sign errors in equation solving, confusion between solutions and equations) compared to controls. Implementation would require integration with existing learning management systems to ensure ecological validity and teacher buy-in.

Finally, larger educator cohorts from several distinct domains, particularly social sciences, humanities, and computer science education, would further probe Semantic Opacity and support the development of interfaces that help experts interpret and engage productively with behavioral manifolds.

8. REFERENCES

1. T. Barnes and J. Stamper. Toward automatic hint generation for logic proof tutoring using historical student data. In International Conference on Intelligent Tutoring Systems, pages 373–382, 2008.
2. S. Bugden and D. Ansari. Individual differences in children’s mathematical competence are related to the intentional but not automatic processing of Arabic numerals. Cognition, 118(1):32–44, 2011.
3. S. Bulathwela, H. Muse, and E. Yilmaz. Eedi: Mining misconceptions from school children at scale. In LAK23: 13th International Learning Analytics and Knowledge Conference, pages 617–623, 2023.
4. A. T. Corbett and J. R. Anderson. Knowledge tracing: Modeling the acquisition of procedural knowledge. User Modeling and User-Adapted Interaction, 4(4):253–278, 1994.
5. J. Hattie and H. Timperley. The power of feedback. Review of Educational Research, 77(1):81–112, 2007.
6. E. J. Hu, Y. Shen, P. Wallis, Z. Allen-Zhu, Y. Li, S. Wang, L. Wang, and W. Chen. LoRA: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685, 2021.
7. P. Lewis, E. Perez, A. Piktus, F. Petroni, V. Karpukhin, N. Goyal, H. Küttler, M. Lewis, W. Yih, T. Rocktäschel, et al. Retrieval-augmented generation for knowledge-intensive NLP tasks. In Advances in Neural Information Processing Systems, volume 33, pages 9459–9474. Curran Associates, Inc., Red Hook, NY, 2020.
8. Z. Liu, I. Papadimitriou, G. Wang, Z. A. Pardos, M. Fleck, and A. Mitrovic. Discerning minds or generic tutors? Systematic evaluation of large language models in mathematics tutoring. arXiv preprint arXiv:2508.06583, 2025.
9. P. Nesher. Towards an instructional theory: The role of student’s misconceptions. For the Learning of Mathematics, 7(2):33–40, 1987.
10. L. Ouyang, J. Wu, X. Jiang, D. Almeida, C. L. Wainwright, P. Mishkin, C. Zhang, S. Agarwal, K. Slama, A. Ray, et al. Training language models to follow instructions with human feedback. In Advances in Neural Information Processing Systems, volume 35, pages 27730–27744. Curran Associates, Inc., Red Hook, NY, 2022.
11. C. Piech, J. Bassen, J. Huang, S. Ganguli, M. Sahami, L. J. Guibas, and J. Sohl-Dickstein. Deep knowledge tracing. In Advances in Neural Information Processing Systems, volume 28, pages 505–513. Curran Associates, Inc., Red Hook, NY, 2015.
12. R. Rafailov, A. Sharma, E. Mitchell, S. Ermon, C. D. Manning, and C. Finn. Direct preference optimization: Your language model is secretly a reward model. In Advances in Neural Information Processing Systems, volume 36, pages 53728–53741. Curran Associates, Inc., Red Hook, NY, 2023.
13. N. Reimers and I. Gurevych. Sentence-BERT: Sentence embeddings using Siamese BERT-networks. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing, pages 3982–3992, 2019.
14. S. Ritter, J. R. Anderson, K. R. Koedinger, and A. Corbett. Cognitive tutor: Applied research in mathematics education. Psychonomic Bulletin & Review, 14(2):249–255, 2007.
15. J. Stamper, K. Koedinger, R. S. Baker, A. Skogsholm, B. Leber, S. Demi, S. Yu, and D. Spencer. Managing the educational dataset lifecycle with DataShop. In International Conference on Artificial Intelligence in Education, pages 557–559, 2012.
16. U. Von Luxburg. A tutorial on spectral clustering. Statistics and Computing, 17(4):395–416, 2007.
17. S. Wang, A. Ororbia, Z. Wu, K. Williams, I. Arroyo, B. Woolf, and N. Heffernan. Diagnostic questions: The NeurIPS 2020 education challenge. arXiv preprint arXiv:2007.12061, 2020.
18. L. Zheng, W. Chiang, Y. Sheng, S. Zhuang, Z. Wu, Y. Zhuang, Z. Lin, Z. Li, D. Li, E. P. Xing, et al. Judging LLM-as-a-judge with MT-Bench and Chatbot Arena. arXiv preprint arXiv:2306.05685, 2023.

© 2026 Copyright is held by the author(s). This work is distributed under the Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International (CC BY-NC-ND 4.0) license.