ABSTRACT

Formative assessment in mathematics should attend not only to answer correctness but also to the procedural evidence embedded in students’ problem-solving processes. Evidence-centered design (ECD) offers a principled basis for connecting learner evidence to task design. However, in classroom contexts, teachers face substantial time and cognitive burden when designing formative assessment tasks based on students’ evidence. This study presents a preliminary implementation of a task–evidence–task cycle for formative assessment using a large language model (LLM). Using 1,406 solution logs collected from 29 students, the LLM infers required knowledge components (KCs) from students’ step-by-step solution logs and constructs diagnostic evidence (e.g., procedural differences, targetable causes, and required KCs). It then uses this evidence to generate two scaffolding tasks for each original task. Expert evaluation showed moderate agreement on the generated KCs and substantial agreement on the appropriateness of the scaffolding tasks. A small follow-up activity with four students further suggested that the generated tasks may help make missing intermediate reasoning steps more visible during re-solving. These findings provide preliminary feasibility evidence for one implemented task–evidence–task cycle, while repeated cyclic use is left for future work.

Keywords

1. INTRODUCTION

In mathematics education, formative assessment plays a key role in supporting students’ learning throughout the problem-solving process. It should therefore attend not only to the correctness of final answers but also to the processes through which those answers are produced.

Mathematical problem solving involves interpreting problem situations and coordinating multiple cognitive components. Koedinger et al.  [7] define these as knowledge components (KCs), or learnable units of knowledge or cognitive function. Because procedural and conceptual knowledge interact and develop together in mathematics learning  [13], students’ procedural work can be interpreted as evidence of KC use during problem solving. Assessment should therefore move beyond answer correctness to examine process-level evidence in students’ work.

Evidence-centered design (ECD) provides a principled framework for making such inferences by specifying relationships among claims, evidence, and tasks  [10, 12]. However, many ECD implementations remain largely task-driven: tasks are specified first, and evidence models are used mainly to interpret task outcomes  [1]. As a result, the process by which diagnostic evidence informs subsequent task design often remains underspecified or is treated as a black box  [5].

To address this gap, this study adopts KCs as an interpretive unit for constructing diagnostic evidence and implements a task–evidence–task cycle in which evidence from students’ solution logs informs subsequent task design. Operationalizing such redesign in classrooms is challenging because extracting KCs from student work and redesigning formative tasks require substantial time and cognitive effort from teachers  [2]. Although recent studies have explored large language models (LLMs) for analyzing student work or generating assessment-related artifacts  [8, 9, 16, 11], most have focused on isolated functions such as task generation, automated scoring, or feedback rather than integrated, evidence-based task redesign  [6].

This paper presents a preliminary implementation of an ECD-grounded task–evidence–task cycle for formative assessment. The LLM is used for two related functions: (1) constructing diagnostic evidence from students’ solution logs, including inferred KCs, and (2) generating scaffolding tasks based on that evidence. The generated tasks are reviewed by two experts, and their appropriateness and potential educational usefulness are examined.

Accordingly, this study addresses the following research questions (RQs):

RQ1. To what extent can an LLM generate meaningful KC candidates required for problem solving from students’ solution logs?

RQ2. To what extent can an LLM generate formative scaffolding tasks based on KCs inferred from students’ solution logs?

2. BACKGROUND

2.1 Evidence-Centered Design for Formative Assessment

Evidence-centered design (ECD) frames assessment as a principled approach to making inferences about learning from observable performance  [10, 12]. By organizing assessment around claims, evidence, and tasks, ECD emphasizes that validity depends not only on task format or difficulty but also on what evidence is collected from learners’ work and how it is interpreted  [8].

This perspective is especially relevant to formative assessment, which aims to identify learners’ current difficulties and provide targeted support  [2]. In ECD-based formative assessment, tasks are not only measurement tools but also means of making learners’ thinking visible. When scaffolding is evidence-based, it should respond to procedural bottlenecks, omissions, and error patterns observed in students’ actual work rather than rely on fixed hints  [15].

Although ECD presupposes an ongoing interplay between evidence and task design  [10], its applications have typically proceeded in a forward direction: tasks are defined first, and evidence models are then developed to interpret the resulting performances. In such task-driven implementations, the link between diagnosis and the design of subsequent tasks is often left implicit  [5, 1]. Recent LLM-based studies have addressed parts of this broader process, including the analysis of student work and the generation of assessment-related outputs  [8, 9, 16, 11]. Most, however, have focused on isolated functions, such as scoring, feedback, or task authoring, rather than on evidence-driven formative task redesign as an integrated process. Moreover, many existing approaches still generate tasks primarily on the basis of predefined learning objectives or answer structures  [3], with relatively little work systematically using evidence from students’ problem-solving processes to guide subsequent task design.

In this study, we adopt an ECD-grounded cyclic Evidence–Task Design Framework in which evidence derived from student solution logs is translated into subsequent formative scaffolding task design.

2.2 Knowledge Components as Units of Evidence

Knowledge components (KCs) are acquired units of cognitive function or knowledge structure that can be inferred from performance on related tasks  [7]. In mathematics, a task typically requires multiple KCs rather than a single one  [13]. Accordingly, assessments based only on answer correctness may fail to capture learners’ actual difficulties.

Prior work has shown that errors, omissions, and incomplete solution steps can serve as evidence for inferring learners’ knowledge states  [4]. In this study, KCs are operationally defined as units of knowledge that enable mathematical performance, such as definitions, conditions, meanings, and interpretation rules  [7]. KC-related errors or omissions in students’ solution processes are interpreted as indicators of learning gaps and used as fine-grained evidence for subsequent scaffolding task design. Recent studies have also explored LLMs as tools for supporting this kind of evidence interpretation and generation process  [4, 11, 16].

3. TASK–EVIDENCE–TASK CYCLE

This study presents one implemented task–evidence–task cycle (Figure 1) in which an LLM converts student solution logs into diagnostic evidence and then uses that evidence to design scaffolding tasks. The cycle links a task space to an evidence space through two prompt-based transformations: diagnosis (task space \(\rightarrow \) evidence space) and generation (evidence space \(\rightarrow \) task space). Knowledge components (KCs) function as the shared representational basis across these stages, linking what is inferred from students’ work to what is subsequently targeted in scaffolding-task design.

3.1 Prompt Design

We implemented the prompting pipeline in two stages: a diagnosis prompt for constructing evidence from student logs, and a generation prompt for producing scaffolding tasks from the diagnosed evidence.

3.1.1 Prompt from Task Space to Evidence Space

The diagnosis prompt takes as input the original task, the final answer, and students’ line-level solution logs. For each task, the logs are grouped into correct and non-correct responses (including incorrect and unfinished responses) based on final-answer correctness. The prompt then asks the LLM to compare the two groups and identify recurring procedural differences, such as omitted conditions, incorrect relational expressions, or points at which solutions stop before completion. Based on these differences, the LLM is prompted to identify targetable causes, that is, aspects of performance that appear to be missing, incomplete, or insufficiently activated and that could plausibly be addressed through instructional support. The prompt also asks the model to infer 10 required KCs associated with those causes and needed to solve the original task. During prompt development, candidate set sizes of 5, 10, and 15 were examined across tasks of different difficulty levels, and 10 was adopted for the present implementation.

To make the diagnosis explicit and traceable, the prompt requires all outputs to be grounded in concrete excerpts from the solution logs, including equations, symbols, and written statements. These excerpts serve as evidence for the identified procedural differences, targetable causes, and related KCs.

3.1.2 Prompt from Evidence Space to Task Space

The generation prompt takes as input the original task, the final answer, the diagnosed procedural differences, the targetable causes, the generated KC candidates, and supporting evidence excerpts from the solution logs. Using these inputs, the LLM generates scaffolding tasks intended to address the diagnosed difficulties revealed in the earlier stage.

The prompt constrains the LLM not to generate tasks that simply rehearse isolated KCs or lead directly to the final answer. Instead, the generated tasks are designed to elicit intermediate products that make students’ reasoning visible, such as stating relevant conditions, constructing relational expressions, or explaining the meaning of a result. Tasks that primarily emphasize answer production or direct computation are excluded.

For each original task, the prompt generates two scaffolding tasks. For each generated task, it also specifies the targetable cause addressed, the supporting evidence excerpt, the related KC, and the intended intermediate product. In this way, the evidence constructed from student work is carried forward into subsequent task design within one implemented task–evidence–task cycle.

4. EVALUATION

This section evaluates the present implementation by reporting (1) the dataset used in the evaluation, (2) the quality of LLM-based KC extraction (RQ1), and (3) expert and student evaluations of the generated scaffolding tasks (RQ2).

Grade 10 students (\(N = 29\)) in South Korea provided handwritten solutions, and all personally identifiable information was removed. These were reviewed by the researchers, transcribed into LaTeXformat, and segmented into a line-level solution log dataset for KC-based diagnosis, with each line representing one written step in a student’s solution. The generated scaffolding tasks were reviewed by two experts (one mathematics teacher and one mathematics education researcher) and tested in a small-scale follow-up activity examining changes in answer accuracy and solution processes. The diagnosis and task generation processes were implemented using GPT-5.2.

4.1 Data Description

Problem-solving data were collected from 15 tasks on quadratic equations, quadratic functions, and quadratic inequalities at the Grade 10 level. These tasks were categorized into three difficulty levels (easy, intermediate, and difficult) based on their cognitive and procedural demands.

Twenty-nine students freely chose which tasks to solve. One solution per student per task was collected and decomposed into line-by-line units, yielding 1,406 solution logs.

Table 1 summarizes answer accuracy and the average number of solution steps by difficulty level. Easy-level tasks showed 85.1% accuracy with 3.08 steps on average, intermediate-level tasks 75.6% with 4.15 steps, and difficult-level tasks 62.7% with 5.99 steps. Overall, lower accuracy tended to be associated with longer solution processes.

At the individual-task level, the two lowest-accuracy tasks showed 17.6% and 33.3% accuracy, respectively; the latter also had the second-highest average number of solution steps.

4.2 KC Extraction from Solution Logs (RQ1)

To examine whether the LLM could meaningfully extract KCs from students’ solution logs, the model generated 10 KC candidates for each of the 15 tasks. Two experts judged the appropriateness of each of the 150 KC candidates using a binary scale (appropriate = 1, inappropriate = 0).

Inter-rater agreement indicated moderate consistency (Cohen’s \(\kappa = 0.52\); overall agreement = 0.78), suggesting that judgments about individual KC candidates were not fully stable.

Task-level analysis further showed a strict validity rate of 0.533, a lenient validity rate of 0.753, and a median Jaccard similarity of 0.750 (Table 2). These results suggest substantial task-level overlap between the LLM-generated KC sets and those identified by expert raters, while also indicating limited agreement on individual KC candidates. The generated KCs should therefore be interpreted as provisional rather than definitive diagnostic units.

We also examined the relationship between the average number of solution steps per task and the average number of valid KCs identified by expert raters. A significant positive Spearman rank correlation was observed (\(\rho = 0.52\), \(p = .048\)), indicating that tasks with longer solution logs tended to yield more expert-validated KC candidates.

4.3 Appropriateness of Generated Scaffolding Tasks (RQ2)

To examine the appropriateness of the generated scaffolding tasks for formative assessment, two experts independently evaluated them using a five-point Likert scale (Table 3). For each original task, two scaffolding tasks were generated, resulting in 30 tasks in total.

Inter-rater agreement, assessed with quadratic weighted Cohen’s kappa, was substantial (0.76 for Scaffolding Task 1 and 0.73 for Scaffolding Task 2). Both experts also assigned mean ratings above the midpoint of the scale (Expert 1: M = 3.53, SD = 1.50; Expert 2: M = 3.83, SD = 1.46), indicating generally positive evaluations and suggesting that the generated scaffolding tasks were reasonably appropriate for formative use.

To obtain preliminary qualitative observations, a follow-up re-solving activity was conducted with four students who had answered incorrectly on the two original tasks with the lowest initial answer accuracy. The students were provided with the original task together with scaffolding tasks that experts had judged to be appropriate.

For the lowest-accuracy task, two of the four students answered correctly in the re-solving attempt, whereas two remained incorrect. For the second-lowest-accuracy task, which involved a longer solution process (Table 4), three students answered correctly and one remained incorrect. Qualitative review of the re-solving processes showed that some students stated previously omitted conditions or added intermediate expressions that were absent from the original solutions. These observations indicate that student responses differed across the two tasks in terms of both final correctness and the procedural changes observed during re-solving.

5. DISCUSSION

Regarding RQ1, the results suggest that an LLM can generate plausible KC candidates from students’ step-by-step solution logs, with moderate consistency in expert judgments. The agreement and task-level overlap measures indicate that the generated KC candidates generally fell within a range experts considered appropriate, although the exact KC sets did not always coincide. Thus, the generated KCs are better interpreted as provisional diagnostic units than as definitive representations of student knowledge. The positive relationship between the number of solution steps and the number of valid KCs further suggests that the LLM responded systematically to differences in procedural evidence rather than producing arbitrary KC sets.

Regarding RQ2, expert evaluations showed substantial agreement on the appropriateness of the generated scaffolding tasks, suggesting that the LLM-produced tasks were generally suitable for formative use. Rather than directly drilling individual KCs or simplifying the original tasks, these tasks were designed to elicit intermediate products that make learners’ thinking visible, such as stating conditions, constructing relational expressions, or articulating the meaning of a solution. This design characteristic aligns with the view that formative assessment tasks should reveal learners’ reasoning processes and support the reconstruction of missing or weak procedural steps  [2, 14].

The follow-up study with four students provided additional qualitative observations about the potential usefulness of the generated scaffolding tasks. In these cases, some students stated previously omitted conditions or added intermediate expressions after working on the generated tasks. These observations indicate that the tasks may have prompted procedural actions that were not visible in the original solutions, although this interpretation remains tentative given the small-scale design. Differences in responses across the two tasks may reflect differences in their cognitive demands, suggesting that the usefulness of scaffolding may depend partly on task characteristics. Future research could examine how the cognitive demand levels of KCs and interactions among multiple KCs relate to scaffolding design and effectiveness.

Theoretically, this study illustrates how a task–evidence–task cycle can be implemented between the task space and the evidence space within an ECD framework. The present design shows how evidence derived from learners’ performances can be organized into diagnostic representations and used to inform subsequent task design in a formative assessment context. In this sense, the study provides a preliminary illustration of how an LLM may support diagnostic interpretation and subsequent task generation within an ECD-based assessment process.

Practically, we envision the proposed pipeline as a teacher-in-the-loop support tool that generates candidate scaffolding tasks from student solution logs. This workflow may help reduce teacher burden by supporting the design of follow-up formative tasks while preserving teacher control over task selection, revision, and classroom use. Future work could examine how this process functions in authentic classroom settings and how the framework may be extended to include the Claim Space within a broader ECD process.

6. CONCLUSIONS AND LIMITATIONS

This study implemented one task–evidence–task cycle by connecting students’ solution logs (task space) with KC-based diagnostic evidence (evidence space) to generate formative scaffolding tasks. Preliminary results suggest that an LLM can generate plausible KC candidates from students’ solution logs and that tasks with richer procedural evidence tended to yield more expert-validated KC candidates. Expert review and a small-scale follow-up activity further suggested that the generated scaffolding tasks may help students recognize and produce intermediate products required for problem solving. Together, these findings provide preliminary feasibility evidence that an LLM may support one evidence-informed task–evidence–task cycle for formative assessment grounded in learners’ observed problem-solving processes.

Several limitations should be noted. First, the present study examined only a single implemented task–evidence–task cycle rather than repeated cycles across multiple rounds. Second, the study was limited to algebraic topics, specifically quadratic equations, quadratic functions, and quadratic inequalities, and did not examine other areas of mathematics such as geometry or statistics. Third, our evaluation of KC extraction focused on precision, but could not assess recall without predefined reference KC sets for each task. Future work should establish such reference sets for comprehensive evaluation. Fourth, the follow-up study involved only a small number of students and did not include a control group. Therefore, the follow-up findings should be interpreted as preliminary qualitative observations.

7. REFERENCES

1. Meirav Arieli-Attali, Sue Ward, Jay Thomas, Benjamin Deonovic, and Alina A. von Davier. The expanded evidence-centered design (e-ECD) for learning and assessment systems: A framework for incorporating learning goals and processes within assessment design. Frontiers in Psychology, 10:853, 2019.
2. Paul Black and Dylan Wiliam. Developing the theory of formative assessment. Educational Assessment, Evaluation and Accountability, 21(1):5–31, 2009.
3. Kuang Wen Chan et al. Automatic item generation in various stem subjects using large language model prompting. Computers and Education: Artificial Intelligence, 8:100344, 2025.
4. Mehmet Arif Demirtas. Identifying and evaluating novel knowledge component models for programming skills. In Proceedings of the 17th International Conference on Educational Data Mining, pages 969–973, 2024.
5. Mingyu Feng, Eric G. Hansen, and Diego Zapata-Rivera. Using evidence-centered design for learning (ECDL) to examine the assistments system. In Proceedings of the Annual Meeting of the American Educational Research Association, San Diego, CA, USA, 2009.
6. Enkelejda Kasneci et al. ChatGPT for good? on opportunities and challenges of large language models for education. Computers and Education: Artificial Intelligence, 4:100193, 2023.
7. Kenneth R. Koedinger, Albert T. Corbett, and Charles Perfetti. The Knowledge-Learning-Instruction Framework: Bridging the science-practice chasm to enhance robust student learning. Cognitive Science, 36(5):757–798, 2012.
8. Holger Maus, Paul Tschisgale, Fabian Kieser, Stefan Petersen, and Peter Wulff. Developing and evaluating a large language model-based automated feedback system grounded in evidence-centered design for supporting physics problem solving. arXiv preprint arXiv:2512.10785, 2025.
9. Jennifer Meyer et al. Using LLMs to bring evidence-based feedback into the classroom: AI-generated feedback increases secondary students’ text revision, motivation, and positive emotions. Computers and Education: Artificial Intelligence, 6:100199, 2024.
10. Robert J. Mislevy, Russell G. Almond, and Janice F. Lukas. A brief introduction to evidence-centered design. ETS Research Report Series, 2003(1):i–29, 2003.
11. Hyeongdon Moon, Richard Lee Davis, Seyed Parsa Neshaei, and Pierre Dillenbourg. Using large multimodal models to extract knowledge components for knowledge tracing from multimedia question information. In Proceedings of the 18th International Conference on Educational Data Mining, pages 342–353, 2025.
12. James W. Pellegrino. Assessment as a positive influence on 21st century teaching and learning: A systems approach to progress. In Patrick Griffin, Barry McGaw, and Esther Care, editors, Assessment and Teaching of 21st Century Skills, pages 167–183. Springer, Dordrecht, 2014.
13. Bethany Rittle-Johnson and Martin Schneider. Developing conceptual and procedural knowledge in mathematics. In Roi Cohen Kadosh and Ann Dowker, editors, The Oxford Handbook of Mathematical Cognition, pages 275–290. Oxford University Press, 2014.
14. Lorrie A. Shepard. The role of assessment in a learning culture. Educational Researcher, 29(7):4–14, 2000.
15. Janneke van de Pol, Monique Volman, and Jos Beishuizen. Scaffolding in teacher–student interaction: A decade of research. Educational Psychology Review, 22(3):271–296, 2010.
16. Yumou Wei, Paulo Carvalho, and John Stamper. Kcluster: An LLM-based clustering approach to knowledge component discovery. In Proceedings of the 18th International Conference on Educational Data Mining, pages 228–240, 2025.

© 2026 Copyright is held by the author(s). This work is distributed under the Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International (CC BY-NC-ND 4.0) license.