ABSTRACT

Video-LLMs hold significant potential for analyzing classroom dynamics to inform pedagogy and drive educational innovation. There is an urgent need to assess the capability of Video-LLMs in educational applications to identify their potential and limitations. Existing educational evaluation datasets often suffer from three major shortcomings: reliance on small-scale, single-label human annotations; a focus on unimodal analysis; and the absence of granular annotations required for multi-step reasoning. To address these gaps, we first introduce an efficient Human-LLM collaborative pipeline for high-quality multimodal annotation. We then propose MTBU-Bench, a comprehensive multimodal benchmark designed to assess teacher behavior understanding. It comprises over 300 classroom scenarios across 15 K-12 and higher education disciplines, organized into four evaluation tasks: Action Recognition, Temporal Boundary Extraction, Spatial Localization, and Behavioral Description. Extensive evaluations of twelve mainstream Video-LLMs reveal critical shortcomings in fine-grained temporal understanding, specifically in the precise localization of action boundaries and the discrimination of similar behaviors. Our benchmark and findings provide a foundation for the targeted development and application of Video-LLMs in educational research. The code and dataset are publicly available at our project homepage: https://github.com/271213/MTBU-Bench.

Keywords

1. INTRODUCTION

In recent years, Video large language models (Video-LLMs), which integrate visual encoders with large-scale pre-trained language models, have demonstrated remarkable performance in video understanding tasks [17, 19, 16]. These models are capable of jointly processing spatiotemporal visual features and semantic reasoning tasks, unlocking new possibilities for comprehending complex scenes [44, 14]. Among the domains empowered by such advancements, education stands out as a critical application area where video understanding facilitates teaching assessment and instructional strategy optimization [45]. In particular, through analyzing instructional actions, spatial positions, and interaction targets within authentic classroom environments, teacher behavior understanding not only provides an objective reflection of teaching effectiveness but also fosters pedagogical self-reflection and offers interpretable insights for professional development [32, 1]. Consequently, evaluating the reasoning capabilities of Video-LLMs in educational settings has become an essential research challenge.

Mainstream Video-LLMs typically rely on static image-based representations, extracting keyframes from input videos to conduct semantic modeling and action recognition [9, 40]. However, such keyframe-driven paradigms pose inherent limitations when applied to fine-grained analysis of teacher behavior in authentic classroom settings [37]. Specifically, the omission of continuous temporal information compromises the model’s ability to accurately delineate action boundaries, often leading to temporal ambiguity as shown in Figure 1. Moreover, for semantically similar behaviors such as "write on the blackboard" versus "clean the blackboard" existing models still struggle with disambiguation, resulting in frequent semantic confusion. These challenges significantly hinder the effectiveness of current Video-LLMs in educational behavior understanding tasks.

Although prior studies have explored video understanding tasks within educational contexts [29, 35], existing benchmarks remain inadequate for systematically evaluating the reasoning capabilities of Video-LLMs. This inadequacy stems from three primary limitations: 1) Current datasets predominantly rely on manually annotated, single teacher action labels, which are costly to produce and prone to annotator bias. They are typically confined to small-scale instructional videos [37, 23]; 2) Existing research is largely limited to unimodal video analysis [11, 12], failing to leverage the complementary strengths of multimodal data, thereby constraining the comprehension of rich classroom interaction dynamics; 3) Present benchmarks tend to focus on coarse, single-label descriptions of teacher action [6], lacking fine-grained annotations of behavior, interaction targets, and contextual features, which are essential for aligning with the multi-step reasoning processes of Video-LLMs.

To address the aforementioned challenges, we propose MTBU-Bench, the first multimodal evaluation benchmark tailored for assessing Video-LLMs on teacher behavior understanding tasks. MTBU-Bench spans 15 academic disciplines (Math, Physics, Biology, Chemistry, Science, Geography, History, Chinese, English, Music; Higher education: Medicine, Political - History, Engineering, Liberal Arts, Natural Sciences.) across K-12 and higher education, comprising over 500 complete, authentic classroom videos from more than 300 distinct teaching scenarios. As illustrated in Figure 2, each instance includes multimodal representations of teacher behavior along with five annotated dimensions: behavior category, spatial location, interaction target, video description, and Question–answer (QA) pairs. To ensure both annotation efficiency and multimodal consistency, we introduce a multi-stage Human–LLM collaborative annotation pipeline. This workflow integrates expert annotations, senior expert validation, and LLM-assisted behavioral description generation. Based on this, we conduct a comprehensive evaluation of ten mainstream Video-LLMs (e.g., GPT-4V, Gemini 2.5 Pro), focusing on four core tasks: teacher action recognition, teacher behavior temporal boundary extraction, teacher behavior spatial localization, and teacher behavior description. Experimental results reveal notable limitations in current Video-LLMs, particularly in recognizing similar behaviors, aligning interaction targets, and modeling environmental context.

2. RELATED WORKS

2.1 Video Understanding Dataset

Video understanding aims to automatically generate natural language descriptions of key events, human actions, and scene contexts within videos, and has been widely applied in educational analysis, video retrieval, and supplementary instruction [27, 7, 38]. With the rapid development of multimodal understanding and generative models, video captioning has evolved from single-sentence summaries to structured, hierarchical, and fine-grained representations. The MiraData dataset, employing the GPT-4V model to generate detailed descriptions from six distinct perspectives [24]. The YouCookII dataset focuses on instructional videos in cooking scenarios, providing temporal boundaries and English descriptions for each procedural step [49]. The ViTT dataset further extends to multi-domain instructional videos, offering short tags averaging three words per segment [21]. To pursue higher-quality and more detailed long-form descriptions, the Koala-36M dataset provides 36 million filtered video clips with detailed, structured captions averaging 200 words [42]. Recent research has also pushed into grounding these descriptions, with ViCaS linking captions to pixel-level segmentation masks [3], and HD-EPIC further grounding fine-grained actions and objects within 3D digital twins from an egocentric perspective [34].

2.2 Datasets for Teacher Analysis

Compared to general video datasets, educational video data place greater emphasis on multimodal alignment, authentic instructional context, and structured annotations. The EduNet dataset, encompassing 20 categories of teacher and student behaviors across 7,851 video clips [37]. TBU is a multi-task teacher behavior dataset that includes three kinds of labels across 200 classroom videos [6]. TCDA focuses on teacher dress code assessment containing 11,879 annotated images with 25 attribute categories teacher dress [12]. TAQR is the benchmark dataset dedicated to teacher action quality ranking, covering four core instructional behaviors 1,200 video clips [11]. However, all of these datasets are limited to manually annotated unimodal data and lack multi-dimensional fine-grained video descriptions, making them insufficient for directly evaluating the reasoning capabilities of Video-LLMs. A detailed comparison of MTBU-Bench with these existing teacher behavior datasets is summarized in Table 1.

2.3 Evaluation of Video-LLMs

With the development of Video-LLMs, a series of evaluation benchmarks has also emerged [47, 48], each with a distinct focus. Video-MME [15] aims for comprehensive evaluation, covering diverse video types, durations from 11 seconds to 1 hour. ALLVB [39] concentrates on ultra-long video understanding, using videos averaging nearly two hours to assess long-sequence modeling capabilities [17]. Other benchmarks focus on more fine-grained abilities[28], such as MotionBench [18], which specifically evaluates the understanding of fine-grained motion. However, these general-purpose benchmarks, despite their advancements in comprehensiveness, duration, and granularity, do not address the specific needs of educational scenarios, such as the fine-grained understanding of teacher behavior[2, 30]. MTBU-Bench is the first comprehensive multimodal benchmark designed to fill the crucial gap of evaluating Video-LLMs understanding of teacher multimodal behavior.

3. MTBU-BENCH

In this section, we introduce MTBU-Bench, the first comprehensive multimodal benchmark tailored for assessing Video-LLMs on teacher behavior understanding tasks. Spanning 15 academic disciplines across K-12 and higher education, the dataset comprises over 500 complete, authentic classroom videos from more than 300 distinct teaching scenarios. To ensure annotation quality and multimodal consistency, we construct the benchmark through a Human-LLM Collaborative Annotation pipeline that effectively integrates expert knowledge with automated generation capabilities. We organize the dataset into four core task subsets: Multimodal Teacher Action Recognition, Teacher Behavior Temporal Boundary Extraction, Teacher Behavior Spatial Localization, and Teacher Behavior Description, designed to systematically evaluate the fine-grained perception and reasoning capabilities of large models in complex educational environments.

3.1 Data Construction Workflow

As shown in Figure 3, we propose a Human-LLM Collaborative Annotation pipeline to construct MTBU-Bench, including video collection, Annotation System Construction, data segmentation and annotation, quality assurance, multimodal extraction and alignment, behavior description generation, question-answer pair generation, description and QA Pair verification.

Video Collection. We collect 500 complete classroom teaching videos spanning multiple educational levels from K-12 to higher education. It covers 15 academic subjects and includes both public lectures and competition-based lessons. The recorded classroom scenarios exhibit diverse camera perspectives, varied instructional formats, and heterogeneous spatial layouts, totaling more than 300 distinct teaching environments. These videos faithfully capture pedagogical characteristics, teacher behaviors, and classroom interaction patterns across different educational stages, offering strong representativeness and practical applicability for multimodal educational research.

Annotation System Construction. To support multimodal reasoning and enable comprehensive modeling of teacher behaviors, we design a structured annotation system composed of four key dimensions. First, we define a set of ten core instructional actions based on their high frequency and semantic distinctiveness in real-world classrooms. Second, we annotate the spatial position of the teacher during action execution to provide grounding for spatiotemporal localization. Third, we annotate the primary interaction object to reflect communicative targets and teaching intent. Finally, we generate structured natural language descriptions that integrate visual, auditory, and contextual cues, offering rich semantic supervision for downstream Video-LLMs. This four-dimensional annotation schema ensures interpretability, consistency, and task alignment across the benchmark.

Data Segmentation and Annotation. To construct a multimodal dataset tailored for teacher behavior understanding, we adopt behavior segments as the fundamental annotation unit. Raw instructional videos are temporally segmented and annotated to ensure that each segment exhibits content completeness, clear temporal boundaries, and semantic consistency with respect to the behavior it represents. The segmentation is conducted using videoManager, with each segment corresponding to the full execution of a distinct teaching action. The start time is defined by the onset of an observable behavioral cue, while the end time is marked by the action’s completion or transition into the next behavior. The granularity of segmentation follows the principle of units of semantic behavior, balancing multimodal observability with the pedagogical coherence of instructional behaviors [25].

To ensure the high quality, accuracy, and pedagogical domain fidelity of the MTBU-Bench dataset, the annotation process was driven by a highly qualified, cross-disciplinary team. This team consisted of one primary researcher, three experienced frontline teachers (each with over 10 years of classroom teaching experience), and four graduate students (two specializing in Education and two in Computer Science).

The annotation process followed a rigorous collaborative workflow. Initially, the primary researcher and the frontline teachers collaboratively finalized the behavior labels to ensure they accurately reflected authentic instructional dynamics. Subsequently, they conducted comprehensive training for the four graduate students. The annotators were provided with detailed annotation guidelines, which included strict label definitions, concrete annotation examples, and standardized operating procedures for the annotation tools.

During the execution phase, the annotation tasks were divided into four subsets. Rather than being separated by educational stages or subject areas, each subset encompassed content from all disciplines and educational levels. All four graduate students were jointly responsible for the data segmentation and annotation tasks across all four subsets. Each behavior instance was meticulously annotated with four key attributes: teacher action, spatial position during the action, interaction target, and start and end timestamps.

Quality Assurance. To guarantee the overall reliability of the dataset, the three frontline teachers independently conducted a Quality Assurance phase, rigorously reviewing and validating the temporal boundaries and semantic labels of the annotated clips. Furthermore, to evaluate inter-annotator agreement, a 10% data overlap was maintained across the subsets. Any discrepancies were resolved through group discussion and majority voting. The final annotation consistency was quantitatively assessed using Fleiss’ Kappa, yielding a score of 0.949, which indicates a "high agreement" level [13]. This review process focuses primarily on verifying and harmonizing the temporal boundaries of annotated behaviors, and enhancing the overall consistency and scientific rigor of the dataset.

Multimodal Extraction and Alignment. We employ a suite of open-source tools including OpenCV¹, Whisper², and FFmpeg³ to automate the extraction and alignment of visual, audio, and textual modalities from classroom videos. Through a timestamp-based synchronization mechanism, teacher actions in the visual stream, spoken content in the audio stream, and semantic information in the text stream are precisely aligned to form temporally consistent multimodal behavior segments. This stage ensures that each annotated teacher behavior instance maintains high consistency and semantic completeness across visual, auditory, and linguistic modalities.

Behavior Description Generation. To enhance model understanding in classroom teacher behavior recognition tasks, we construct a four-dimensional description for each annotated video segment. Each dimension is designed to complement behavior semantics from a distinct perspective, thereby improving the model’s capability to recognize fine-grained instructional behaviors in complex classroom environments. Building upon high-quality annotations, we leverage the
Video-LLMs to generate four types of structured natural language descriptions for each behavior clip, as detailed below:

• Fine-grained description of teacher actions. This dimension provides fine-grained descriptions of core instructional actions to distinguish semantically different but visually similar behaviors. For example, "clean the blackboard" and "write on the blackboard" share similar visual cues standing in front of the blackboard and using hand gestures yet their pedagogical intents diverge, often leading to misinterpretation by Video-LLMs. By documenting micro-actions (e.g., hand-raising, body-turning, or using teaching aids) and subtle temporal patterns, this dimension enhances the model’s capacity to disambiguate near-identical actions and capture critical shifts in instructional orientation.
• Spatial location of behavior execution. This dimension captures the teacher’s spatial position, which serves as a critical indicator of instructional intent. For instance, positions such as the classroom corners or the blackboard center correlate with specific actions like "patrol the classroom" or "lecture". Explicitly encoding spatial context enhances the model’s contextual grounding and classification accuracy by leveraging the strong spatial dependencies of instructional behaviors. Ultimately, spatial location functions as a discriminative feature that significantly improves action recognition performance.
• Primary interaction target during behavior execution. This dimension explicitly identifies interaction targets such as students, blackboards, or multimedia devices to clarify the object associated with each instructional behavior. Distinguishing these targets is essential for interpreting pedagogical intent and serves as a primary criterion for differentiating between lecture driven actions and interactive driven actions. For instance, engaging with students signifies interactive behavior, while manipulating multimedia denotes a control action. Ultimately, these annotations support downstream tasks like pedagogical strategy analysis and classroom structure modeling.
• Instructional environment context. This dimension captures environmental cues such as classroom layout,
  teacher attire, and student status to provide essential context for interpreting instructional actions. For example, laboratory settings with lab benches often imply demonstrative actions, while traditional attire combined with a projection screen aligns with "lecture" or "multimedia teaching" scenarios. These cues are pivotal for filtering non-instructional behaviors and enhancing model precision and robustness. Ultimately, these features ensure semantic coherence in behavioral analysis and reinforce the reliability of multimodal alignment.

These structured descriptions provide rich semantic grounding for subsequent behavior understanding and multimodal evaluation tasks. Behavior description generation process is shown in Figure 4.

Question–Answer Pair Generation. Building upon the structured behavior descriptions, we further employed Video-LLMs to generate QA pairs for each classroom behavior segment. Each QA pair targets one of four key dimensions: the teacher’s instructional action, spatial position, primary interaction object, and the surrounding instructional environment. The integration of QA pairs not only enriches the semantic depth of the dataset enabling models to acquire more fine-grained insights into teacher behaviors but also strengthens cross-modal alignment ability. Moreover, this QA-based design lays a high-quality foundation for downstream applications such as instructional question answering systems and fine-tuning large models for classroom behavior analysis and evaluation in educational settings [46, 36, 43].

Description and QA Pair Verification. Domain experts review the model-generated descriptions and question–answer pairs with respect to four key dimensions: instructional actions, spatial positioning, primary interaction targets, and the surrounding classroom environment.

3.2 Dataset Subset Composition

As shown in Figure 5, MTBU-Bench comprises four distinct subsets, each tailored to a specific evaluation objective: multimodal teacher action recognition, teacher behavior temporal boundary extraction, teacher behavior spatial localization, and teacher behavior description. These subsets are designed with a clear structural organization, comprehensive annotations, and a strong task-driven focus, making the dataset well-suited for advancing research in educational video understanding across multiple dimensions.

3.2.1 Multimodal Teacher Action Recognition Subset

Teacher Action Labels. The Multimodal Teacher Action Recognition Subset (MTAR Subset) is designed to provide standardized data for instructional action recognition, focusing on ten of the most frequently observed teacher actions in classroom settings: Write on the blackboard: The teacher uses chalk or a marker to write text, formulas, or draw diagrams on the physical blackboard or whiteboard. Clean the blackboard: The teacher uses an eraser or cloth to remove content from the physical blackboard or whiteboard. Multimedia teaching: The teacher utilizes digital platforms (e.g., presenting slides, playing videos) as the primary medium to deliver instructional content. Operating multimedia: The physical action of the teacher manipulating technological devices (e.g., typing on keyboard, clicking mouse, adjusting projector) to control the flow of digital content. Display teaching aids: The teacher physically holds up or demonstrates physical objects, models, or printed materials to the students to facilitate understanding. Interact with students: The teacher engages in direct, two-way communication with students, including asking questions, answering queries, or guiding group discussions. Lecture: The teacher delivers verbal explanations of academic concepts to the entire class, typically standing still without intense physical interactions with props or the board. Patrol the classroom: The teacher walks dynamically among the classroom to monitor progress, observe independent work, or provide localized guidance. Point to the podium: The teacher uses hand gestures or a pointer to direct students’ visual attention to specific content on the blackboard or projection screen. Make a bow: A formal, ritualistic action where the teacher bows to the students, typically occurring at the very beginning or end of a lesson. [22, 26, 5].The data distribution is shown in Table 2.

This subset contains a total of 8,000 multimodal samples, covering the ten core instructional action. Each sample includes a high-quality video clip, synchronized audio, and its transcribed text, ensuring precise alignment across the three modalities. This subset is suitable for training and evaluating multimodal action recognition models and supports research directions such as cross-modal retrieval, audio-visual joint modeling, and multimodal feature fusion. It serves as foundational training data for multimodal models in educational scenarios.

3.2.2 Teacher Behavior Temporal Boundary Extraction Subset

To address the bottlenecks of current Video-LLMs in long video understanding and temporal localization, we construct the Teacher Behavior Temporal Boundary Extraction Subset. Unlike most existing benchmarks that rely on short trimmed clips, this subset comprises 400 complete classroom teaching videos, with an average duration exceeding 40 minutes per session.

These untrimmed videos fully preserve the continuity and complexity of authentic instructional activities. We provide dense, frame-level annotations, precisely marking the start and end timestamps of every teacher action throughout the entire lesson. This setup presents a significant challenge for models: they are required not only to recognize specific action categories but also to precisely delineate action boundaries within a long-horizon context, thereby testing their capability to handle temporal redundancy and transition ambiguity in real-world continuous video streams. Examples from the Teacher Spatiotemporal Behavior Localization Subset are provided in Figure 6.

3.2.3 Teacher Behavior Spatial Localization Subset

To evaluate the fine-grained spatial reasoning capabilities of Video-LLMs within complex classroom environments, we construct the Teacher Behavior Spatial Localization Subset (TBSL Subset). To ensure the reliability of spatial reasoning evaluation, this subset consists of 4,000 high-quality short video clips carefully selected from the 8,000 clips in the MTAR subset. We strictly filtered for instances where the teacher’s spatial orientation and positional information are highly discernible, and the continuous action duration exceeds 10 seconds.

Teacher Spatial Position and Interaction Target Labels. Each sample is annotated with a structured behavior triplet, comprising the teacher action, their spatial position relative to the classroom layout, and the corresponding primary interaction target. Spatial positions include being On the podium: where the teacher is located in the defined instructional zone at the front of the classroom, typically near the blackboard and main desk; Among students: where the teacher is located within the student seating area, moving through the aisles or standing next to desks; and Behind the classroom: where the teacher is positioned at the very back of the room, typically to observe the entire class’s status or during student-led presentations. Furthermore, primary interaction targets include the "Blackboard" for traditional physical writing surfaces; "Multimedia PPT" for digital slides or educational software interfaces displayed on a screen; "Multimedia" for hardware devices such as computers, projectors, or interactive smartboards; "Teaching aids" for physical objects, biological specimens, geometric models, or chemical apparatus; and "Students" for individual students or the class as a whole. The data distribution is shown in Table 2. These triplets provide fine-grained and semantically rich supervision for modeling spatial reasoning in instructional scenarios. The teacher’s interaction target offers implicit cues regarding their spatial orientation; this correlation between interaction semantics and spatial positioning is critical for Video-LLMs to build accurate representations of instructional context.

3.2.4 Teacher Behavior Description Subset

The Teacher Behavior Description Subset is dedicated to supporting video-to-text generation and question answering tasks within educational contexts. For each behavior segment, the subset provides a detailed natural language description that encapsulates semantic elements such as the type of action, spatial location, and interaction target. To further enhance the model’s reasoning capabilities in complex instructional scenarios, each description is paired with a corresponding video-based QA pair, enabling both generative and comprehension-based evaluations of multimodal language models, example data is displayed in Figure 7.

In total, the subset contains 24,000 natural language descriptions aligned with corresponding video segments, along with 24,000 QA pairs grounded in the semantic content of those segments. These 24,000 samples are directly derived from the 8,000 MTAR clips by generating three distinct descriptions and corresponding QA pairs from different semantic perspectives for each clip. Each description averages 26 words in length and captures rich semantic dimensions, including teacher actions, classroom environment, and interactive elements. This subset provides a standardized benchmark resource for fine-tuning and evaluating multimodal large models in educational scenarios.

4. EXPERIMENTS

4.1 Experimental Setup

To evaluate the fundamental capabilities of mainstream Video-LLMs on the teacher behavior understanding task, we conducted a systematic zero-shot evaluation of ten representative models, including GPT-4V, Gemini 2.5 Pro, Video-LLaVA, LLaMA 4-17B, Qwen2.5-Omni, Spark X1 (education-oriented), Grok3, Qwen-QVQ, Claude 3.5 Sonnet, and Kimi-latest. All experiments were conducted under a zero-shot inference setting without any task-specific fine-tuning or training. The evaluation was performed on the four core subtasks of MTBU-Bench Teacher Action Recognition, Teacher Behavior Temporal Boundary Extraction, Teacher Behavior Spatial Localization Subset , and Behavior Description, using a unified standard to ensure comparability and reproducibility of the results.

In terms of test scale, we constructed a comprehensive zero-shot evaluation dataset tailored to the specific characteristics of each task. For Teacher Action Recognition, Spatial Localization, and Behavior Description, we utilized a dataset comprising 4,000 samples, with 400 samples drawn from each of the 10 core teacher action categories. For Teacher Behavior Temporal Boundary Extraction, considering the current limitations of Video-LLMs in long-context understanding, we curated a specialized test set of action-intensive segments selected from the complete annotated classroom videos. This set consists of 100 clips with a duration of 4 minutes and 100 clips with a duration of 8 minutes. All models were evaluated on the same sampled data to ensure consistency and reliability of the experimental results.

4.2 Teacher Action Recognition

For this task, we specifically introduced two Omni-LLMs, Qwen2.5-Omni and Gemini 2.5 Pro-Omni, to evaluate their performance with full multimodal inputs (video, audio, and text). To isolate and assess the specific contribution of the auditory modality to classroom behavior understanding, we also established a baseline vision-language setting for all ten models. This comparative setup allows us to validate the benefits of supplementary audio information.

We evaluate model performance on both overall accuracy and individual instructional actions, using Accuracy, Precision, Recall, and F1 Score as the primary metrics. As summarized in Table 3, the multimodal Gemini 2.5 Pro-Omni (0.5650 Acc), Qwen2.5-Omni (0.5575 Acc) significantly outperforms its unimodal variant, Gemini 2.5 Pro (0.5325 Acc), Qwen2.5-Omni-Single (0.5150 Acc). This demonstrates that inputs encompassing video, audio, and text enhance a model’s ability to capture diverse cues from complex instructional scenes. Furthermore, Gemini 2.5 Pro-Omni achieves the best overall accuracy (0.5650) and a standout precision (0.7363). Closely following are Qwen2.5-Omni and Spark X1. In contrast, LLaMA4-17B displays a high precision (0.6561) but a comparatively low recall (0.4675), suggesting a more conservative prediction strategy.

For individual action categories. The results reveal that actions with distinct and concrete visual features (such as "Write on the blackboard" and "Display teaching aids") tend to be recognized more reliably across models. Qwen2.5-Omni and Spark X1 are particularly strong on such categories, achieving up to 100% accuracy. In contrast, abstract or weakly visual actions such as "Point to the podium" and "Make a bow" remain challenging across the board, often exhibiting recognition accuracy below 10%, as presented in Figure 8. These findings underscore the necessity of cross-modal fusion to resolve visual ambiguity and to improve semantic discrimination in fine-grained instructional behavior understanding.

Compared to unimodal video inputs, multimodal integration demonstrates superior performance in teacher action recognition tasks. Specifically, the audio modality contributes linguistic content and ambient classroom cues, helping to disambiguate semantically similar actions that are visually confounded. For instance, "Write on the blackboard" and "Clean the blackboard", which appear visually alike but can be distinguished through corresponding audio signals. When writing on the blackboard, teachers often verbalize the content being written, whereas during clean the blackboard, they typically remain silent. The textual modality further enhances the model’s understanding of instructional goals and behavioral intent, addressing the limitations of visual data in modeling contextual dependencies.

4.3 Teacher Behavior Temporal Boundary Extraction

To evaluate the temporal localization capabilities of Video-LLMs in continuous video streams, we conducted specific experiments on the Teacher Behavior Temporal Boundary Extraction Subset (TBTB Subset). Although the TBTB Subset provides complete classroom annotations with an average duration exceeding 40 minutes, current Video-LLMs still suffer from significant computational bottlenecks and reasoning deficiencies when processing ultra-long temporal contexts (Long-context), making it difficult to effectively comprehend the complexity of a full lesson. Therefore, we curated a test set by extracting "action-intensive" segments from the TBTB Subset, consisting of 100 clips with a duration of 4 minutes and 100 clips with a duration of 8 minutes. This design aims to maximally test the models’ temporal sensitivity within a range accessible to their current capabilities. For evaluation metrics, we employ temporal Intersection over Union (tIoU) to measure the overlap between predicted segments and ground truth annotations. Additionally, we use mAP@0.3 and mAP@0.7 to assess model performance under two distinct strictness levels: "coarse localization" and "fine-grained localization," respectively. The experimental data is presented in Table 4.

The experimental results expose a "Precision Bottleneck" in temporal grounding among current models. While models demonstrate a degree of usability in coarse localization (mAP@0.3) on 4-minute videos (with Gemini 2.5 Pro reaching up to 48.89%), indicating their ability to perceive the approximate intervals of action occurrences; however, performance drops precipitously once precise boundary regression (IoU=0.7) is required. Even the best-performing model achieves an mAP@0.7 of only 6.32% in the shorter 4-minute setting, while some models score 0. This reveals that existing Video-LLMs function more as "semantic event detectors" rather than localizers with high-resolution temporal regression capabilities, failing to meet the demand for precise timestamps in educational analysis.

4.4 Teacher Behavior Spatial Localization

As video duration increases, models exhibit Catastrophic Performance Decay. When the test video duration extends from 4 minutes to 8 minutes, the average coarse localization accuracy (mAP@0.3) across all models drops by approximately 50%, highlighting the difficulty models face in maintaining attention over longer contexts. Most critically, in the 8-minute setting, the mAP@0.7 scores for all tested models fall to zero (0.00%). This "zero-shot failure" phenomenon indicates that as the temporal context expands, the redundancy and accumulation of visual information completely overwhelm the models’ fine-grained perception mechanisms, resulting in a total loss of the ability to precisely delineate action boundaries in long continuous streams.

The data reveals a severe Divergence between Semantic Retrieval and Temporal Alignment in Video-LLMs. Although tIoU metrics maintain a range of 16%-37%, suggesting that models can successfully comprehend and retrieve approximate content segments containing the target actions (i.e., "finding the content"), the extremely low mAP@0.7 proves that this semantic understanding does not translate into precise geometric or temporal alignment (i.e., "failing to localize accurately"). This suggests that current architecture designs may over-rely on semantic correlation calculations similar to "image-text matching," while lacking dedicated mechanisms to explicitly model temporal boundary mutations and state transitions.

This dimension measures the model’s ability to accurately identify the spatial location and the interaction target of the teacher while performing a specific instructional action.

As shown in Figure 9, all models achieved high performance in the spatial localization task, with accuracy exceeding 70%. This suggests a generalizable capability in recognizing stable spatial configurations commonly found in classrooms. For instance, the podium area is often characterized by frontal blackboards and multimedia screens, while interactions among students are typically associated with dense seating arrangements and more dynamic visual contexts.

In terms of interactive object recognition, model performance varied more significantly. Notably, Spark X1 achieved the highest accuracy in this dimension (71.2%), substantially outperforming other models. This can be attributed to its education-oriented design, which enhances its capacity to attend to pedagogically relevant visual cues and instructional targets. Models such as Qwen2.5-Omni and Gemini 2.5 Pro also demonstrated balanced performance across both spatial and interaction-level reasoning, reflecting strong cross-modal alignment.

The two tasks are inherently interrelated: when teachers are positioned on the podium, their interaction targets are predominantly the blackboard or multimedia devices; When interacting with students, their spatial position typically shifts among students seating area. This spatial-interaction coupling provides mutual semantic cues that can reinforce model prediction accuracy. Together, accurate recognition of spatial positions and interaction targets offers deeper insights into instructional intent and classroom orchestration strategies.

4.5 Teacher Behavior Description

We further evaluate models’ ability of video understanding by the teacher behavior description task, using two complementary approaches: traditional textual similarity metrics and semantic scoring via LLM-as-a-judge. In traditional metric-based evaluation, the evaluation results are shown in Table 5 in terms of ROUGE [31], CIDEr [41], BLEU [33], METEOR [4]. Qwen2.5-Omni achieves the highest CIDEr score (0.5097), reflecting its superior ability to reproduce fine-grained instructional details that align with expert references. Gemini 2.5 Pro leads in BLEU (0.2327), METEOR (0.4795), and ROUGE (0.4810), demonstrating a strong balance between surface-level lexical accuracy and semantic fluency. GPT-4V and Claude 3.5 Sonnet exhibit relatively balanced but slightly lower scores across all metrics, while education-oriented model Spark X1 maintains competitive performance, particularly in CIDEr (0.5077), which highlights its alignment with the educational domain.

To further enhance the objectivity and robustness of the evaluation, we introduce the LLM-as-a-Judge [8, 10] mechanism, employing DeepSeek-R1 to compute semantic alignment score for the model-generated teacher behavior descriptions. To ensure the reliability of this automated scoring mechanism, we further introduce GPT-5 to conduct an independent re-check on all teacher behavior description samples and calculate the Pearson correlation coefficient (γ) between its scores and the original scores from DeepSeek-R1. The results show an overall correlation of γ = 0.87, indicating a high consistency between the two scoring methods. For the samples where γ \(<\) 0.8 (totaling 212 instances), we further invited three experts in the field of education to perform a manual re-evaluation to verify the reasonableness of the scores. This multi-layer evaluation mechanism ensures that both lexical similarity and deep semantic consistency are taken into account, thereby significantly increasing the reliability of our benchmarking results.

Specifically, we design a rigorous 100-point weighted deduction scheme to quantify semantic fidelity, as detailed in Table 6. The scoring framework comprises six dimensions: Core Action Accuracy (30%) serves as the primary metric, imposing the most severe penalties for action misclassification to ensure fundamental validity. Spatiotemporal Localization (15%) and Interactive Object Recognition (15%) jointly evaluate the precision of spatial grounding and target alignment. Content Consistency (20%) strictly penalizes logical contradictions and hallucinations, while Environmental Detail Fidelity (10%) and Linguistic Specification (10%) assess the richness of contextual cues and terminological correctness. This hierarchical weighting strategy ensures that the evaluation prioritizes action-centric semantic precision while maintaining the overall structural coherence of the generated descriptions.

Evaluation scores are presented in Figure 10. Scoring results reveal that Gemini 2.5 Pro(70.9), Spark X1(69.1) and Qwen2.5-Omni(69.0) achieve relatively high semantic alignment, indicating their strong capacity to model key instructional components such as action intent, spatial positioning, and interaction targets. GPT-4V(67.2), Qwen-QVQ(66.2) and Claude 3.5 Sonnet (66.1) followed closely, suggesting a high degree of linguistic coherence, though it exhibited occasional misinterpretations of behavioral details. As an automated semantic evaluation framework, LLM-as-a-judge provides interpretable and discriminative metrics that go beyond surface-level textual overlap, offering deeper insights into whether large models truly comprehend instructional behavior. Scoring examples from DeepSeek-R1 are provided in Figure 11.

5. CONCLUSION

In this work, we addressed the critical need to evaluate Video-LLMs in educational contexts, particularly for teacher behavior understanding. We proposed a multi-stage Human-LLM collaborative annotation pipeline that ensured annotation efficiency and multimodal consistency. Through this pipeline, we introduced MTBU-Bench, the first multimodal benchmark tailored for this task, spanning 15 K-12 and higher education disciplines across 300+ classroom scenarios. Through large-scale evaluations of ten mainstream Video-LLMs across action recognition, spatiotemporal localization and behavior description, we uncovered key insights. Firstly, audio and text modalities were critical for disambiguating visually similar behaviors by incorporating verbal cues and contextual intent. Secondly, abstract actions posed significant challenges, highlighting limitations in interpreting weakly visualized behaviors. Thirdly, all models struggled with fine-grained distinctions in interaction targets and temporal boundary extraction in long-form videos. Overall, MTBU-Bench provided a rigorous framework for advancing Video-LLMs in education, while our findings highlighted urgent needs for improved temporal modeling, cross-modal alignment, and handling of abstract instructional behaviors. Furthermore, we recognize that our current classroom videos are predominantly sourced from mainland China, which introduces a limitation in cross-cultural generalization. Since instructional norms and spatial layouts vary significantly across cultures, scaling this benchmark to encompass global diverse educational settings remains an important direction for future research. We hope it will inspire further developments in instructional video understanding, MLLMs alignment, and behavior-aware educational applications.

Ethical Statement

The construction of MTBU-Bench was conducted in strict adherence to ethical guidelines for human subjects research and was formally approved by our Institutional Review Board (Approval No. IMU20260402-0017). To safeguard the privacy of participating educators and K-12 students, all Personally Identifiable Information (PII) was systematically removed, and robust facial blurring was applied to recognizable individuals. The dataset is released exclusively for non-commercial, academic research under a strict Data Use Agreement to prevent misuse.

6. ACKNOWLEDGMENTS

This work was funded by the National Natural Science Foundation of China (Nos. 62567005 and 62406127), and Natural Science Foundation of Inner Mongolia Autonomous Region of China (No. 2025MS06004).

7. REFERENCES

1. K. Ahmad et al. Data-driven artificial intelligence in education: A comprehensive review. IEEE Trans. Learn. Technol., 17:12–31, 2023.
2. M. A. AlAfnan et al. ChatGPT as an educational tool: Opportunities, challenges, and recommendations for communication, business writing, and composition courses. J. Artif. Intell. Technol., 3(2):60–68, 2023.
3. A. Athar et al. Vicas: A dataset for combining holistic and pixel-level video understanding using captions with grounded segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 19023–19035, June 2025.
4. S. Banerjee and A. Lavie. METEOR: An automatic metric for MT evaluation with improved correlation with human judgments. In Proc. ACL Workshop Intrinsic Extrinsic Eval. Measures Mach. Transl. Summarization, pages 65–72, 2005.
5. X. B. Bruce, Y. Liu, and K. C. C. Chan. Multimodal fusion via teacher-student network for indoor action recognition. In Proc. AAAI Conf. Artif. Intell., volume 35, pages 3199–3207, 2021.
6. T. Cai, Y. Xiong, C. He, C. Wu, and S. Zhou. Tbu: A large-scale multi-mask video dataset for teacher behavior understanding. In 2024 IEEE International Conference on Multimedia and Expo (ICME), pages 1–6. IEEE, 2024.
7. N. Chaturvedi. LLMs and NLP for generalized learning in AI-enhanced educational videos and powering curated videos with generative intelligence. In Proc. 1st Workshop NLP Sci. (NLP4Science), pages 148–154, 2024.
8. D. Chen et al. MLLM-as-a-judge: Assessing multimodal LLM-as-a-judge with vision-language benchmark. In Forty-first International Conference on Machine Learning, pages 6562–6595, 2024.
9. DeepMind. Gemini 2.5 Pro, 2025. Accessed: 2025-07-30.
10. M. Desmond et al. EvalAssist: LLM-as-a-judge simplified. In Proc. AAAI Conf. Artif. Intell., volume 39, pages 29637–29639, 2025.
11. M. Fang, X. Du, Q. Liu, Y. Zhou, Q. Liang, and S. Liu. Which is the better teacher action? a new ranking model and dataset. In ICASSP 2024-2024 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pages 7695–7699. IEEE, 2024.
12. M. Fang et al. A teacher classroom dress assessment method based on a new assessment dataset. In Proc. Int. Joint Conf. Artif. Intell. (IJCAI), pages 7251–7259, 2024.
13. J. L. Fleiss. Measuring nominal scale agreement among many raters. Psychol. Bull., 76(5):378–382, 1971.
14. C. Fu et al. Video-MME: The first-ever comprehensive evaluation benchmark of multi-modal LLMs in video analysis, 2024. arXiv preprint.
15. C. Fu et al. Video-mme: The first-ever comprehensive evaluation benchmark of multi-modal llms in video analysis. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 24108–24118, June 2025.
16. D. Guo et al. Deepseek-R1: Incentivizing reasoning capability in LLMs via reinforcement learning, 2025. arXiv preprint.
17. Y. Guo et al. Vtg-llm: Integrating timestamp knowledge into video LLMs for enhanced video temporal grounding. In Proc. AAAI Conf. Artif. Intell., volume 39, pages 3302–3310, 2025.
18. W. Hong, Y. Cheng, Z. Yang, W. Wang, L. Wang, X. Gu, S. Huang, Y. Dong, and J. Tang. Motionbench: Benchmarking and improving fine-grained video motion understanding for vision language models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 8450–8460, June 2025.
19. B. Huang, X. Wang, H. Chen, Z. Song, and W. Zhu. Vtimellm: Empower llm to grasp video moments. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14271–14280, 2024.
20. C. Huang, J. Zhu, Y. Ji, W. Shi, M. Yang, H. Guo, J. Ling, P. De Meo, Z. Li, and Z. Chen. A multi-modal dataset for teacher behavior analysis in offline classrooms. Scientific Data, 12(1):1115, 2025.
21. G. Huang et al. Multimodal pretraining for dense video captioning, 2020. arXiv preprint.
22. H. Jhuang, J. Gall, S. Zuffi, C. Schmid, and M. J. Black. Towards understanding action recognition. In Proceedings of the IEEE international conference on computer vision, pages 3192–3199, 2013.
23. J. Jia, J. Song, Q. Hu, S. Tang, and S. Xu. Tar: A dataset of teacher-teaching action recognition. In 2023 8th International Conference on Image, Vision and Computing (ICIVC), pages 676–681. IEEE, 2023.
24. X. Ju et al. Miradata: A large-scale video dataset with long durations and structured captions. Adv. Neural Inf. Process. Syst. (NeurIPS), 37:48955–48970, 2024.
25. R. Khirodkar et al. Harmony4D: A video dataset for in-the-wild close human interactions. Adv. Neural Inf. Process. Syst. (NeurIPS), 37:107270–107285, 2024.
26. Y. Kong and Y. Fu. Human action recognition and prediction: A survey. Int. J. Comput. Vis., 130(5):1366–1401, 2022.
27. K. Li, Y. Wang, Y. He, Y. Li, Y. Wang, Y. Liu, Z. Wang, J. Xu, G. Chen, P. Luo, et al. Mvbench: A comprehensive multi-modal video understanding benchmark. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 22195–22206, 2024.
28. Z.-Z. Li et al. From system 1 to system 2: A survey of reasoning large language models, 2025. arXiv preprint.
29. Z. Liao. CH-CC: A chinese multimodal classroom atmosphere analysis dataset based on teachers’ behavior and voice, 2023.
30. B. Lin et al. Video-LLaVA: Learning united visual representation by alignment before projection, 2023. arXiv preprint.
31. C.-Y. Lin. ROUGE: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pages 74–81, Barcelona, Spain, July 2004. Association for Computational Linguistics.
32. Q. Liu et al. Exploration of the characteristics of teachers’ multimodal behaviours in problem-oriented teaching activities with different response levels. Brit. J. Educ. Technol., 55(1):181–207, 2024.
33. K. Papineni et al. BLEU: A method for automatic evaluation of machine translation. In Proc. Annu. Meet. Assoc. Comput. Linguistics, pages 311–318, 2002.
34. T. Perrett et al. Hd-epic: A highly-detailed egocentric video dataset. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 23901–23913, June 2025.
35. A. Sabuncuoglu and T. M. Sezgin. Multimodal group activity dataset for classroom engagement level prediction, 2023. arXiv preprint.
36. M. R. Salehi et al. ActionAtlas: A videoqa benchmark for domain-specialized action recognition. Adv. Neural Inf. Process. Syst. (NeurIPS), 37:137372–137402, 2024.
37. V. Sharma, M. Gupta, A. Kumar, and D. Mishra. Edunet: A new video dataset for understanding human activity in the classroom environment. Sensors, 21(17):1424–8220, 2021.
38. Y. Shi, R. Liang, and Y. Xu. EducationQ: Evaluating LLMs’ teaching capabilities through multi-agent dialogue framework, 2025. arXiv preprint.
39. X. Tan, Y. Luo, Y. Ye, F. Liu, and Z. Cai. Allvb: All-in-one long video understanding benchmark. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 7211–7219, 2025.
40. Y. Tang, J. Bi, S. Xu, L. Song, S. Liang, T. Wang, D. Zhang, J. An, J. Lin, R. Zhu, A. Vosoughi, C. Huang, Z. Zhang, P. Liu, M. Feng, F. Zheng, J. Zhang, P. Luo, J. Luo, and C. Xu. Video understanding with large language models: A survey. IEEE Transactions on Circuits and Systems for Video Technology, 36(2):1355–1376, 2026.
41. R. Vedantam, C. Lawrence Zitnick, and D. Parikh. Cider: Consensus-based image description evaluation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4566–4575, 2015.
42. Q. Wang et al. Koala-36m: A large-scale video dataset improving consistency between fine-grained conditions and video content. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 8428–8437, June 2025.
43. J. Xiao, N. Huang, H. Qin, D. Li, Y. Li, F. Zhu, Z. Tao, J. Yu, L. Lin, T.-S. Chua, et al. Videoqa in the era of llms: An empirical study. International Journal of Computer Vision, 133(7):3970–3993, 2025.
44. J. Xu, Z. Guo, J. He, H. Hu, T. He, S. Bai, K. Chen, J. Wang, Y. Fan, K. Dang, B. Zhang, X. Wang, Y. Chu, and J. Lin. Qwen2.5-omni technical report, 2025.
45. L. Yan et al. Practical and ethical challenges of large language models in education: A systematic scoping review. Brit. J. Educ. Technol., 55(1):90–112, 2024.
46. Z. Yu, D. Xu, J. Yu, T. Yu, Z. Zhao, Y. Zhuang, and D. Tao. Activitynet-qa: A dataset for understanding complex web videos via question answering. In Proceedings of the AAAI conference on artificial intelligence, volume 33, pages 9127–9134, 2019.
47. Z. Zhang, Z. Jia, H. Wu, C. Li, Z. Chen, Y. Zhou, W. Sun, X. Liu, X. Min, W. Lin, and G. Zhai. Q-bench-video: Benchmark the video quality understanding of lmms. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 3229–3239, June 2025.
48. B. Zhou, H. Yang, D. Chen, J. Ye, T. Bai, J. Yu, S. Zhang, D. Lin, C. He, and W. Li. Urbench: A comprehensive benchmark for evaluating large multimodal models in multi-view urban scenarios. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 10707–10715, 2025.
49. L. Zhou, C. Xu, and J. Corso. Towards automatic learning of procedures from web instructional videos. In Proceedings of the AAAI conference on artificial intelligence, volume 32, pages 7590–7598, 2018.

© 2026 Copyright is held by the author(s). This work is distributed under the Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International (CC BY-NC-ND 4.0) license.