ABSTRACT

Urgency detection in online course forums aims to identify posts that need a timely instructor response. A key challenge is cross-context generalization: models trained on Western MOOC forums may not work well in non-Western settings where help-seeking norms and urgency cues differ. This study trains an urgency model on a Western MOOC dataset and evaluates it on a second Western benchmark and a new Indian dataset of 305 forum posts (two-coder weighted κ=0.7308). Using a 3-level actionability mapping (non-actionable, TF-actionable, instructor-actionable), the Western-trained model performs worse in the Indian setting than in the Western benchmark (Stanford ρ=0.381 vs. Indian ρ=0.292) and retrieves few instructor-actionable posts under limited review (Indian recall@50=0.194). Class balancing yields a small gain (Indian ρ=0.306), while adding a small amount of labeled Indian training data improves performance on a held-out Indian test split (ρ=0.401; recall@50=0.511) without materially changing Western benchmark dataset results.

Keywords

INTRODUCTION

Discussion forums are a primary channel for learner support in MOOCs and other large-scale online courses. Learners use them to ask questions, report technical issues, and seek clarification, but at scale they also create a recurring bottleneck where teaching teams must decide what to respond to first. Some posts describe time-sensitive blockers (e.g., access problems, deadline constraints, exam logistics, or escalating confusion) where delayed help can meaningfully affect learning progress. This motivates urgency detection: prioritizing posts that need faster staff attention under limited capacity.

Educational data mining has increasingly treated forum discourse as actionable data, enabling instructor-facing support tools that go beyond participation summaries. Prior work has studied closely related prioritization problems such as predicting when an instructor is likely to intervene in a discussion thread, using a combination of language cues and discussion structure [5,6].

A key concern is generalization. Urgency is shaped not only by what is being asked, but also by how learners express time pressure and needs which are patterns that can vary across platforms, course designs, and broader communication norms. Most models are built and validated within relatively narrow settings, which makes it unclear whether they remain reliable when deployed in new contexts. This matters because the highest-urgency, instructor-actionable cases are often the most operationally important to detect yet may be sparse in training data.

This paper frames urgency detection explicitly as a cross-context robustness problem. It extends the work of Švábenský et al. [12] and evaluates whether urgency models trained in a Western MOOC context transfer to (i) a second Western benchmark dataset and (ii) a non-Western MOOC forum context, and it examines two pragmatic strategies for improving reliability in the new setting: incorporating a small amount of labeled target-context data and addressing class imbalance for instructor-actionable posts.

We address the following research questions:

RQ1: How well do urgency detection models trained on Western MOOC forum data generalize to a non-Western MOOC forum context?

RQ2: How does class imbalance affect model performance under cross-context transfer, especially for instructor-actionable posts?

RQ3: To what extent does adding a small amount of labeled non-Western data to the training data improve model performance?

Related Work

Urgency detection in MOOC discussion forums

Urgency detection is typically framed as predicting which learner posts need timely staff attention in large-scale online courses. Much of the early work used the Stanford MOOCPosts benchmark [1] and simplified the 7-point urgency construct into a binary “urgent vs. not urgent” decision, highlighting that truly urgent cases are rare and easy to miss in large volumes of routine posts [2]. This line of work positions urgency as an instructor-support problem: models are valuable only insofar as they help teaching teams prioritize limited response capacity.

More recent work has emphasized that strong performance on a single benchmark can overstate real-world usefulness when course topics, learner populations, or platform norms change. Švábenský et al. [12] made this concern explicit by evaluating urgency prediction under dataset shift and by treating urgency as an ordered construct rather than only a binary cutoff, clarifying that generalization is a central challenge for deployment.

Urgency detection is also related to the broader problem of predicting instructor intervention in MOOC discussion forums. Chaturvedi et al. [6] studied whether instructor intervention could be predicted from MOOC forum activity and showed that intervention prediction depends on signals beyond the post text itself, including course and discussion context. Chandrasekaran et al. [5] similarly examined instructor intervention across MOOC forum threads and highlighted the role of forum structure and instructional context in determining when staff responses are needed. These studies show that actionability may not be a property of post text alone. Whether a post requires a teaching assistant or instructor response can depend on the course design, available support channels, learner expectations, and local instructional practices.

From feature engineering to embedding-based models

Modeling approaches for urgency detection have broadly followed trends in NLP for educational text: from lexical feature baselines to representation learning with neural embeddings. For example, Guo et al. [7] proposed a deep learning approach for identifying urgent posts in MOOC forums, and Khodeir [8] used BERT-based text representations with a recurrent architecture for urgency classification. These approaches aim to capture meaning beyond surface word counts and to be more tolerant of noisy forum writing.

A parallel strand incorporates additional signals beyond raw text, such as post metadata or learner/context cues, motivated by the idea that urgency is not only “what is said” but also “when and how it is asked.” Alrajhi et al. [3] combined textual and numerical features for predicting urgent responses, and Yu et al. [13] explored Bayesian deep learning to estimate instructor intervention need with uncertainty, reflecting interest in decision support settings where false negatives on high-priority posts are costly.

Generalizability, class imbalance, and evaluation gaps

Two persistent issues recur across the literature: (i) limited cross-context generalization and (ii) severe class imbalance. Urgent posts are typically a small minority in Western MOOC benchmarks [2], which can encourage models that achieve good average error while still failing on the cases that matter most operationally. This motivates both evaluation protocols that stress robustness under shift [12] and training strategies that explicitly address imbalance. For example, Alrajhi et al. [4] focus on urgency detection under imbalance and study methods intended to improve detection of the minority class.

Finally, there is increasing recognition that “stronger” NLP architectures do not automatically translate into better educational utility without careful evaluation. Work on broader educational forum-post classification emphasizes systematic comparisons and robustness-focused metrics rather than assuming newer models dominate by default [9,10]. This motivates evaluations that reflect instructor-facing workflows (e.g., whether high-priority posts appear near the top of a ranked list), not only aggregate error.

Building directly on the cross-dataset framing in Švábenský et al. [12], this paper targets the remaining gap between benchmark performance and deployment-relevant reliability under context shift by evaluating transfer to a non-Western forum setting and by testing pragmatic strategies that can be applied when only limited target-context labels are available.

Research Methods

This section describes the data and procedures used to train and evaluate predictive models of discussion-forum post urgency. This paper’s methodology adapts the full pipeline from Towards Generalizable Detection of Urgency of Discussion Forum Posts [12] including the 7-point ordinal urgency construct, coder protocol, lightweight NLP preprocessing, and cross-dataset generalizability evaluation. It then extends it in two ways that align with the motivation and gaps: (1) evaluation on an additional non-Western MOOC forum dataset labeled on the same scale, and (2) two deployment-relevant strategies intended to improve robustness under context shift: low-resource target-domain adaptation and class-imbalance mitigation.

Data Collection and Properties

This paper reuses the datasets used by Švábenský et al. [12]: (1) UPenn MOOC forum-post dataset (used here as the main training dataset), and (2) the Stanford MOOCPosts dataset (used here as a separate Western test dataset). In addition, this paper introduces a new Indian MOOC forum-post dataset to test how well models trained on Western MOOC posts work in a different context.

UPenn MOOC forum posts (Western training dataset). Following Švábenský et al. (2023), the training data consists of forum posts from nine UPenn MOOCs offered between 2012–2015. The original dataset construction sampled 500 posts per course and removed posts that were non-English or contained only symbols, only math, or only links, resulting in 3,503 posts from 2,882 students. Each post includes a student ID, timestamp, and post text. Typos and grammatical errors were kept as written.

Stanford MOOCPosts (Western test dataset). Performance is evaluated on the Stanford MOOCPosts dataset, a benchmark dataset introduced by Agrawal and Paepcke [1]. It contains 29,604 anonymized discussion-forum posts collected from 11 Stanford MOOCs. The 11 courses are grouped into three broad areas (Humanities/Sciences, Medicine, and Education) with approximately 10,000 posts per area. This benchmark is used here as a separate Western test dataset, consistent with its use for generalization testing in Švábenský et al. [12].

Indian MOOC forum posts (Non-Western dataset). A new dataset of 305 forum posts was collected from the Discourse forum of an online MOOC degree program in India. Posts were scraped from discussions occurring between 2024-2025 and were collected with the following fields: post ID, title, timestamp, and course ID. The raw text was cleaned by removing HTML and excluding posts that were not in English. The final sample includes posts drawn from four data science–related courses (Programming in Python; Statistics for Data Science; Business Data Management; Strategies for Professional Growth), with roughly 75 posts sampled per course. This dataset is used to evaluate performance in a new setting and to support later experiments that add a small amount of Indian training data.

Table 1 summarizes the three datasets, including their sources, number of courses, number of posts, and collection period.

Data Anonymization

The UPenn and Stanford datasets were used in their already de-identified forms [12], without additional anonymization. For the Indian dataset, personally identifying information was redacted during scraping using the GPT-4-0613 model-based procedure in Singhal et al. [11]. Each post was processed individually using the same prompt structure, which instructs the model to redact specific categories of PII and preserve the original text formatting. It was further checked during the labeling phase by two coders.

Data Labeling

This study adopts the urgency construct and labeling protocol used in [12] and then refines it. Urgency is defined as the degree to which an instructor response is needed quickly, expressed on a 7-point ordinal scale (1 = not urgent at all, 7 = extremely urgent) [12]. The same scale descriptions are used to ensure direct comparability across datasets and settings. Ordinal scores correspond to the following anchors:

1: No reason to read the post

2: Not actionable, read if time

3: Not actionable, may be interesting

4: Neutral, respond if spare time

5: Somewhat urgent, good idea to reply; a teaching assistant might suffice

6: Very urgent, good idea for the instructor to reply

7: Extremely urgent, instructor definitely needs to reply

For the Indian dataset, posts are labeled using the same scale and training structure as [12]: coders complete structured practice on held-out examples, then label posts independently. Inter-rater reliability is assessed using weighted Cohen’s κ (linear weights), consistent with the ordinal nature of the scale [12]. In this study, the Indian dataset was labeled by 2 coders and achieved κ = 0.7308. Disagreements were resolved via adjudication (discussion to consensus) to produce a single final label per post.

After the first round of labeling, it was observed that the urgency labels are imbalanced in both UPenn and Stanford datasets, with substantially more posts at the lower and middle urgency levels and comparatively fewer posts at the highest urgency levels (6–7). This skew is important because the rare high-urgency cases are the most instructionally actionable, yet they are underrepresented in the data. On the other hand, the Indian dataset contained a much higher proportion of high-urgency posts.

To align the prediction task with an instructor-facing triage workflow and reduce sparsity at the upper end of the urgency scale, the original 7-point urgency annotations were mapped into three actionability categories, as shown in Table 2. The 7-point scale was used for annotation and comparability with prior work, while the 3-level actionability labels were used for all model training and evaluation reported in this paper. This mapping preserves the key operational distinction in the original scale: whether a post is non-actionable, teaching-assistant-actionable, or instructor-actionable. The resulting 3-level scheme provides more stable categories for modeling and evaluation under severe class imbalance, and all reported metrics should therefore be interpreted on this 1–3 actionability scale rather than on the original 1–7 urgency scale.

Data Automated Pre-Processing

A lightweight preprocessing pipeline matching Švábenský et al. [12] was used to reduce superficial variation while retaining the linguistic cues that may signal urgency. The following steps were applied in order:

• Convert text to lowercase.
• Replace non-alphanumeric characters with spaces (preserving letters and digits).
• Normalize whitespace (remove duplicates).
• Remove common English stopwords.
• Apply stemming to reduce inflected forms to a shared root.

Spelling and grammar correction were not performed intentionally, as these artifacts may differ systematically across contexts.

Model Training and Cross-Validation

This section describes the shared model-training and validation procedure used to answer RQ1 (cross-context generalization), RQ2 (class-imbalance mitigation), and RQ3 (low-resource target-domain adaptation). Following the design principle established in Sections 3.1–3.4, the modeling pipeline holds the base learner and representation fixed to the strongest-performing configuration in [12] and varies only the training procedure for imbalance handling (RQ2) and training data composition (RQ3).

Model and feature representation

Urgency is modeled using support vector regression (SVR) with an RBF kernel, trained on Universal Sentence Encoder v4 (USE) embeddings of post text. SVR hyperparameters are left at default settings, consistent with the baseline configuration in [12]. Each post is embedded using USE v4 (TensorFlow Hub) to obtain a fixed-length dense representation, which is used directly as the input feature vector for SVR. The purpose of this study is not to establish a new state-of-the-art model for urgency detection, but to test cross-context robustness under a controlled and comparable modeling pipeline. For this reason, the USE + SVR configuration is held fixed across experiments. This design makes it possible to attribute changes in performance primarily to evaluation context, class-imbalance mitigation, and target-context supervision rather than to changes in model architecture.

Cross-validation protocol (student-level grouping)

Model selection and in-domain validation use 10-fold student-level cross-validation implemented with GroupKFold, where the grouping variable is the student identifier. This ensures that all posts from a given student occur in a single fold, preventing leakage across train and validation splits. For all cross-validation results, metrics are computed on the held-out fold and then averaged across folds.

For any setting in which training data combines UPenn and Indian examples (RQ3), student identifiers from the Indian dataset are offset by a large constant prior to grouping to avoid accidental ID collisions across datasets.

Training and evaluation settings by research question

RQ1: Cross-context generalization (Western → non-Western).

For RQ1, the model is trained using the full UPenn training dataset. Ten-fold student-level cross-validation is computed on UPenn. After cross-validation, the model is refit on the full UPenn training set and evaluated separately on the two external datasets: Stanford MOOCPosts (Western benchmark) and the Indian dataset (non-Western setting). External evaluations are performed without combining the two test datasets.

RQ2: Class-imbalance mitigation (in-fold SMOTE).

RQ2 evaluates whether mitigating label imbalance improves performance while holding the learner and representation fixed. In this condition, SMOTE oversampling is applied in the USE embedding space to rebalance the refined 3-level labels (Section 3.3). Oversampling is performed only on the training split within each cross-validation fold; validation folds remain untouched. Concretely, within each GroupKFold split, SMOTE is applied to the training fold to generate synthetic minority examples, SVR is trained on the resampled training fold, and evaluation is performed on the original validation fold.

Because minority-class counts vary across folds, the SMOTE neighborhood parameter is set adaptively per fold. If any class has fewer than two examples in the training fold, SMOTE is not well-defined; in that case, oversampling is skipped for that fold and SVR is trained on the unresampled training fold.

After cross-validation, a final RQ2 model for external testing is trained by applying SMOTE once to the full UPenn training set (again in embedding space) and fitting SVR on the resulting oversampled set. This model is then evaluated separately on Stanford MOOCPosts and the Indian dataset. No resampling is applied to any test set.

RQ3: Low-resource target-domain adaptation.

For RQ3, the goal is to test whether adding a small amount of labeled target-domain data improves performance in the Indian setting. The Indian dataset is split 50/50 into an Indian-train subset and an Indian-test subset using a stratified split over the refined 3-level urgency labels (Section 3.3) to preserve label proportions across the two halves. The training pool is then formed by concatenating all UPenn training examples with the Indian-train subset. Ten-fold student-level GroupKFold is run on this combined training pool. After cross-validation, the model is refit on the full combined training pool and evaluated separately on Stanford MOOCPosts and the held-out Indian-test subset.

Model Generalizability Evaluation

Model generalizability is evaluated by measuring performance under dataset shift and low-resource target-data inclusion, aligned with RQ1–RQ3. All reported results are computed on held-out data only (validation folds in cross-validation and external test sets). No resampling or rebalancing is applied to any validation or test split.

Performance is summarized using root mean squared error (RMSE) and Spearman’s rank correlation (ρ) between predicted urgency and gold labels. RMSE captures average prediction error magnitude, while Spearman’s ρ measures ordinal agreement and is appropriate for urgency as an ordered construct. Predictions are treated as continuous regression outputs and are not rounded prior to computing either metric.

Since the intended use case is prioritization under limited staff time, evaluation additionally includes top-k precision and recall for the highest actionability level. For each evaluation split, posts are ranked by predicted urgency (descending). “Instructor-actionable” corresponds to Level 3 in the refined label scheme (Section 3.3). Precision@k is the fraction of the top-k ranked posts that are instructor-actionable and Recall@k is the fraction of all instructor-actionable posts that appear in the top-k. Unless otherwise noted, results report k = 10 and 50 to reflect small and moderate review budgets.

For training settings that use the UPenn dataset, in-domain performance is estimated via 10-fold student-level cross-validation (Section 3.5). RMSE, Spearman’s ρ, and top-k precision/recall are computed on each held-out fold and then averaged across folds.

Generalizability across contexts is assessed through external evaluation on Stanford MOOCPosts and the Indian dataset. For RQ1, a model trained on UPenn is evaluated separately on Stanford and on the full Indian dataset. For RQ2, the class-balancing condition (SMOTE applied during training, Section 3.5) is evaluated separately on Stanford and on the full Indian dataset. For RQ3, the low-resource target-data condition is evaluated on Stanford and on the held-out Indian-test split (with the Indian 50/50 split defined in Section 3.5). Across conditions, results are reported for (i) UPenn cross-validation (when applicable), (ii) Stanford test performance, and (iii) Indian test performance, using RMSE, Spearman’s ρ, and the top-k instructor-actionable precision/recall metrics defined above.

RESULTS AND DISCUSSION

This section reports findings for RQ1–RQ3 using the refined 3-level actionability labels from Section 3.3 (1 = non-actionable, 2 = TF-actionable, 3 = instructor-actionable). Overall predictive performance is summarized with RMSE and Spearman’s ρ. To connect performance to an instructor-facing workflow, triage utility is summarized using precision@10 (concentration of instructor-actionable posts in the top 10) and recall@50 (coverage of instructor-actionable posts when reviewing the top 50).

All results are computed using the refined 3-level actionability labels, not the original 7-point annotation scale. SVR predictions are continuous and are not rounded before computing RMSE or Spearman’s ρ. Triage metrics are computed for label 3 (instructor-actionable) only. The Indian column is evaluated on the full Indian dataset for RQ1–RQ2 (N = 305; instructor-actionable = 93) and on the held-out Indian-test split for RQ3 (N = 153; instructor-actionable = 47).

RQ1: Cross-context generalization

The RQ1 results show that the UPenn-trained model transfers better to the Stanford benchmark than to the Indian dataset. On Stanford, the model achieves RMSE = 0.5895 and Spearman’s ρ = 0.3812. On the Indian dataset, performance drops to RMSE = 1.0061 and Spearman’s ρ = 0.2918. The triage-oriented results show a similar pattern: Indian precision@10 is 0.40 and recall@50 is 0.194, meaning that the model retrieves only a small share of instructor-actionable posts within a limited review budget.

One likely reason for this drop is the difference in actionability distributions. In the UPenn training data, instructor-actionable posts are rare, so the model learns urgency patterns from a setting where high-actionability cases are uncommon. In the Indian dataset, instructor-actionable posts are much more frequent and may also be expressed through different linguistic and pragmatic cues. As a result, the model is being asked to prioritize a different mix of learner needs than the one represented in its training data. This suggests a context-shift explanation rather than a purely “harder dataset” explanation.

These results should be interpreted as evidence of cross-context sensitivity, not as evidence of culture alone. The Indian dataset differs from the Western benchmarks along several dimensions, including platform, institutional setting, course design, learner population, and actionability distribution. The lower transfer performance suggests that a model trained in one benchmark setting may not fully capture the local cues that define instructor actionability in another.

RQ2: Class-imbalance mitigation

The RQ2 results show that SMOTE produces a small but limited improvement in the Indian setting. Compared with the UPenn-only model, the SMOTE model lowers Indian RMSE from 1.0061 to 0.9822 and increases Spearman’s ρ from 0.2918 to 0.3064. The triage-oriented metrics also improve as precision@10 rises from 0.40 to 0.60 and recall@50 increases from 0.194 to 0.237. These results suggest that class rebalancing helps the model move more instructor-actionable posts toward the top of the ranked list, which is useful for an instructor-facing triage workflow.

However, the improvement is limited. The SMOTE model still performs substantially worse on the Indian dataset than on the Stanford benchmark, so class imbalance cannot be the whole explanation for the transfer gap. SMOTE can rebalance the source training distribution, but it cannot introduce target-context cues that are missing from the UPenn data, such as how learners in the Indian setting signal urgency, what kinds of requests are common, or how problems are framed in that course and platform environment. In other words, increasing exposure to rare instructor-actionable cases helps, but it does not necessarily solve cross-context generalization.

This distinction matters because urgency detection is ultimately a triage task. A false positive may ask staff to review a lower-priority post, but a false negative may leave an instructor-actionable issue unresolved. The increase in precision@10 is therefore encouraging because it means the top of the ranked list contains more instructor-actionable posts. At the same time, recall@50 remains low, meaning that many instructor-actionable posts are still not retrieved within a realistic review budget. This is why top-k precision and recall are important alongside RMSE and Spearman’s ρ, because aggregate score accuracy can miss whether the model is helping staff find the posts that most need timely attention.

RQ3: Low-resource target-domain adaptation.

The RQ3 results show that adding a small amount of labeled Indian data produces the strongest improvement in the Indian setting. Compared with the UPenn-only model, the UPenn + Indian-train model lowers Indian RMSE from 1.0061 to 0.6932 and increases Spearman’s ρ from 0.2918 to 0.4012. The triage metrics also improve substantially as precision@10 rises from 0.40 to 0.60 and recall@50 increases from 0.194 to 0.511. This means that the adapted model retrieves more than half of the instructor-actionable posts in the held-out Indian test set within the top 50 ranked posts.

These gains suggest that even limited target-context supervision can help the model learn local patterns of actionability. Adding Indian training examples may help the model distinguish posts that can be handled by a teaching assistant from those that require instructor attention in this specific course and platform environment. This is especially important because the Indian dataset has a much larger share of instructor-actionable posts than the UPenn training data. The target-context examples therefore help recalibrate what high actionability looks like in the new setting.

At the same time, this result should be interpreted carefully. The RQ3 Indian result is evaluated on the held-out Indian test split rather than the full Indian dataset, so it is not directly identical to the RQ1 and RQ2 Indian evaluations. However, the direction and size of the improvement are still practically meaningful. The main implication is that local calibration may be more useful than source-only rebalancing when deploying urgency models in new educational contexts. The result does not imply that every institution needs a large labeled dataset before deployment. Rather, it suggests that even modest target-context supervision can improve triage utility when the target setting differs from the source training context.

Comparison with Existing Work

The results extend prior work on MOOC forum urgency detection in a more specific way than simply testing another dataset. Almatrafi et al. [2] framed urgent MOOC posts as a “needle in a haystack” problem: high-priority posts are rare but missing them can matter for learner support. That framing fits the UPenn training data used in this study, where instructor-actionable posts are extremely sparse. However, the Indian dataset complicates this assumption because instructor-actionable posts are not rare in the same way. This difference matters because a model trained in a source context where high-actionability cases are unusual may be poorly calibrated for a target context where such cases are much more common. Thus, the present findings suggest that urgency detection is not only a minority-class detection problem; it is also a distribution-shift problem.

This distinction also sharpens the contribution relative to Švábenský et al. [12]. Their work showed that urgency prediction should be evaluated under dataset shift and treated as an ordered construct rather than only as a binary urgent/not-urgent classification task. The present study follows that logic by retaining a comparable urgency construct and modeling pipeline but tests a different kind of shift. The UPenn-to-Stanford comparison evaluates transfer between Western MOOC benchmarks, while the UPenn-to-Indian comparison evaluates transfer to a setting that differs in platform, institutional context, learner population, and actionability distribution. The larger degradation on the Indian dataset shows that benchmark-to-benchmark generalization may understate the difficulty of deploying urgency models in substantially different educational settings.

The findings also connect urgency detection to the broader literature on instructor intervention. Chaturvedi et al. [6] modeled instructor intervention in MOOC forums using signals from forum activity and course context, and Chandrasekaran et al. [5] examined intervention prediction across forum threads, highlighting the role of forum structure and instructional context. The present results support this broader interpretation of actionability. A post’s need for instructor attention is not determined by its wording alone; it also depends on the course design, available support structures, and local expectations about who should respond. This helps explain why a model can perform reasonably on Stanford while degrading on the Indian dataset, even though all datasets contain MOOC forum posts. The relevant shift is not simply geographic or cultural, but instructional and operational.

The SMOTE results further qualify prior work on imbalance in urgency detection. Alrajhi et al. [4] focus on class imbalance as a central challenge for automatic urgency detection. The present results support the importance of that issue, since SMOTE improves the concentration of instructor-actionable posts near the top of the ranked list. At the same time, the improvement is limited: rebalancing the UPenn training data does not make the model transfer well to the Indian setting. This suggests that imbalance mitigation can improve exposure to rare source-context labels but cannot replace target-context information when the cues, frequency, or meaning of actionability differ across settings.

Finally, the results align with broader educational forum classification work by Sha et al. [9,10], who caution against evaluating educational text classifiers only through narrow model-performance comparisons. The present study makes a similar point from a deployment perspective. The main issue is not whether USE + SVR is the best possible architecture, but whether a model that appears useful under one evaluation context remains useful when the instructional context changes. By holding the model fixed, this study shows that changes in training composition and target context can substantially affect both ordinal prediction quality and triage utility. The contribution is therefore not a new classifier, but evidence that urgency detection systems need target-context validation, actionability-aware evaluation, and metrics that reflect how teaching teams review and prioritize forum posts.

Opportunities for Future Work

A first direction is to strengthen the generalization claims through broader and more realistic validation. This includes evaluating a fuller transfer matrix across UPenn, Stanford, Indian, and additional datasets to separate source-dataset, target-dataset, platform, and regional/contextual effects. It also includes collecting additional target-context datasets across multiple institutions, disciplines, and platforms, including potentially multilingual settings. Because the Indian dataset in this study is small and concentrated in data science-related courses, broader validation is needed before making stronger claims about non-Western MOOC contexts more generally. Future work should also test sensitivity to different sampling and split strategies and evaluate whether adaptation learned in one target context transfers to another.

In parallel, the triage framing can be made more decision-relevant by aligning k with actual staffing constraints, reporting class-wise false positives and false negatives, assigning higher costs to missed instructor-actionable posts, and evaluating the stability of actionability thresholds across contexts via calibration. A second direction is to move from “regression that is evaluated as triage” toward methods that are explicitly optimized for triage. This includes learning-to-rank or cost-sensitive objectives that prioritize instructor-actionable retrieval, alternatives to SMOTE that avoid synthetic embedding points, and richer adaptation strategies that leverage unlabeled target posts. Future work should also compare additional model families, including ordinal regression, random forests, transformer-based classifiers, and uncertainty-aware models. Finally, error analyses that categorize missed instructor-actionable posts by pedagogical function, and small instructor-in-the-loop studies that measure time saved and perceived usefulness, would connect offline metrics to instructional outcomes and clarify which improvements matter most in practice.

Conclusion

This paper extends prior work on MOOC forum-post urgency prediction by evaluating cross-context generalization in a setting that includes both Western benchmarks and a new non-Western forum dataset, while retaining a comparable modeling pipeline to isolate the effect of context shift. The results show that models trained on Western MOOC posts can generalize reasonably to another Western benchmark but degrade substantially when transferred to the Indian context, particularly on the instructor-actionable cases that are most important for instructional triage. Because the Indian dataset differs from the Western benchmarks along multiple dimensions, these findings should be interpreted as evidence of cross-context sensitivity rather than as proof of a purely cultural effect. Incorporating triage-oriented evaluation further clarifies this gap by highlighting how differences in ranking behavior under constrained review budgets can matter even when global performance metrics appear similar.

Across the robustness strategies examined, low-resource target-domain adaptation by adding a small amount of labeled target-context data provides the largest and most operationally meaningful improvements, outperforming class-imbalance mitigation via oversampling. Taken together, these findings suggest that reliable urgency triage across diverse educational contexts will likely require at least modest target-domain supervision and evaluation that reflects instructor workflows, rather than relying on source-only training and global fit metrics alone. For EDM systems intended to guide instructor attention, benchmark performance should therefore be accompanied by target-context validation, triage-oriented metrics, and explicit attention to missed instructor-actionable posts. Future work should broaden validation across additional contexts and move toward triage-optimized training and calibration so that actionability decisions remain stable and useful when deployed.

ACKNOWLEDGMENTS

The author thanks Švábenský et al. for making the UPenn MOOC forum-post dataset available for research use and for providing the methodological foundation for this study. The author also thanks the IIT Madras BS in Data Science and Applications program for granting permission to use the Indian forum data. The author is grateful to Hyejoo Yun and Jaidev Deshpande for their contributions to the initial conceptualization of the project.

ChatGPT was used for editing and language refinement during preparation of this paper; all research design, analysis, interpretation, and final content decisions remain the responsibility of the author.

REFERENCES

1. A. Agrawal and A. Paepcke. The Stanford MOOC Posts Dataset, 2014. http://infolab.stanford.edu/~paepcke/stanfordMOOCForumPostsSet.tar.gz.
2. O. Almatrafi, A. Johri, and H. Rangwala. Needle in a haystack: Identifying learner posts that require urgent response in MOOC discussion forums. Computers & Education, 118:1–9, 2018. https://doi.org/10.1016/j.compedu.2017.11.002.
3. L. Alrajhi, K. Alharbi, and A. I. Cristea. A Multidimensional Deep Learner Model of Urgent Instructor Intervention Need in MOOC Forum Posts. In V. Kumar and C. Troussas, editors, Intelligent Tutoring Systems, pages 226–236, Cham, 2020. Springer International Publishing. https://doi.org/10.1007/978-3-030-49663-0_27.
4. Alrajhi, L., Alamri, A., Pereira, F.D. et al. Solving the imbalanced data issue: automatic urgency detection for instructor assistance in MOOC discussion forums. User Model User-Adap Inter 34, 797–852 (2024). https://doi.org/10.1007/s11257-023-09381-y
5. M. K. Chandrasekaran, M.-Y. Kan, B. C. Y. Tan, and K. Ragupathi. Learning Instructor Intervention from MOOC Forums: Early Results and Issues. In Proceedings of the 8th International Conference on Educational Data Mining, pages 218–225, 2015. https://doi.org/10.48550/arXiv.1504.07206
6. S. Chaturvedi, D. Goldwasser, and H. Daumé III. Predicting Instructor’s Intervention in MOOC forums. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics, Volume 1: Long Papers, pages 1501–1511, 2014. https://doi.org/10.3115/v1/P14-1141.
7. S. X. Guo, X. Sun, S. X. Wang, Y. Gao, and J. Feng. Attention-Based Character-Word Hybrid Neural Networks With Semantic and Structural Information for Identifying of Urgent Posts in MOOC Discussion Forums. IEEE Access, 7:120522–120532, 2019. https://doi.org/10.1109/ACCESS.2019.2929211.
8. N. A. Khodeir. Bi-GRU Urgent Classification for MOOC Discussion Forums Based on BERT. IEEE Access, 9:58243–58255, 2021. https://doi.org/10.1109/ACCESS.2021.3072734.
9. L. Sha, M. Raković, Y. Li, A. Whitelock-Wainwright, D. Carroll, D. Gašević, and G. Chen. Which Hammer Should I Use? A Systematic Evaluation of Approaches for Classifying Educational Forum Posts. International Educational Data Mining Society, 2021. https://eric.ed.gov/?q=ED615664&id=ED615664.
10. L. Sha, M. Raković, J. Lin, Q. Guan, A. Whitelock-Wainwright, D. Gašević, and G. Chen. Is the Latest the Greatest? A Comparative Study of Automatic Approaches for Classifying Educational Forum Posts. IEEE Transactions on Learning Technologies, XX:1–14, 2022. https://doi.org/10.1109/TLT.2022.3227013.
11. S. Singhal, A. F. Zambrano, M. Pankiewicz, X. Liu, C. Porter, and R. S. Baker. De-Identifying Student Personally Identifying Information with GPT-4. In Proceedings of the 17th International Conference on Educational Data Mining, pages 559–565, 2024. https://doi.org/10.5281/zenodo.12729884.
12. V. Švábenský, R. S. Baker, A. Zambrano, Y. Zou, and S. Slater. Towards Generalizable Detection of Urgency of Discussion Forum Posts. In M. Feng, T. Käser, and P. Talukdar, editors, Proceedings of the 16th International Conference on Educational Data Mining, pages 302–309, Bengaluru, India, 2023. International Educational Data Mining Society. https://doi.org/10.5281/zenodo.8115790.
13. J. Yu, L. Alrajhi, A. Harit, Z. Sun, A. I. Cristea, and L. Shi. Exploring Bayesian Deep Learning for Urgent Instructor Intervention Need in MOOC Forums. In A. I. Cristea and C. Troussas, editors, Intelligent Tutoring Systems, pages 78–90, Cham, 2021. Springer International Publishing. https://doi.org/10.1007/978-3-030-80421-3_10.

© 2026 Copyright is held by the author(s). This work is distributed under the Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International (CC BY-NC-ND 4.0) license.