ABSTRACT

Course recommender systems can assist students in identifying suitable or appealing courses by leveraging user interaction data, which shows previous engagements between users and courses. However, a prevalent issue with existing course recommender systems is their tendency to prioritize accuracy over explainability. The "black-box" nature of these complex models presents a challenge: accurately characterizing and modeling users’ preferences while also providing explicit, comprehensive, and explainable user profiles. To address this limitation, we propose a novel Knowledge Entity-Aware Model for course recommendation called KEAM, which supports explicit user profile generation based on detailed information from a knowledge graph to enhance students’ comprehension of the rationales behind the recommendations. Specifically, we exploit the information encoded in a knowledge graph to build connections between units using a neural network by replacing the hidden units. Next, the model is trained to capture students’ preferences and create user profiles for explainable recommendations. Comprehensive experiments have been conducted on two real-world online datasets to evaluate the proposed model’s effectiveness and explainability.

Keywords

1. INTRODUCTION

Massive Open Online Courses (MOOCs) have garnered considerable attention as an alternative educational pattern to traditional university classrooms, particularly during the COVID-19 pandemic. However, with a vast array of courses available, it can be challenging for students to select the appropriate ones. Therefore, course recommender systems demonstrate their potential in assisting students with course selection and effectively alleviating the problem of information overload  [14].

Given a set of historical courses that a user enrolled in, the key factor of generating effective recommendations relies on the accurate creation of the modeling of user preferences or the user’s profile. Many works have been proposed to construct user profiles that reflect their interests and preferences based on their historical data. For example, Pardos et al.  [21] proposed a skip-gram model to represent each course as an embedding vector based on all the historical courses to represent a user’s preference. Morsomme and Alferez  [18] used a topic model to build students’ interest profiles based on their historical data and check how their interests change over time. Then, a content-based match algorithm is used to recommend courses whose content best matches the student’s preferences. Jing and Tang  [10] proposed a model to train users’ latent preferences from their MOOC course page visit history and recommend courses by matching their interests and course information. Zhang et al.  [36] propose a hierarchical reinforcement learning algorithm to generate refined user profiles that remove the noisy courses to capture the user’s current academic interest accurately from the profile.

While the aforementioned methods excel in modeling user preferences, they do have some limitations. One significant drawback is the limited emphasis on explainability of the recommendations. Due to the ‘black-box’ nature of most deep learning models, the primary focus has often been on improving model accuracy, neglecting the crucial aspect of explainability in recommendations. As a result, the representations of user preferences often remain inscrutable. However, recent research has emphasized the importance of explainable recommender systems, which are not solely measured by accuracy but also incorporate aspects like transparency, rationality, and trust  [1, 24]. Previous research indicates that explainable course recommender systems can significantly improve students’ comprehension of the underlying reasons for recommendations, ultimately fostering trust and satisfaction in the systems  [14]. To address the issue of limited explainability, one approach is to provide a descriptive summary of the system’s understanding of the user’s profile or preferences, allowing users to review and adjust their profiles as needed  [3]. Given the importance of course selection, we consider explicitly presenting the user’s profile as an opportunity to enhance users’ understanding of the recommendation results. Another limitation of previous studies lies in their inadequate integration of external entities or information, such as knowledge concepts, to enhance the accuracy and depth of recommendations  [9, 31]. To handle this issue, we propose the integration of a knowledge graph into course recommender systems. This integration not only improves the accuracy and granularity of user profiling but also enhances the system’s interpretability.

To address the challenges mentioned above, we introduce a well-designed knowledge graph into course recommender systems and present a novel deep learning-based model called the Knowledge Entity-Aware Model (KEAM) for interpretable course recommendations. KEAM explicitly constructs user profiles leveraging knowledge and interaction graphs and provides personalized recommendations with enhanced explainability. Specifically, we first propose an autoencoder architecture that integrates course features from knowledge graphs into its hidden layers to generate comprehensive user profiles. Once the model is trained, we derive recommendations tailored to the obtained user profiles. Also, the knowledge entities in user profiles shed light on the rationale behind each course recommendation. To validate the effectiveness and explainability of our approach, we conduct extensive experiments on two real-world public datasets.

2. RELATED WORK

2.1 Course Recommendation

Course recommendation in the educational environment presents a multifaceted challenge, largely due to the diversity of factors influencing student course selection, including future career aspirations, skill enhancement goals, and credit requirements  [16].

Historically, the development of course recommendation systems predominantly focused on content-based or collaborative filtering (CF) methods  [10, 18, 19, 20]. Scholars Walk  [23] employs a randomized wandering approach to acknowledge sequential course relationships. Wagner et al.  [27] employed the traditional machine learning technique KNN to assist students in mitigating dropout risks. However, they did not provide discussions about the explainability of their model. With the development of deep learning, this technique is also widely used in course recommendation  [7, 36], like adapting the skip-gram model to learn course vector representations from enrollment sequences, enhancing recommendation diversity  [22]. The course2vec model  [21], utilizing a neural network framework, further exemplifies this trend, predicting course probability distributions for recommendations. Yu et al.  [34] presented an end-to-end hierarchical reinforcement learning (HRL)model for concept expansion in MOOCs, which employed a two-level HRL mechanism of seed selection and concept expansion. MoodleREC  [4] leverages keyword-based queries to generate ranked lists of LOs, incorporating quality indicators and social feedback to inform selections. Gao et al.  [6] propose a novel online course recommendation model that integrates a deep convolutional neural network with negative sequence mining. Despite their performance efficacy, the ‘black box’ nature of deep learning models raises interpretability concerns  [14, 15].

However, in the educational sphere, the ability to comprehend the recommendations made by these systems is crucial, highlighting the necessity for interpretable course recommender systems. As noted by Williamson and Kizilcec  [30], ‘the trust educators and learners place in a model is contingent on its explainability’. Thus, our primary objective is to delve into the potential of explainable course recommendation systems.

2.2 Knowledge Graph

Knowledge Graphs(KGs) encapsulate personalized information about students and courses, enhancing recommender systems’ ability to comprehend students’ specific needs and the characteristics of courses, which are directed graphs in which nodes represent resource entities and edges labeled relationships between them. In the course recommendation domain, nodes can represent students, courses, and their associated features and attributes, including categories, lecturers, types of courses(mandatory/elective), and so forth. Edges can represent direct relationships between two nodes, such as a student enrolling in a course, an instructor teaching a course, a course being a mandatory requirement, or a course being affiliated with a specific college.

Zhang et al.  [35] introduced collaborative knowledge graph embedding, leveraging entity semantic capturing capabilities of TransR  [13] to enhance implicit item representations. Ye et al.  [32] utilized knowledge graphs to enrich user and item low-dimensional entity representations, yet they encountered challenges in capturing complex, higher-order entity relationships. To address this limitation, Hu et al.  [8] and Ma et al.  [17] proposed the concept of meta-path representations, providing a detailed description of the interaction contexts between user-item pairs. Jiang et al.  [9] formulated the knowledge concept recommendation as an intensive learning task combined with MOOC knowledge graphs. Yang et al.  [31] explored how meta-graphs facilitate the discovery of nuanced user/item representations and the extraction of meaningful structures, enriching the semantics of recommendations.

In our work, we leverage KGs for course recommendation, not only enhancing performance but also significantly improving interpretability. Distinct from other knowledge graph-based recommender systems  [28, 29], our approach involves explicitly constructing a profile for each user. This strategy enhances the personalized aspect of our recommendations, allowing each user a deeper understanding of his/her individual preferences and behaviors, which can enhance the explainability of our model.

3. THE PROPOSED KEAM MODEL

The overview of the proposed model is shown in Figure 1. It consists of three components: User Profile Generation component, User Profile Refinement component, and Recommendation & Explanation component.

3.1 User Profile Generation

A neural network is fundamentally a framework for modeling any desired function, which typically consists of an input layer, hidden layers, and an output layer. Inspired by the autoencoder [2], our approach adopts an unsupervised methodology by setting the target value identical to the input. This strategy involves the model attempting to ‘predict’ input embedding from itself in the output layer. The basic process is to initially encode the input embedding into a low-dimension representation, followed by an attempt to decode this representation back to its original embedding (input embedding). It enables the model to learn significant latent features about input embedding in the hidden layers, thereby learning the latent embedding of the input data. Thus, the embeddings in the hidden layer can be construed as implicit factors.

Similar to other deep learning techniques, the architecture of this kind of autoencoder does not inherently reveal the specific meanings of these implicit factors. To develop a model with enhanced interpretability, we aim to integrate the knowledge graphs to design a knowledge entity-aware model to convert implicit factors into explicit factors. Specifically, we propose to give meaning to the connection with the hidden layer and each neuron by exploiting feature information explicitly in knowledge graphs. The user profile generation component mainly contains three parts: input layer, hidden layer, and output layer. For each user \(u_i\), the input layer receives the user \(u_i\)’s interaction information. It means the model aggregates the courses that \(u_i\) has interacted with, denoted as \(C(i)\). The input layer to the hidden layer should have been fully connected, but as mentioned previously, such a model would lose interpretability. So, the hidden layer and its connections are substituted by the knowledge graph, thus having an explicit representation of the meaning associated with both hidden nodes and their mutual connections. This means that each neuron represents an entity in the adopted KG, and the edge between two nodes exists if the corresponding KG entities are connected. For each course \(c_j\), We have relevant knowledge entities through knowledge graphs, denoted as \(K(j)\). After calculating the weights of the knowledge entities in the hidden layer, we obtain the output through a linear layer, which is as consistent with the input as possible.

Hence, in the training process, the KEAM model learns how to reconstruct the input information using the latent factors generated in the hidden layer. For each student, we have latent embeddings of the course features. In fact, these representations are no longer implicit in our model because each neuron corresponds to a knowledge entity in KGs. Given the sparsity of interaction data in the input layer, our approach incorporates randomly selected negative instances, which are courses with no interaction records. This inclusion is predicated on the assumption that the absence of interactions implies user disinterest rather than unawareness. By integrating both positive and negative instances, we aim to enhance the training efficacy and robustness of our model.

3.2 User Profile Refinement

During the generation of user profiles, a masking multiplier matrix is utilized to conceal the majority of neurons in the hidden layer. As a result, the generated profiles primarily consist of knowledge entities encountered in the user’s interaction history. Given that the number of interactions varies among users, this approach results in profiles with differing dimensions. Additionally, since users typically engage with only a small subset of available courses, their profiles tend to be limited in scope.

To address these limitations, we propose incorporating insights from the broader user community. By leveraging collaborative information, we can model user preferences more accurately and shape more robust profiles. Specifically, we introduce a refinement strategy that utilizes natural language processing (NLP) techniques for the semantic analysis of user profiles.

These prediction results are derived directly from the model’s output. Then, we select the word with the highest similarity score between any of the user’s past courses and this new course. This selected word is then incorporated into the user’s profile, ensuring that their profile is enriched with the most pertinent and representative knowledge entity preferences. By adopting this approach, we aim to create more comprehensive and accurate user profiles that reflect their unique interests and preferences.

3.3 Recommendation and Explanation

After refining user profiles, we employ a simple but highly effective collaborative filtering technique to generate recommendations. This involves calculating cosine similarity between user profiles to construct groups of similar users. We denote the similarity between a user \(u_i\) and another user \(u_k\) within this group as \(s_{i k}\). The prediction of course \(c_j\) is formulated by aggregating the comprehensive preferences of users within a similar user group, which can be systematically defined as: \[ R_{i j}=\frac {\sum _{k=1}^n s_{i k} \cdot r_{k, j}}{\sum _{k=1}^n\left |s_{i k}\right |}, \] where \(r_{k, j}\) means user \(u_k\) has interaction with course \(c_j\), value 1 if the interaction has taken place, 0 otherwise. Notably, our approach to collaborative filtering hinges on the similarity of user profiles rather than on interaction records. This distinction is crucial as it significantly enhances the interpretability of the recommendations, effectively addressing a common challenge associated with traditional collaborative filtering methods.

After scoring each course, we generate a personalized Top-N recommendation list tailored to each user. To improve comprehension, we present these recommendations alongside the user’s individual profile, which includes the knowledge entities associated with each suggested course. This approach provides a clear rationale for each recommendation, ensuring they align with the user’s specific interests and preferences.

4. EXPERIMENT

4.1 Dataset

In the context of this work, we focus on the scenario of course recommendation within an MOOC environment. The datasets in our analysis are collected from the XuetangX¹ MOOC platform. The XuetangX dataset provided by  [36] classifies courses into 23 various categories. The MOOCCube dataset, provided by  [33], includes 25,161 kinds of knowledge entities about courses to construct knowledge graphs in our work. The dataset statistics are introduced in Table 1.

In this paper, the XuetangX dataset presented certain limitations that are the majority of courses being confined to a singular category, thereby limiting the construction of a holistic knowledge graph. Hence, we extend the categorization framework from the existing 23 categories to 243 category attributes on the XuetangX dataset.

4.2 Experimental Settings

The datasets were split into training, validation, and testing sets, where \(x\%\) was used for training and validation, and \((1-x\%)\) was used for testing. We used 90\(\%\) as the value of \(x\). To estimate the effectiveness of our model, we rely on three widely recognized metrics: Recall@\(K\), Hit Ratio(HR@\(K\)) and Normalized Discounted Cumulative Gain(NDCG@\(k\)) of Top-\(K\) recommendations.

We compared our model to different baseline methods given below:

• PMF  [25], a traditional recommendation model that relies solely on the user-item rating matrix.
• NMF  [12], factorizes a non-negative matrix into the product of two or more non-negative matrices based on the user-item interaction matrix.
• Item-based KNN  [26], models user and item based on item similarity obtained by interaction information.
• Popularity [5], provides the most popular items for users.
• LightFM \(+\) BPR/WARP  [11], combines collaborative filtering and content-based technique utilizing the interaction information and features of users and items to obtain recommendations, these two baselines differ in the loss function. Bayesian Personalized Ranking (BPR) focuses on optimizing overall relative ranking, while Weighted Approximate-Rank Pairwise (WARP) focuses on top-ranking accuracy.

4.3 Performance Comparison

Table 2 and Table 3 show the overall result, and we summarize the result as follows:

• KEAM outperforms all baseline models and accomplishes a notable enhancement in recommendation performance on both two datasets. Our model incorporates knowledge entities and effectively integrates knowledge graph information while generating user profiles. Moreover, our model leverages a method to fuse diverse information in education scenarios. This underscores the effectiveness of our model in handling and utilizing complex data structures while achieving improved recommendation accuracy.
• LightFM, a hybrid recommender system that is well-known for its stability and popularity in the recommendation domain, has been further enhanced in our study by its combination with Bayesian Personalized Ranking (BPR) to address implicit feedback. While LightFM combined with BPR (LightFM+BPR) and combined with WARP (LightFM + WARP) show commendable performance in course recommendation scenarios, it falls short of the effectiveness exhibited by our proposed model. The reason for this disparity can be attributed to their oversight of the knowledge graph’s potential. Furthermore, our model, KEAM, explicitly constructs user profiles, significantly enhancing the interpretability of the recommendations. Therefore, we argue that our model not only demonstrates superior transparency but also outperforms LightFM+BPR in terms of overall performance.
• Overall, each model exhibits superior performance on the MOOCCube dataset compared to the XuetangX dataset. We argue that the enhanced performance is due to the greater density of user-course interactions within the MOOCCube dataset and its relatively lower sparsity. These natures contribute to the ability of models to learn user preferences more effectively.

5. CONCLUSION

In this paper, we introduce a novel Knowledge Entity-Aware Model for explainable course recommendations, which can make recommendations explainable and explicitly build students’ profiles by incorporating knowledge entities in knowledge graphs. We highlight the significance of knowledge graphs, which can help generate more accurate and explainable recommendations. Moreover, we replace the hidden neurons utilizing the knowledge entities in knowledge graphs to explicitly create user profiles. Through extensive experiments, we validate the effectiveness and explainability of KEAM.

However, there are some limitations in our current model. One significant constraint is that it treats all knowledge entities uniformly, disregarding their hierarchical relationships. This approach may constrain both the model’s performance and interpretability. To address this, we plan to enhance our model in the future by incorporating hierarchical knowledge entities. Additionally, we intend to conduct a comprehensive user study involving real users.

6. ACKNOWLEDGMENTS

This work was supported by JSPS KAKENHI Grant Numbers JP17KT0154, JP20H00622 and JP24K20903.

7. REFERENCES

1. O. Alkan, E. M. Daly, A. Botea, A. N. Valente, and P. Pedemonte. Where can my career take me? harnessing dialogue for interactive career goal recommendations. In Proceedings of the 24th International Conference on Intelligent User Interfaces, pages 603–613, 2019.
2. V. Bellini, A. Schiavone, T. Di Noia, A. Ragone, and E. Di Sciascio. Knowledge-aware autoencoders for explainable recommender systems. In Proc. 3rd Workshop Deep Learning for Recommender Systems, pages 24–31, 2018.
3. M. A. Chatti, M. Guesmi, and A. Muslim. Visualization for recommendation explainability: A survey and new perspectives. arXiv preprint arXiv:2305.11755, 2023.
4. C. De Medio, C. Limongelli, F. Sciarrone, and M. Temperini. Moodlerec: A recommendation system for creating courses using the moodle e-learning platform. Computers in Human Behavior, 104:106168, 2020.
5. A. Elbadrawy and G. Karypis. Domain-aware grade prediction and top-n course recommendation. In Proceedings of the 10th ACM conference on recommender systems, pages 183–190, 2016.
6. M. Gao, Y. Luo, and X. Hu. Online course recommendation using deep convolutional neural network with negative sequence mining. Wireless Communications and Mobile Computing, 2022, 2022.
7. J. Gong, S. Wang, J. Wang, W. Feng, H. Peng, J. Tang, and P. S. Yu. Attentional graph convolutional networks for knowledge concept recommendation in moocs in a heterogeneous view. In Proceedings of the 43rd international ACM SIGIR conference on research and development in information retrieval, pages 79–88, 2020.
8. B. Hu, C. Shi, W. X. Zhao, and P. S. Yu. Leveraging meta-path based context for top-n recommendation with a neural co-attention model. In Proceedings of the 24th ACM SIGKDD international conference on knowledge discovery & data mining, pages 1531–1540, 2018.
9. L. Jiang, K. Liu, Y. Wang, D. Wang, P. Wang, Y. Fu, and M. Yin. Reinforced explainable knowledge concept recommendation in moocs. ACM Trans. Intell. Syst. Technol., 14(3), apr 2023.
10. X. Jing and J. Tang. Guess you like: course recommendation in moocs. In Proceedings of the international conference on web intelligence, pages 783–789, 2017.
11. M. Kula. Metadata embeddings for user and item cold-start recommendations. arXiv preprint arXiv:1507.08439, 2015.
12. D. Lee and H. Seung. Learning the parts of objects by non-negative matrix factorization. Nature, 401:788–91, 11 1999.
13. Y. Lin, Z. Liu, M. Sun, Y. Liu, and X. Zhu. Learning entity and relation embeddings for knowledge graph completion. In Proceedings of the AAAI conference on artificial intelligence, volume 29, 2015.
14. B. Ma, M. Lu, Y. Taniguchi, and S. Konomi. Courseq: the impact of visual and interactive course recommendation in university environments. Research and practice in technology enhanced learning, 16:1–24, 2021.
15. B. Ma, M. Lu, Y. Taniguchi, and S. Konomi. Exploration and explanation: An interactive course recommendation system for university environments. In IUI Workshops, 2021.
16. B. Ma, M. Lu, Y. Taniguchi, and S. Konomi. Investigating course choice motivations in university environments. Smart Learning Environments, 8(1):1–18, 2021.
17. W. Ma, M. Zhang, Y. Cao, W. Jin, C. Wang, Y. Liu, S. Ma, and X. Ren. Jointly learning explainable rules for recommendation with knowledge graph. In The world wide web conference, pages 1210–1221, 2019.
18. R. Morsomme and S. V. Alferez. Content-based course recommender system for liberal arts education. International educational data mining society, 2019.
19. S. Morsy, G. Karypis, et al. Will this course increase or decrease your gpa? towards grade-aware course recommendation. Journal of Educational Data Mining, 11(2):20–46, 2019.
20. J. Naren, M. Z. Banu, and S. Lohavani. Recommendation system for students’ course selection. In Smart Systems and IoT: Innovations in Computing: Proceeding of SSIC 2019, pages 825–834. Springer, 2020.
21. Z. A. Pardos, Z. Fan, and W. Jiang. Connectionist recommendation in the wild: on the utility and scrutability of neural networks for personalized course guidance. User modeling and user-adapted interaction, 29:487–525, 2019.
22. Z. A. Pardos and W. Jiang. Combating the filter bubble: Designing for serendipity in a university course recommendation system. arXiv preprint arXiv:1907.01591, 2019.
23. A. Polyzou, A. N. Nikolakopoulos, and G. Karypis. Scholars walk: A markov chain framework for course recommendation. International Educational Data Mining Society, 2019.
24. P. Pu, L. Chen, and R. Hu. Evaluating recommender systems from the user’s perspective: survey of the state of the art. User Modeling and User-Adapted Interaction, 22:317–355, 2012.
25. R. Salakhutdinov and A. Mnih. Probabilistic matrix factorization. In Proceedings of the 20th International Conference on Neural Information Processing Systems, pages 1257–1264, 2007.
26. B. Sarwar, G. Karypis, J. Konstan, and J. Riedl. Item-based collaborative filtering recommendation algorithms. In Proceedings of the 10th international conference on World Wide Web, pages 285–295, 2001.
27. K. Wagner, A. Merceron, P. Sauer, and N. Pinkwart. Can the paths of successful students help other students with their course enrollments? In Proceedings of the 16th International Conference on Educational Data Mining, pages 171–182, Bengaluru, India, July 2023. International Educational Data Mining Society.
28. H. Wang, M. Zhao, X. Xie, W. Li, and M. Guo. Knowledge graph convolutional networks for recommender systems. In The world wide web conference, pages 3307–3313, 2019.
29. X. Wang, X. He, Y. Cao, M. Liu, and T.-S. Chua. Kgat: Knowledge graph attention network for recommendation. In Proceedings of the 25th ACM SIGKDD international conference on knowledge discovery & data mining, pages 950–958, 2019.
30. K. Williamson and R. F. Kizilcec. Effects of algorithmic transparency in bayesian knowledge tracing on trust and perceived accuracy. In Proceedings of the 14th International Conference on Educational Data Mining, EDM 2021, virtual, June 29 - July 2, 2021. International Educational Data Mining Society, 2021.
31. Y. Yang, C. Zhang, X. Song, Z. Dong, H. Zhu, and W. Li. Contextualized knowledge graph embedding for explainable talent training course recommendation. ACM Trans. Inf. Syst., 42(2), sep 2023.
32. Q. Ye, C.-Y. Hsieh, Z. Yang, Y. Kang, J. Chen, D. Cao, S. He, and T. Hou. A unified drug–target interaction prediction framework based on knowledge graph and recommendation system. Nature communications, 12(1):6775, 2021.
33. J. Yu, G. Luo, T. Xiao, Q. Zhong, Y. Wang, W. Feng, J. Luo, C. Wang, L. Hou, J. Li, Z. Liu, and J. Tang. MOOCCube: A large-scale data repository for NLP applications in MOOCs. In D. Jurafsky, J. Chai, N. Schluter, and J. Tetreault, editors, Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, pages 3135–3142, Online, July 2020. Association for Computational Linguistics.
34. J. Yu, C. Wang, G. Luo, L. Hou, J. Li, J. Tang, M. Huang, and Z. Liu. Expanrl: Hierarchical reinforcement learning for course concept expansion in moocs. In Proceedings of the 1st conference of the asia-pacific chapter of the association for computational linguistics and the 10th international joint conference on natural language processing, pages 770–780, 2020.
35. F. Zhang, N. J. Yuan, D. Lian, X. Xie, and W.-Y. Ma. Collaborative knowledge base embedding for recommender systems. In Proceedings of the 22nd ACM SIGKDD international conference on knowledge discovery and data mining, pages 353–362, 2016.
36. J. Zhang, B. Hao, B. Chen, C. Li, H. Chen, and J. Sun. Hierarchical reinforcement learning for course recommendation in moocs. In Proc. AAAI conf artificial intelligence, volume 33(1), pages 435–442, 2019.