# MOOCRep: A Unified Pre-trained Embedding of MOOC Entities

arXiv: https://arxiv.org/abs/2107.05154 · HTML: https://mlsanity.com/paper/2107.05154/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 18 of its 35 references that are in the map.
- First posted on arXiv 12 Jul 2021.

## Builds on (most cited first, 18 of 18)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [Deep contextualized word representations](https://mlsanity.com/paper/1802.05365/) (2018)
- [Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translation](https://mlsanity.com/paper/1609.08144/) (2016)
- [SciBERT: A Pretrained Language Model for Scientific Text](https://mlsanity.com/paper/1903.10676/) (2019)
- [Variational Graph Auto-Encoders](https://mlsanity.com/paper/1611.07308/) (2016)
- [SPECTER: Document-level Representation Learning using Citation-informed Transformers](https://mlsanity.com/paper/2004.07180/) (2020)
- [Advances in Pre-Training Distributed Word Representations](https://mlsanity.com/paper/1712.09405/) (2017)
- [PyTorch-BigGraph: A Large-scale Graph Embedding System](https://mlsanity.com/paper/1903.12287/) (2019)
- [An Empirical Evaluation of doc2vec with Practical Insights into Document Embedding Generation](https://mlsanity.com/paper/1607.05368/) (2016)
- [Attentional Graph Convolutional Networks for Knowledge Concept Recommendation in MOOCs in a Heterogeneous View](https://mlsanity.com/paper/2006.13257/) (2020)
- [What Should I Learn First: Introducing LectureBank for NLP Education and Prerequisite Chain Learning](https://mlsanity.com/paper/1811.12181/) (2018)
- [Engaging with Massive Online Courses](https://mlsanity.com/paper/1403.3100/) (2014)
- [Personalized Course Sequence Recommendations](https://mlsanity.com/paper/1512.09176/) (2015)
- [Inferring Concept Prerequisite Relations from Online Educational Resources](https://mlsanity.com/paper/1811.12640/) (2018)
- [Segatron: Segment-Aware Transformer for Language Modeling and Understanding](https://mlsanity.com/paper/2004.14996/) (2020)

## Abstract (arXiv)

Many machine learning models have been built to tackle information overload issues on Massive Open Online Courses (MOOC) platforms. These models rely on learning powerful representations of MOOC entities. However, they suffer from the problem of scarce expert label data. To overcome this problem, we propose to learn pre-trained representations of MOOC entities using abundant unlabeled data from the structure of MOOCs which can directly be applied to the downstream tasks. While existing pre-training methods have been successful in NLP areas as they learn powerful textual representation, their models do not leverage the richer information about MOOC entities. This richer information includes the graph relationship between the lectures, concepts, and courses along with the domain knowledge about the complexity of a concept. We develop MOOCRep, a novel method based on Transformer language model trained with two pre-training objectives : 1) graph-based objective to capture the powerful signal of entities and relations that exist in the graph, and 2) domain-oriented objective to effectively incorporate the complexity level of concepts. Our experiments reveal that MOOCRep's embeddings outperform state-of-the-art representation learning methods on two tasks important for education community, concept pre-requisite prediction and lecture recommendation.