# EduBERT: Pretrained Deep Language Models for Learning Analytics

arXiv: https://arxiv.org/abs/1912.00690 · HTML: https://mlsanity.com/paper/1912.00690/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 2 Dec 2019.

## Built on by (most cited first, 3 of 3)

- [Exploring Bayesian Deep Learning for Urgent Instructor Intervention Need in MOOC Forums](https://mlsanity.com/paper/2104.12643/) (2021)
- [The Life Cycle of Large Language Models: A Review of Biases in Education](https://mlsanity.com/paper/2407.11203/) (2024)
- [Deep Learning for Opinion Mining and Topic Classification of Course Reviews](https://mlsanity.com/paper/2304.03394/) (2023)

## Abstract (arXiv)

The use of large pretrained neural networks to create contextualized word embeddings has drastically improved performance on several natural language processing (NLP) tasks. These computationally expensive models have begun to be applied to domain-specific NLP tasks such as re-hospitalization prediction from clinical notes. This paper demonstrates that using large pretrained models produces excellent results on common learning analytics tasks. Pre-training deep language models using student forum data from a wide array of online courses improves performance beyond the state of the art on three text classification tasks. We also show that a smaller, distilled version of our model produces the best results on two of the three tasks while limiting computational cost. We make both models available to the research community at large.