# Process-BERT: A Framework for Representation Learning on Educational Process Data

arXiv: https://arxiv.org/abs/2204.13607 · HTML: https://mlsanity.com/paper/2204.13607/

## Key facts

- Cited by 2 papers in the map (data through 7 Oct 2026).
- Builds on 5 of its 33 references that are in the map.
- First posted on arXiv 28 Apr 2022.

## Builds on (most cited first, 5 of 5)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [Stimuli-Sensitive Hawkes Processes for Personalized Student Procrastination Modeling](https://mlsanity.com/paper/2102.00089/) (2021)

## Built on by (most cited first, 2 of 2)

- [See, Think, Act: Online Shopper Behavior Simulation with VLM Agents](https://mlsanity.com/paper/2510.19245/) (2025)
- [LLM Prompt Evaluation for Educational Applications](https://mlsanity.com/paper/2601.16134/) (2026)

## Abstract (arXiv)

Educational process data, i.e., logs of detailed student activities in computerized or online learning platforms, has the potential to offer deep insights into how students learn. One can use process data for many downstream tasks such as learning outcome prediction and automatically delivering personalized intervention. However, analyzing process data is challenging since the specific format of process data varies a lot depending on different learning/testing scenarios. In this paper, we propose a framework for learning representations of educational process data that is applicable across many different learning scenarios. Our framework consists of a pre-training step that uses BERT-type objectives to learn representations from sequential process data and a fine-tuning step that further adjusts these representations on downstream prediction tasks. We apply our framework to the 2019 nation's report card data mining competition dataset that consists of student problem-solving process data and detail the specific models we use in this scenario. We conduct both quantitative and qualitative experiments to show that our framework results in process data representations that are both predictive and informative.