# Beyond Shallow Heuristics: Leveraging Human Intuition for Curriculum Learning

arXiv: https://arxiv.org/abs/2508.19873 · HTML: https://mlsanity.com/paper/2508.19873/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 3 of its 19 references that are in the map.
- First posted on arXiv 27 Aug 2025.

## Builds on (most cited first, 3 of 3)

- [Learning to Execute](https://mlsanity.com/paper/1410.4615/) (2014)
- [A Corpus for Automatic Readability Assessment and Text Simplification of German](https://mlsanity.com/paper/1909.09067/) (2019)
- [Data Selection Curriculum for Neural Machine Translation](https://mlsanity.com/paper/2203.13867/) (2022)

## Built on by (most cited first, 2 of 2)

- [Four Quadrants of Difficulty: A Simple Categorisation and its Limits](https://mlsanity.com/paper/2601.01488/) (2026)
- [Disentangling Curriculum Learning in NLP: Towards a Unifying Taxonomy](https://mlsanity.com/paper/2607.18984/) (2026)

## Abstract (arXiv)

Curriculum learning (CL) aims to improve training by presenting data from "easy" to "hard", yet defining and measuring linguistic difficulty remains an open challenge. We investigate whether human-curated simple language can serve as an effective signal for CL. Using the article-level labels from the Simple Wikipedia corpus, we compare label-based curricula to competence-based strategies relying on shallow heuristics. Our experiments with a BERT-tiny model show that adding simple data alone yields no clear benefit. However, structuring it via a curriculum -- especially when introduced first -- consistently improves perplexity, particularly on simple language. In contrast, competence-based curricula lead to no consistent gains over random ordering, probably because they fail to effectively separate the two classes. Our results suggest that human intuition about linguistic difficulty can guide CL for language model pre-training.