# On the effect of curriculum learning with developmental data for grammar acquisition

arXiv: https://arxiv.org/abs/2311.00128 · HTML: https://mlsanity.com/paper/2311.00128/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 6 of its 18 references that are in the map.
- First posted on arXiv 31 Oct 2023.

## Builds on (most cited first, 6 of 6)

- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Pointer Sentinel Mixture Models](https://mlsanity.com/paper/1609.07843/) (2016)
- [HuggingFace's Transformers: State-of-the-art Natural Language Processing](https://mlsanity.com/paper/1910.03771/) (2019)
- [Grokking of Hierarchical Structure in Vanilla Transformers](https://mlsanity.com/paper/2305.18741/) (2023)
- [Curriculum learning for language modeling](https://mlsanity.com/paper/2108.02170/) (2021)
- [How to Plant Trees in Language Models: Data and Architectural Effects on the Emergence of Syntactic Inductive Biases](https://mlsanity.com/paper/2305.19905/) (2023)

## Built on by (most cited first, 5 of 5)

- [Findings of the BabyLM Challenge: Sample-Efficient Pretraining on Developmentally Plausible Corpora](https://mlsanity.com/paper/2504.08165/) (2025)
- [Banyan: Improved Representation Learning with Explicit Structure](https://mlsanity.com/paper/2407.17771/) (2024)
- [Influence-driven Curriculum Learning for Pre-training on Limited Data](https://mlsanity.com/paper/2508.15475/) (2025)
- [TRA: Better Length Generalisation with Threshold Relative Attention](https://mlsanity.com/paper/2503.23174/) (2025)
- [Level Up: Defining and Exploiting Transitional Problems for Curriculum Learning](https://mlsanity.com/paper/2603.13761/) (2026)

## Abstract (arXiv)

This work explores the degree to which grammar acquisition is driven by language `simplicity' and the source modality (speech vs. text) of data. Using BabyBERTa as a probe, we find that grammar acquisition is largely driven by exposure to speech data, and in particular through exposure to two of the BabyLM training corpora: AO-Childes and Open Subtitles. We arrive at this finding by examining various ways of presenting input data to our model. First, we assess the impact of various sequence-level complexity based curricula. We then examine the impact of learning over `blocks' -- covering spans of text that are balanced for the number of tokens in each of the source corpora (rather than number of lines). Finally, we explore curricula that vary the degree to which the model is exposed to different corpora. In all cases, we find that over-exposure to AO-Childes and Open Subtitles significantly drives performance. We verify these findings through a comparable control dataset in which exposure to these corpora, and speech more generally, is limited by design. Our findings indicate that it is not the proportion of tokens occupied by high-utility data that aids acquisition, but rather the proportion of training steps assigned to such data. We hope this encourages future research into the use of more developmentally plausible linguistic data (which tends to be more scarce) to augment general purpose pre-training regimes.