# Language acquisition: do children and language models follow similar learning stages?

arXiv: https://arxiv.org/abs/2306.03586 · HTML: https://mlsanity.com/paper/2306.03586/

## Key facts

- Cited by 41 papers in the map (data through 14 Oct 2026).
- Builds on 21 of its 62 references that are in the map.
- First posted on arXiv 6 Jun 2023.

## Builds on (most cited first, 21 of 21)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Pointer Sentinel Mixture Models](https://mlsanity.com/paper/1609.07843/) (2016)
- [Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language models](https://mlsanity.com/paper/2206.04615/) (2022)
- [Neural Network Acceptability Judgments](https://mlsanity.com/paper/1805.12471/) (2018)
- [Assessing the Ability of LSTMs to Learn Syntax-Sensitive Dependencies](https://mlsanity.com/paper/1611.01368/) (2016)
- [Colorless green recurrent networks dream hierarchically](https://mlsanity.com/paper/1803.11138/) (2018)
- [What Will it Take to Fix Benchmarking in Natural Language Understanding?](https://mlsanity.com/paper/2104.02145/) (2021)
- [Interpreting and improving natural-language processing (in machines) with natural language-processing (in the brain)](https://mlsanity.com/paper/1905.11833/) (2019)
- [What Artificial Neural Networks Can Tell Us About Human Language Acquisition](https://mlsanity.com/paper/2208.07998/) (2022)
- [The emergence of number and syntax units in LSTM language models](https://mlsanity.com/paper/1903.07435/) (2019)
- [Does syntax need to grow on trees? Sources of hierarchical inductive bias in sequence-to-sequence networks](https://mlsanity.com/paper/2001.03632/) (2020)
- [Cognitive Science in the era of Artificial Intelligence: A roadmap for reverse-engineering the infant language-learner](https://mlsanity.com/paper/1607.08723/) (2016)
- [Mechanisms for Handling Nested Dependencies in Neural-Network Language Models and Humans](https://mlsanity.com/paper/2006.11098/) (2020)
- [Word Acquisition in Neural Language Models](https://mlsanity.com/paper/2110.02406/) (2021)
- [jiant: A Software Toolkit for Research on General-Purpose Text Understanding Models](https://mlsanity.com/paper/2003.02249/) (2020)
- [Neural Language Models are not Born Equal to Fit Brain Data, but Training Helps](https://mlsanity.com/paper/2207.03380/) (2022)
- [Analysing Neural Language Models: Contextual Decomposition Reveals Default Reasoning in Number and Gender Assignment](https://mlsanity.com/paper/1909.08975/) (2019)
- [What they do when in doubt: a study of inductive biases in seq2seq learners](https://mlsanity.com/paper/2006.14953/) (2020)
- [Information-Restricted Neural Language Models Reveal Different Brain Regions' Sensitivity to Semantics, Syntax and Context](https://mlsanity.com/paper/2302.14389/) (2023)
- [Language Modelling as a Multi-Task Problem](https://mlsanity.com/paper/2101.11287/) (2021)
- [Causal Transformers Perform Below Chance on Recursive Nested Constructions, Unlike Humans](https://mlsanity.com/paper/2110.07240/) (2021)

## Built on by (most cited first, 30 of 41)

- [DoGE: Domain Reweighting with Generalization Estimation](https://mlsanity.com/paper/2310.15393/) (2023)
- [Characterizing Learning Curves During Language Model Pre-Training: Learning, Forgetting, and Stability](https://mlsanity.com/paper/2308.15419/) (2023)
- [A polar coordinate system represents syntax in large language models](https://mlsanity.com/paper/2412.05571/) (2024)
- [Fine-tuning Reinforcement Learning Models is Secretly a Forgetting Mitigation Problem](https://mlsanity.com/paper/2402.02868/) (2024)
- [Large Language Models: The Need for Nuance in Current Debates and a Pragmatic Perspective on Understanding](https://mlsanity.com/paper/2310.19671/) (2023)
- [Training Language Models via Neural Cellular Automata](https://mlsanity.com/paper/2603.10055/) (2026)
- [DevBench: A multimodal developmental benchmark for language learning](https://mlsanity.com/paper/2406.10215/) (2024)
- [A Systematic Assessment of Language Models with Linguistic Minimal Pairs in Chinese](https://mlsanity.com/paper/2411.06096/) (2024)
- [Babysit A Language Model From Scratch: Interactive Language Learning by Trials and Demonstrations](https://mlsanity.com/paper/2405.13828/) (2024)
- [From Babble to Words: Pre-Training Language Models on Continuous Streams of Phonemes](https://mlsanity.com/paper/2410.22906/) (2024)
- [Once Upon a Time: Interactive Learning for Storytelling with Small Language Models](https://mlsanity.com/paper/2509.15714/) (2025)
- [Baby's CoThought: Leveraging Large Language Models for Enhanced Reasoning in Compact Models](https://mlsanity.com/paper/2308.01684/) (2023)
- [Natural Language Processing RELIES on Linguistics](https://mlsanity.com/paper/2405.05966/) (2024)
- [Large Language Models as Psychological Simulators: A Methodological Guide](https://mlsanity.com/paper/2506.16702/) (2025)
- [The Mechanistic Emergence of Symbol Grounding in Language Models](https://mlsanity.com/paper/2510.13796/) (2025)
- [Orion-14B: Open-source Multilingual Large Language Models](https://mlsanity.com/paper/2401.12246/) (2024)
- [BAMBINO-LM: (Bilingual-)Human-Inspired Continual Pretraining of BabyLM](https://mlsanity.com/paper/2406.11418/) (2024)
- [A Distributional Perspective on Word Learning in Neural Language Models](https://mlsanity.com/paper/2502.05892/) (2025)
- [Unraveling Syntax: Language Modeling and the Substructure of Grammars](https://mlsanity.com/paper/2510.02524/) (2025)
- [Development of Cognitive Intelligence in Pre-trained Language Models](https://mlsanity.com/paper/2407.01047/) (2024)
- [ChemActor: Enhancing Automated Extraction of Chemical Synthesis Actions with LLM-Generated Data](https://mlsanity.com/paper/2506.23520/) (2025)
- [Discourse Heuristics For Paradoxically Moral Self-Correction](https://mlsanity.com/paper/2507.00985/) (2025)
- [Convergence and Divergence of Language Models under Different Random Seeds](https://mlsanity.com/paper/2509.26643/) (2025)
- [Evaluating Large Language Models' Ability Using a Psychiatric Screening Tool Based on Metaphor and Sarcasm Scenarios](https://mlsanity.com/paper/2309.10744/) (2023)
- [The potential -- and the pitfalls -- of using pre-trained language models as cognitive science theories](https://mlsanity.com/paper/2501.12651/) (2025)
- [Shades of Zero: Distinguishing Impossibility from Inconceivability](https://mlsanity.com/paper/2502.20469/) (2025)
- [Computer Vision Models Show Human-Like Sensitivity to Geometric and Topological Concepts](https://mlsanity.com/paper/2505.13281/) (2025)
- [The Learning Dynamics of Subword Segmentation for Morphologically Diverse Languages](https://mlsanity.com/paper/2511.09197/) (2025)
- [Emergence of Phonemic, Syntactic, and Semantic Representations in Artificial Neural Networks](https://mlsanity.com/paper/2601.18617/) (2026)
- [Humans and transformer LMs: Abstraction drives language learning](https://mlsanity.com/paper/2603.17475/) (2026)

## Abstract (arXiv)

During language acquisition, children follow a typical sequence of learning stages, whereby they first learn to categorize phonemes before they develop their lexicon and eventually master increasingly complex syntactic structures. However, the computational principles that lead to this learning trajectory remain largely unknown. To investigate this, we here compare the learning trajectories of deep language models to those of children. Specifically, we test whether, during its training, GPT-2 exhibits stages of language acquisition comparable to those observed in children aged between 18 months and 6 years. For this, we train 48 GPT-2 models from scratch and evaluate their syntactic and semantic abilities at each training step, using 96 probes curated from the BLiMP, Zorro and BIG-Bench benchmarks. We then compare these evaluations with the behavior of 54 children during language production. Our analyses reveal three main findings. First, similarly to children, the language models tend to learn linguistic skills in a systematic order. Second, this learning scheme is parallel: the language tasks that are learned last improve from the very first training steps. Third, some - but not all - learning stages are shared between children and these language models. Overall, these results shed new light on the principles of language acquisition, and highlight important divergences in how humans and modern algorithms learn to process natural language.