# Developmental approach reveals the statistical learning of Neural Language Models: Transformers generalize from the most abstract statistical patterns

arXiv: https://arxiv.org/abs/2606.27460 · HTML: https://mlsanity.com/paper/2606.27460/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 9 of its 61 references that are in the map.
- First posted on arXiv 25 Jun 2026.

## Builds on (most cited first, 9 of 9)

- [Assessing the Ability of LSTMs to Learn Syntax-Sensitive Dependencies](https://mlsanity.com/paper/1611.01368/) (2016)
- [Neural Language Models as Psycholinguistic Subjects: Representations of Syntactic State](https://mlsanity.com/paper/1903.03260/) (2019)
- [Frequency Effects on Syntactic Rule Learning in Transformers](https://mlsanity.com/paper/2109.07020/) (2021)
- [Grokking of Hierarchical Structure in Vanilla Transformers](https://mlsanity.com/paper/2305.18741/) (2023)
- [How Linguistics Learned to Stop Worrying and Love the Language Models](https://mlsanity.com/paper/2501.17047/) (2025)
- [Mechanisms for Handling Nested Dependencies in Neural-Network Language Models and Humans](https://mlsanity.com/paper/2006.11098/) (2020)
- [Neural reality of argument structure constructions](https://mlsanity.com/paper/2202.12246/) (2022)
- [Construction Grammar Provides Unique Insight into Neural Language Models](https://mlsanity.com/paper/2302.02178/) (2023)
- [Derivational Morphology Reveals Analogical Generalization in Large Language Models](https://mlsanity.com/paper/2411.07990/) (2024)

## Built on by (most cited first, 1 of 1)

- [Neural Language Models Learn the Contextual Distributions of Dependency Structures: a statistical learning theory to compositionality](https://mlsanity.com/paper/2609.34936/) (2026)

## Abstract (arXiv)

In this study, we use a developmental approach to investigate the statistical learning and mental representation of neural language models (NLM). A series of Generative Transformer models are trained on a synthetic grammar. The model states are saved at multiple stages in the course of training. Through analyzing how the internal representations of these models change in the developmental path, we found that NLMs acquire the most abstract global statistical knowledge at the beginning of learning and later acquire the relatively local statistical dependencies. This learning path contains many over-generalizations from the very beginning and these over-generalizations are gradually constrained in the later stage of learning. Based on this observation, we propose a new framework to explain the statistical learning and language cognition of NLMs.