# Filling in the Mechanisms: How do LMs Learn Filler-Gap Dependencies under Developmental Constraints?

arXiv: https://arxiv.org/abs/2604.14459 · HTML: https://mlsanity.com/paper/2604.14459/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 24 of its 70 references that are in the map.
- First posted on arXiv 15 Apr 2026.

## Builds on (most cited first, 24 of 24)

- [The Pile: An 800GB Dataset of Diverse Text for Language Modeling](https://mlsanity.com/paper/2101.00027/) (2020)
- [Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling](https://mlsanity.com/paper/2304.01373/) (2023)
- [Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 small](https://mlsanity.com/paper/2211.00593/) (2022)
- [BLiMP: The Benchmark of Linguistic Minimal Pairs for English](https://mlsanity.com/paper/1912.00582/) (2019)
- [Targeted Syntactic Evaluation of Language Models](https://mlsanity.com/paper/1808.09031/) (2018)
- [Finding Alignments Between Interpretable Causal Variables and Distributed Neural Representations](https://mlsanity.com/paper/2303.02536/) (2023)
- [Neural Language Models as Psycholinguistic Subjects: Representations of Syntactic State](https://mlsanity.com/paper/1903.03260/) (2019)
- [Syntactic Structure from Deep Learning](https://mlsanity.com/paper/2004.10827/) (2020)
- [Findings of the BabyLM Challenge: Sample-Efficient Pretraining on Developmentally Plausible Corpora](https://mlsanity.com/paper/2504.08165/) (2025)
- [What do RNN Language Models Learn about Filler-Gap Dependencies?](https://mlsanity.com/paper/1809.00042/) (2018)
- [Probing for the Usage of Grammatical Number](https://mlsanity.com/paper/2204.08831/) (2022)
- [pyvene: A Library for Understanding and Improving PyTorch Models via Interventions](https://mlsanity.com/paper/2403.07809/) (2024)
- [How Linguistics Learned to Stop Worrying and Love the Language Models](https://mlsanity.com/paper/2501.17047/) (2025)
- [CausalGym: Benchmarking causal interpretability methods on linguistic tasks](https://mlsanity.com/paper/2402.12560/) (2024)
- [Using Priming to Uncover the Organization of Syntactic Representations in Neural Language Models](https://mlsanity.com/paper/1909.10579/) (2019)
- [Causal Interventions Reveal Shared Structure Across English Filler-Gap Constructions](https://mlsanity.com/paper/2505.16002/) (2025)
- [Pushdown Layers: Encoding Recursive Structure in Transformer Language Models](https://mlsanity.com/paper/2310.19089/) (2023)
- [Words, Subwords, and Morphemes: What Really Matters in the Surprisal-Reading Time Relationship?](https://mlsanity.com/paper/2310.17774/) (2023)
- [Verb Conjugation in Transformers Is Determined by Linear Encodings of Subject Number](https://mlsanity.com/paper/2310.15151/) (2023)
- [Different types of syntactic agreement recruit the same units within large language models](https://mlsanity.com/paper/2512.03676/) (2025)
- [The Impact of Token Granularity on the Predictive Power of Language Model Surprisal](https://mlsanity.com/paper/2412.11940/) (2024)
- [How Adults Understand What Young Children Say](https://mlsanity.com/paper/2206.07807/) (2022)
- [Reframing linguistic bootstrapping as joint inference using visually-grounded grammar induction models](https://mlsanity.com/paper/2406.11977/) (2024)
- [What Exactly do Children Receive in Language Acquisition? A Case Study on CHILDES with Automated Detection of Filler-Gap Dependencies](https://mlsanity.com/paper/2603.02082/) (2026)

## Abstract (arXiv)

For humans, filler-gap dependencies require a shared representation across different syntactic constructions. Although causal analyses suggest this may also be true for LLMs (Boguraev et al., 2025), it is still unclear if such a representation also exists for language models trained on developmentally feasible quantities of data. We applied Distributed Alignment Search (DAS, Geiger et al. (2024)) to LMs trained on varying amounts of data from the BabyLM challenge (Warstadt et al., 2023), to evaluate whether representations of filler-gap dependencies transfer between wh-questions and topicalization, which greatly vary in terms of their input frequency. Our results suggest shared, yet item-sensitive mechanisms may develop with limited training data. More importantly, LMs still require far more data than humans to learn comparable generalizations, highlighting the need for language-specific biases in models of language acquisition.