mlsanityData to 14 Oct 2026

Filling in the Mechanisms: How do LMs Learn Filler-Gap Dependencies under Developmental Constraints?

arXiv 2604.14459 ↗First posted 15 Apr 2026cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

For humans, filler-gap dependencies require a shared representation across different syntactic constructions. Although causal analyses suggest this may also be true for LLMs (Boguraev et al., 2025), it is still unclear if such a representation also exists for language models trained on developmentally feasible quantities of data. We applied Distributed Alignment Search (DAS, Geiger et al. (2024)) to LMs trained on varying amounts of data from the BabyLM challenge (Warstadt et al., 2023), to evaluate whether representations of filler-gap dependencies transfer between wh-questions and topicalization, which greatly vary in terms of their input frequency. Our results suggest shared, yet item-sensitive mechanisms may develop with limited training data. More importantly, LMs still require far more data than humans to learn comparable generalizations, highlighting the need for language-specific biases in models of language acquisition.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 24 of its 70 references that are in the map. First posted on arXiv 15 Apr 2026.

Builds on

24 of 70 references.

References · 24

  1. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,411
  2. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,786
  3. 2022Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 smallcited by 1,165
  4. 2019BLiMP: The Benchmark of Linguistic Minimal Pairs for Englishcited by 564
  5. 2018Targeted Syntactic Evaluation of Language Modelscited by 242
  6. 2023Finding Alignments Between Interpretable Causal Variables and Distributed Neural Representationscited by 209
  7. 2019Neural Language Models as Psycholinguistic Subjects: Representations of Syntactic Statecited by 138
  8. 2020Syntactic Structure from Deep Learningcited by 99
  9. 2025Findings of the BabyLM Challenge: Sample-Efficient Pretraining on Developmentally Plausible Corporacited by 88
  10. 2018What do RNN Language Models Learn about Filler-Gap Dependencies?cited by 86
  11. 2022Probing for the Usage of Grammatical Numbercited by 59
  12. 2024pyvene: A Library for Understanding and Improving PyTorch Models via Interventionscited by 54
12 more
  1. 2025How Linguistics Learned to Stop Worrying and Love the Language Modelscited by 51
  2. 2024CausalGym: Benchmarking causal interpretability methods on linguistic taskscited by 46
  3. 2019Using Priming to Uncover the Organization of Syntactic Representations in Neural Language Modelscited by 29
  4. 2025Causal Interventions Reveal Shared Structure Across English Filler-Gap Constructionscited by 21
  5. 2023Pushdown Layers: Encoding Recursive Structure in Transformer Language Modelscited by 19
  6. 2023Words, Subwords, and Morphemes: What Really Matters in the Surprisal-Reading Time Relationship?cited by 13
  7. 2023Verb Conjugation in Transformers Is Determined by Linear Encodings of Subject Numbercited by 12
  8. 2025Different types of syntactic agreement recruit the same units within large language modelscited by 8
  9. 2024The Impact of Token Granularity on the Predictive Power of Language Model Surprisalcited by 5
  10. 2022How Adults Understand What Young Children Saycited by 2
  11. 2024Reframing linguistic bootstrapping as joint inference using visually-grounded grammar induction modelscited by 2
  12. 2026What Exactly do Children Receive in Language Acquisition? A Case Study on CHILDES with Automated Detection of Filler-Gap Dependenciescited by 1

Built on by

Not cited yet.