mlsanityData to 14 Oct 2026

Causal Interventions Reveal Shared Structure Across English Filler-Gap Constructions

arXiv 2505.16002 ↗First posted 21 May 2025cs.CL cs.AIcited by 21

Not on the map. Why?

Abstract (arXiv)

Language Models (LMs) have emerged as powerful sources of evidence for linguists seeking to develop theories of syntax. In this paper, we argue that causal interpretability methods, applied to LMs, can greatly enhance the value of such evidence by helping us characterize the abstract mechanisms that LMs learn to use. Our empirical focus is a set of English filler-gap dependency constructions (e.g., questions, relative clauses). Linguistic theories largely agree that these constructions share many properties. Using experiments based in Distributed Interchange Interventions, we show that LMs converge on similar abstract analyses of these constructions. These analyses also reveal previously overlooked factors -- relating to frequency, filler type, and surrounding context -- that could motivate changes to standard linguistic theory. Overall, these results suggest that mechanistic, internal analyses of LMs can push linguistic theory forward.

Key facts. Cited by 21 papers in the map (data through 14 Oct 2026). Builds on 18 of its 48 references that are in the map. First posted on arXiv 21 May 2025.

Builds on

18 of 48 references.

References · 18

  1. 2022Locating and Editing Factual Associations in GPTcited by 2,691
  2. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,786
  3. 2022Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 smallcited by 1,165
  4. 2021Causal Abstractions of Neural Networkscited by 422
  5. 2020Universal Dependencies v2: An Evergrowing Multilingual Treebank Collectioncited by 271
  6. 2023Finding Alignments Between Interpretable Causal Variables and Distributed Neural Representationscited by 209
  7. 2023Causal Abstraction: A Theoretical Foundation for Mechanistic Interpretabilitycited by 205
  8. 2020Causal Mediation Analysis for Interpreting Neural NLP: The Case of Gender Biascited by 191
  9. 2020A Systematic Assessment of Syntactic Generalization in Neural Language Modelscited by 165
  10. 2021Causal Analysis of Syntactic Agreement Mechanisms in Neural Language Modelscited by 131
  11. 2019The emergence of number and syntax units in LSTM language modelscited by 94
  12. 2018What do RNN Language Models Learn about Filler-Gap Dependencies?cited by 86
6 more
  1. 2022Probing for the Usage of Grammatical Numbercited by 59
  2. 2024pyvene: A Library for Understanding and Improving PyTorch Models via Interventionscited by 54
  3. 2025How Linguistics Learned to Stop Worrying and Love the Language Modelscited by 51
  4. 2024CausalGym: Benchmarking causal interpretability methods on linguistic taskscited by 46
  5. 2019Using Priming to Uncover the Organization of Syntactic Representations in Neural Language Modelscited by 29
  6. 2024Characterizing the Role of Similarity in the Property Inferences of Language Modelscited by 11

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 6; 2026: 15.

21 papers.

Sort:

By area:

  1. 2025How Causal Abstraction Underpins Computational Explanationcited by 11
  2. 2025Different types of syntactic agreement recruit the same units within large language modelscited by 8
  3. 2026What Exactly do Children Receive in Language Acquisition? A Case Study on CHILDES with Automated Detection of Filler-Gap Dependenciescited by 1
  4. 2026Perturbation: A simple and efficient adversarial tracer for representation learning in language modelscited by 1
  5. 2026Bucketing the Good Apples: A Method for Diagnosing and Improving Causal Abstractioncited by 1
  6. 2025Language models as tools for investigating the distinction between possible and impossible natural languagescited by 0
  7. 2025Large language models have learned to use languagecited by 0
  8. 2025Large language models are not about natural languagecited by 0
  9. 2025Linguists should learn to love speech-based deep learning modelscited by 0
  10. 2026Are Language Models Models?cited by 0
  11. 2026Across the Levels of Analysis: Explaining Predictive Processing in Humans Requires More Than Machine-Estimated Probabilitiescited by 0
  12. 2026You Can't Fight in Here! This is BBS!cited by 0
9 more
  1. 2026Causal Drawbridges: Characterizing Gradient Blocking of Syntactic Islands in Transformer LMscited by 0
  2. 2026Filling in the Mechanisms: How do LMs Learn Filler-Gap Dependencies under Developmental Constraints?cited by 0
  3. 2026Fine-Grained Analysis of Shared Syntactic Mechanisms in Language Modelscited by 0
  4. 2026Not-So-Strange Love: Language Models and Generative Linguistic Theories are More Compatible than They Appearcited by 0
  5. 2026Causal Interventions on Continuous Variables: A Case Study on Verb Bias in Steering Vectors for In-Context Learningcited by 0
  6. 2026A Mechanistic Understanding of Pronoun Fidelity in LLMscited by 0
  7. 2026When Context Misleads: Surprisal, Energy and Attention Entropy as Metrics of Coherence Illusions in LLMscited by 0
  8. 2026Cognitive Convergence: Deep Similarities Between Large Language Models and Human Cognitioncited by 0
  9. 2026Causal Interventions Reveal Typologically Organized Syntactic Mechanisms in Multilingual Language Modelscited by 0