mlsanityData to 14 Oct 2026

Causal Drawbridges: Characterizing Gradient Blocking of Syntactic Islands in Transformer LMs

arXiv 2604.13950 ↗First posted 15 Apr 2026cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

We show how causal interventions in Transformer models provide insights into English syntax by focusing on a long-standing challenge for syntactic theory: syntactic islands. Extraction from coordinated verb phrases is often degraded, yet acceptability varies gradiently with lexical content (e.g., "I know what he hates art and loves" vs. "I know what he looked down and saw"). We show that modern Transformer language models replicate human judgments across this gradient. Using causal interventions that isolate functionally relevant subspaces in Transformer blocks, attention modules, and MLPs, we demonstrate that extraction from coordination islands engages the same filler-gap mechanisms as canonical wh-dependencies, but that these mechanisms are selectively blocked to varying degrees. By projecting a large corpus of unrelated text onto these causally identified subspaces, we derive a novel linguistic hypothesis: the conjunction "and" is represented differently in extractable versus non-extractable constructions, corresponding to expressions encoding relational dependencies versus purely conjunctive uses. These results illustrate how mechanistic interpretability can inform syntax, generating new hypotheses about linguistic representation and processing.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 20 of its 50 references that are in the map. First posted on arXiv 15 Apr 2026.

Builds on

20 of 50 references.

References · 20

  1. 2022Locating and Editing Factual Associations in GPTcited by 2,691
  2. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,218
  3. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,786
  4. 2023Sparse Autoencoders Find Highly Interpretable Features in Language Modelscited by 1,420
  5. 2022Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 smallcited by 1,165
  6. 2019BLiMP: The Benchmark of Linguistic Minimal Pairs for Englishcited by 564
  7. 2016Assessing the Ability of LSTMs to Learn Syntax-Sensitive Dependenciescited by 354
  8. 2023Finding Alignments Between Interpretable Causal Variables and Distributed Neural Representationscited by 209
  9. 2024Transcoders Find Interpretable LLM Feature Circuitscited by 200
  10. 2019Neural Language Models as Psycholinguistic Subjects: Representations of Syntactic Statecited by 138
  11. 2018What do RNN Language Models Learn about Filler-Gap Dependencies?cited by 86
  12. 2022minicons: Enabling Flexible Behavioral and Representational Analyses of Transformer Language Modelscited by 58
8 more
  1. 2024pyvene: A Library for Understanding and Improving PyTorch Models via Interventionscited by 54
  2. 2025How Linguistics Learned to Stop Worrying and Love the Language Modelscited by 51
  3. 2024CausalGym: Benchmarking causal interpretability methods on linguistic taskscited by 46
  4. 2019Using Priming to Uncover the Organization of Syntactic Representations in Neural Language Modelscited by 29
  5. 2025Large Language Models Are Human-Like Internallycited by 29
  6. 2025Causal Interventions Reveal Shared Structure Across English Filler-Gap Constructionscited by 21
  7. 2024Bridging the Empirical-Theoretical Gap in Neural Network Formal Language Learning Using Minimum Description Lengthcited by 14
  8. 2024Generalizations across filler-gap dependencies in neural language modelscited by 12

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Fine-Grained Analysis of Shared Syntactic Mechanisms in Language Models
  2. 2021Syntactic Perturbations Reveal Representational Correlates of Hierarchical Phrase Structure in Pretrained Language Models
  3. 2025TRACE: Training and Inference-Time Interpretability Analysis for Language Models
  4. 2025Linguistic Interpretability of Transformer-based Language Models: a systematic review
  5. 2026Causal Interventions Reveal Typologically Organized Syntactic Mechanisms in Multilingual Language Models
  6. 2026Causal and Interpretable Structures in LLM Compositional Tasks
  7. 2025In Machina N400: Pinpointing Where a Causal Language Model Detects Semantic Violations
  8. 2024Analysis and Visualization of Linguistic Structures in Large Language Models: Neural Representations of Verb-Particle Constructions in BERT