mlsanityData to 7 Oct 2026

Causal Analysis of Syntactic Agreement Mechanisms in Neural Language Models

arXiv 2106.06087 ↗First posted 10 Jun 2021cs.CLcited by 127

Not on the map. Why?

Abstract (arXiv)

Targeted syntactic evaluations have demonstrated the ability of language models to perform subject-verb agreement given difficult contexts. To elucidate the mechanisms by which the models accomplish this behavior, this study applies causal mediation analysis to pre-trained neural language models. We investigate the magnitude of models' preferences for grammatical inflections, as well as whether neurons process subject-verb agreement similarly across sentences with different syntactic structures. We uncover similarities and differences across architectures and model sizes -- notably, that larger models do not necessarily learn stronger preferences. We also observe two distinct mechanisms for producing subject-verb agreement depending on the syntactic structure of the input sentence. Finally, we find that language models rely on similar sets of neurons when given sentences with similar syntactic structure.

Key facts. Cited by 127 papers in the map (data through 7 Oct 2026). Builds on 32 of its 44 references that are in the map. First posted on arXiv 10 Jun 2021.

Builds on

32 of 44 references.

References · 32

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  4. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  5. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,801
  6. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,830
  7. 2019BERT Rediscovers the Classical NLP Pipelinecited by 1,046
  8. 2019Designing and Interpreting Probes with Control Taskscited by 608
  9. 2019What do you learn from context? Probing for sentence structure in contextualized word representationscited by 436
  10. 2019Linguistic Knowledge and Transferability of Contextual Representationscited by 382
  11. 2016Assessing the Ability of LSTMs to Learn Syntax-Sensitive Dependenciescited by 350
  12. 2013Direct and Indirect Effectscited by 276
20 more
  1. 2018Analysis Methods in Neural Language Processing: A Surveycited by 276
  2. 2018Targeted Syntactic Evaluation of Language Modelscited by 239
  3. 2018Colorless green recurrent networks dream hierarchicallycited by 210
  4. 2019Assessing BERT's Syntactic Abilitiescited by 191
  5. 2020Causal Mediation Analysis for Interpreting Neural NLP: The Case of Gender Biascited by 183
  6. 2021Masked Language Modeling and the Distributional Hypothesis: Order Word Matters Pre-training for Littlecited by 176
  7. 2020A Systematic Assessment of Syntactic Generalization in Neural Language Modelscited by 163
  8. 2019Neural Language Models as Psycholinguistic Subjects: Representations of Syntactic Statecited by 137
  9. 2020Finding Universal Grammatical Relations in Multilingual BERTcited by 108
  10. 2018Under the Hood: Using Diagnostic Classifiers to Investigate and Improve how Language Models Track Agreement Informationcited by 97
  11. 2019The emergence of number and syntax units in LSTM language modelscited by 92
  12. 2019Do Attention Heads in BERT Track Syntactic Dependencies?cited by 87
  13. 2018What do RNN Language Models Learn about Filler-Gap Dependencies?cited by 86
  14. 2020Cross-Linguistic Syntactic Evaluation of Word Prediction Modelscited by 54
  15. 2020Mechanisms for Handling Nested Dependencies in Neural-Network Language Models and Humanscited by 42
  16. 2021Refining Targeted Syntactic Evaluation of Language Modelscited by 41
  17. 2019Quantity doesn't buy quality syntax with neural language modelscited by 25
  18. 2019Structural Supervision Improves Learning of Non-Local Grammatical Dependenciescited by 24
  19. 2019Analysing Neural Language Models: Contextual Decomposition Reveals Default Reasoning in Number and Gender Assignmentcited by 20
  20. 2020Influence Paths for Characterizing Subject-Verb Number Agreement in LSTM Language Modelscited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 4; 2022: 20; 2023: 21; 2024: 31; 2025: 33; 2026: 18.

127 papers.

Sort:

By area:

  1. 2022Locating and Editing Factual Associations in GPTcited by 2,609
  2. 2022Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 smallcited by 1,130
  3. 2023The Geometry of Truth: Emergent Linear Structure in Large Language Model Representations of True/False Datasetscited by 673
  4. 2024Sparse Feature Circuits: Discovering and Editing Interpretable Causal Graphs in Language Modelscited by 412
  5. 2023Towards Best Practices of Activation Patching in Language Models: Metrics and Methodscited by 328
  6. 2023Finding Alignments Between Interpretable Causal Variables and Distributed Neural Representationscited by 197
  7. 2023Causal Abstraction: A Theoretical Foundation for Mechanistic Interpretabilitycited by 194
  8. 2024How to use and interpret activation patchingcited by 191
  9. 2023Localizing Model Behavior with Path Patchingcited by 182
  10. 2021Causal Inference in Natural Language Processing: Estimation, Prediction, Interpretation and Beyondcited by 177
  11. 2023The Hydra Effect: Emergent Self-repair in Language Model Computationscited by 145
  12. 2022Towards Faithful Model Explanation in NLP: A Surveycited by 129
38 more
  1. 2022Toward Transparent AI: A Survey on Interpreting the Inner Structures of Deep Neural Networkscited by 126
  2. 2024Making Reasoning Matter: Measuring and Improving Faithfulness of Chain-of-Thought Reasoningcited by 108
  3. 2024AtP*: An efficient and scalable method for localizing LLM behaviour to componentscited by 100
  4. 2022What Artificial Neural Networks Can Tell Us About Human Language Acquisitioncited by 97
  5. 2023Language Model Behavior: A Comprehensive Surveycited by 92
  6. 2022Interpreting Language Models with Contrastive Explanationscited by 91
  7. 2023A Mechanistic Interpretation of Arithmetic Reasoning in Language Models using Causal Mediation Analysiscited by 88
  8. 2022Learning and Evaluating Graph Neural Network Explanations based on Counterfactual and Factual Reasoningcited by 78
  9. 2024RAVEL: Evaluating Interpretability Methods on Disentangling Language Model Representationscited by 77
  10. 2022A Causal Framework to Quantify the Robustness of Mathematical Reasoning with Language Modelscited by 68
  11. 2024Identifying Functionally Important Features with End-to-End Sparse Dictionary Learningcited by 64
  12. 2024LLM Circuit Analyses Are Consistent Across Training and Scalecited by 60
  13. 2022Probing for the Usage of Grammatical Numbercited by 57
  14. 2022Measuring Causal Effects of Data Statistics on Language Model's `Factual' Predictionscited by 57
  15. 2022CEBaB: Estimating the Causal Effects of Real-World Concepts on NLP Model Behaviorcited by 54
  16. 2024What does the Knowledge Neuron Thesis Have to do with Knowledge?cited by 49
  17. 2025MIB: A Mechanistic Interpretability Benchmarkcited by 49
  18. 2024Recurrent Neural Networks Learn to Store and Generate Sequences using Non-Linear Representationscited by 45
  19. 2023Faithful Explanations of Black-box NLP Models Using LLM-generated Counterfactualscited by 44
  20. 2024Interpreting and Improving Large Language Models in Arithmetic Calculationcited by 43
  21. 2024Mechanistic Understanding and Mitigation of Language Model Non-Factual Hallucinationscited by 38
  22. 2025Bayesian Teaching Enables Probabilistic Reasoning in Large Language Modelscited by 36
  23. 2025Language Models use Lookbacks to Track Beliefscited by 36
  24. 2021Sparse Interventions in Language Models with Differentiable Maskingcited by 34
  25. 2024Evaluating Open-Source Sparse Autoencoders on Disentangling Factual Knowledge in GPT-2 Smallcited by 34
  26. 2022Coloring the Blank Slate: Pre-training Imparts a Hierarchical Inductive Bias to Sequence-to-sequence Modelscited by 28
  27. 2023Dissociating language and thought in large language modelscited by 27
  28. 2023Trojan Activation Attack: Red-Teaming Large Language Models using Activation Steering for Safety-Alignmentcited by 27
  29. 2024The Same But Different: Structural Similarities and Differences in Multilingual Language Modelingcited by 27
  30. 2024On Behalf of the Stakeholders: Trends in NLP Model Interpretability in the Era of LLMscited by 24
  31. 2024The Quest for the Right Mediator: Surveying Mechanistic Interpretability Through the Lens of Causal Mediation Analysiscited by 24
  32. 2024On the Similarity of Circuits across Languages: a Case Study on the Subject-verb Agreement Taskcited by 24
  33. 2025Mixing Mechanisms: How Language Models Retrieve Bound Entities In-Contextcited by 24
  34. 2023The Life Cycle of Knowledge in Big Language Models: A Surveycited by 23
  35. 2023A Causal View of Entity Bias in (Large) Language Modelscited by 21
  36. 2024Missed Causes and Ambiguous Effects: Counterfactuals Pose Challenges for Interpreting Neural Networkscited by 21
  37. 2025Causal Interventions Reveal Shared Structure Across English Filler-Gap Constructionscited by 21
  38. 2022Naturalistic Causal Probing for Morpho-Syntaxcited by 19

50 most cited of 127 All 127 (JSON)