mlsanityData to 14 Oct 2026

Verb Conjugation in Transformers Is Determined by Linear Encodings of Subject Number

arXiv 2310.15151 ↗First posted 23 Oct 2023cs.CL cs.AI cs.LGcited by 12

Not on the map. Why?

Abstract (arXiv)

Deep architectures such as Transformers are sometimes criticized for having uninterpretable "black-box" representations. We use causal intervention analysis to show that, in fact, some linguistic features are represented in a linear, interpretable format. Specifically, we show that BERT's ability to conjugate verbs relies on a linear encoding of subject number that can be manipulated with predictable effects on conjugation accuracy. This encoding is found in the subject position at the first layer and the verb position at the last layer, but distributed across positions at middle layers, particularly when there are multiple cues to subject number.

Key facts. Cited by 12 papers in the map (data through 14 Oct 2026). Builds on 21 of its 30 references that are in the map. First posted on arXiv 23 Oct 2023.

Builds on

21 of 30 references.

References · 21

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2017Interpretability Beyond Feature Attribution: Quantitative Testing with Concept Activation Vectors (TCAV)cited by 1,319
  3. 2020A Primer in BERTology: What we know about how BERT workscited by 815
  4. 2020Null It Out: Guarding Protected Attributes by Iterative Nullspace Projectioncited by 455
  5. 2019What do you learn from context? Probing for sentence structure in contextualized word representationscited by 438
  6. 2019Linguistic Knowledge and Transferability of Contextual Representationscited by 383
  7. 2016Assessing the Ability of LSTMs to Learn Syntax-Sensitive Dependenciescited by 354
  8. 2019Revealing the Dark Secrets of BERTcited by 301
  9. 2018Targeted Syntactic Evaluation of Language Modelscited by 242
  10. 2016Fine-grained Analysis of Sentence Embeddings Using Auxiliary Prediction Taskscited by 216
  11. 2019Assessing BERT's Syntactic Abilitiescited by 192
  12. 2017What do Neural Machine Translation Models Learn about Morphology?cited by 166
9 more
  1. 2021Causal Analysis of Syntactic Agreement Mechanisms in Neural Language Modelscited by 131
  2. 2018What Is One Grain of Sand in the Desert? Analyzing Individual Neurons in Deep NLP Modelscited by 124
  3. 2018Under the Hood: Using Diagnostic Classifiers to Investigate and Improve how Language Models Track Agreement Informationcited by 97
  4. 2019The emergence of number and syntax units in LSTM language modelscited by 94
  5. 2022Probing for the Usage of Grammatical Numbercited by 59
  6. 2019Investigating BERT's Knowledge of Language: Five Analysis Methods with NPIscited by 58
  7. 2021What if This Modified That? Syntactic Interventions via Counterfactual Embeddingscited by 39
  8. 2019Analytical Methods for Interpretable Ultradense Word Embeddingscited by 13
  9. 2022When Does Syntax Mediate Neural Language Model Performance? Evidence from Dropout Probescited by 12

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 4; 2025: 2; 2026: 6.

12 papers.

  1. 2024ReFT: Representation Finetuning for Language Modelscited by 197
  2. 2024CausalGym: Benchmarking causal interpretability methods on linguistic taskscited by 46
  3. 2025Bayesian Teaching Enables Probabilistic Reasoning in Large Language Modelscited by 37
  4. 2024Language Models as Models of Languagecited by 24
  5. 2025LinguaLens: Towards Interpreting Linguistic Mechanisms of Large Language Models via Sparse Auto-Encodercited by 14
  6. 2024Philosophy of Cognitive Science in the Age of Deep Learningcited by 5
  7. 2026Humans and transformer LMs: Abstraction drives language learningcited by 2
  8. 2026The Grammar of Transformers: A Systematic Review of Interpretability Research on Syntactic Knowledge in Language Modelscited by 1
  9. 2026Refusal Beyond a Single Direction: A Preliminary Comparison of Diff-in-Means and INLPcited by 1
  10. 2026Filling in the Mechanisms: How do LMs Learn Filler-Gap Dependencies under Developmental Constraints?cited by 0
  11. 2026Causal Interventions on Continuous Variables: A Case Study on Verb Bias in Steering Vectors for In-Context Learningcited by 0
  12. 2026Cognitive Convergence: Deep Similarities Between Large Language Models and Human Cognitioncited by 0