mlsanityData to 7 Oct 2026

Under the Hood: Using Diagnostic Classifiers to Investigate and Improve how Language Models Track Agreement Information

arXiv 1808.08079 ↗First posted 24 Aug 2018cs.CL cs.AIcited by 97

Not on the map. Why?

Abstract (arXiv)

How do neural language models keep track of number agreement between subject and verb? We show that `diagnostic classifiers', trained to predict number from the internal states of a language model, provide a detailed understanding of how, when, and where this information is represented. Moreover, they give us insight into when and where number information is corrupted in cases where the language model ends up making agreement errors. To demonstrate the causal role played by the representations we find, we then use agreement information to influence the course of the LSTM during the processing of difficult sentences. Results from such an intervention reveal a large increase in the language model's accuracy. Together, these results show that diagnostic classifiers give us an unrivalled detailed look into the representation of linguistic information in neural models, and demonstrate that this knowledge can be used to improve their performance.

Key facts. Cited by 97 papers in the map (data through 7 Oct 2026). First posted on arXiv 24 Aug 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 25; 2022: 16; 2023: 17; 2024: 21; 2025: 8; 2026: 10.

97 papers.

Sort:

By area:

  1. 2021Probing Classifiers: Promises, Shortcomings, and Advancescited by 831
  2. 2021Causal Abstractions of Neural Networkscited by 401
  3. 2023Emergent Linear Representations in World Models of Self-Supervised Sequence Modelscited by 313
  4. 2023How does GPT-2 compute greater-than?: Interpreting mathematical abilities in a pre-trained language modelcited by 199
  5. 2025AxBench: Steering LLMs? Even Simple Baselines Outperform Sparse Autoencoderscited by 199
  6. 2023Causal Abstraction: A Theoretical Foundation for Mechanistic Interpretabilitycited by 194
  7. 2021Masked Language Modeling and the Distributional Hypothesis: Order Word Matters Pre-training for Littlecited by 176
  8. 2021Causal Analysis of Syntactic Agreement Mechanisms in Neural Language Modelscited by 127
  9. 2021Inducing Causal Structure for Interpretable Neural Networkscited by 107
  10. 2022State-of-the-art generalisation research in NLP: A taxonomy and reviewcited by 97
  11. 2021Why Do Pretrained Language Models Help in Downstream Tasks? An Analysis of Head and Prompt Tuningcited by 82
  12. 2021Language Model Evaluation Beyond Perplexitycited by 77
38 more
  1. 2021Contrastive Explanations for Model Interpretabilitycited by 75
  2. 2025A Survey on Sparse Autoencoders: Interpreting the Internal Mechanisms of Large Language Modelscited by 73
  3. 2023Towards Vision-Language Mechanistic Interpretability: A Causal Tracing Tool for BLIPcited by 62
  4. 2021On the Pitfalls of Analyzing Individual Neurons in Language Modelscited by 61
  5. 2021Counterfactual Interventions Reveal the Causal Effect of Relative Clause Representations on Agreement Predictioncited by 58
  6. 2022Probing for the Usage of Grammatical Numbercited by 57
  7. 2022Measuring Causal Effects of Data Statistics on Language Model's `Factual' Predictionscited by 57
  8. 2025Large Language Models Share Representations of Latent Grammatical Concepts Across Typologically Diverse Languagescited by 57
  9. 2023Codebook Features: Sparse and Discrete Interpretability for Neural Networkscited by 45
  10. 2024Recurrent Neural Networks Learn to Store and Generate Sequences using Non-Linear Representationscited by 45
  11. 2021Sparse Interventions in Language Models with Differentiable Maskingcited by 34
  12. 2021Local Interpretations for Explainable Natural Language Processing: A Surveycited by 31
  13. 2021Language Models Use Monotonicity to Assess NPI Licensingcited by 30
  14. 2024What Do VLMs NOTICE? A Mechanistic Interpretability Pipeline for Gaussian-Noise-free Text-Image Corruption and Evaluationcited by 29
  15. 2023Dissociating language and thought in large language modelscited by 27
  16. 2024Holmes: A Benchmark to Assess the Linguistic Competence of Language Modelscited by 26
  17. 2024Mechanistic?cited by 25
  18. 2024On Behalf of the Stakeholders: Trends in NLP Model Interpretability in the Era of LLMscited by 24
  19. 2024The Quest for the Right Mediator: Surveying Mechanistic Interpretability Through the Lens of Causal Mediation Analysiscited by 24
  20. 2025Mechanistic Interpretability of Emotion Inference in Large Language Modelscited by 24
  21. 2023Subspace Chronicles: How Linguistic Information Emerges, Shifts and Interacts during Language Model Trainingcited by 23
  22. 2023An Overview on Language Models: Recent Developments and Outlookcited by 22
  23. 2026Manifold Steering Reveals the Shared Geometry of Neural Network Representation and Behaviorcited by 22
  24. 2024Missed Causes and Ambiguous Effects: Counterfactuals Pose Challenges for Interpreting Neural Networkscited by 21
  25. 2025HyperSteer: Activation Steering at Scale with Hypernetworkscited by 20
  26. 2022Naturalistic Causal Probing for Morpho-Syntaxcited by 19
  27. 2026Language Model Circuits Are Sparse in the Neuron Basiscited by 18
  28. 2021Probing artificial neural networks: insights from neurosciencecited by 17
  29. 2024A Philosophical Introduction to Language Models - Part II: The Way Forwardcited by 17
  30. 2021The Language Model Understood the Prompt was Ambiguous: Probing Syntactic Uncertainty Through Generationcited by 15
  31. 2024Layer by Layer: Uncovering Where Multi-Task Learning Happens in Instruction-Tuned Large Language Modelscited by 15
  32. 2023Operationalising Representation in Natural Language Processingcited by 14
  33. 2024Does ChatGPT Have a Mind?cited by 13
  34. 2024Gumbel Counterfactual Generation From Language Modelscited by 13
  35. 2022When Does Syntax Mediate Neural Language Model Performance? Evidence from Dropout Probescited by 12
  36. 2023Investigating semantic subspaces of Transformer sentence embeddings through linear structural probingcited by 12
  37. 2023Verb Conjugation in Transformers Is Determined by Linear Encodings of Subject Numbercited by 12
  38. 2024What Should Embeddings Embed? Autoregressive Models Represent Latent Generating Distributionscited by 10

50 most cited of 97 All 97 (JSON)