mlsanityData to 14 Oct 2026

Sparse Interventions in Language Models with Differentiable Masking

arXiv 2112.06837 ↗First posted 13 Dec 2021cs.CL cs.LGcited by 34

Not on the map. Why?

Abstract (arXiv)

There has been a lot of interest in understanding what information is captured by hidden representations of language models (LMs). Typically, interpretation methods i) do not guarantee that the model actually uses the encoded information, and ii) do not discover small subsets of neurons responsible for a considered phenomenon. Inspired by causal mediation analysis, we propose a method that discovers within a neural LM a small subset of neurons responsible for a particular linguistic phenomenon, i.e., subsets causing a change in the corresponding token emission probabilities. We use a differentiable relaxation to approximately search through the combinatorial space. An L0 regularization term ensures that the search converges to discrete and sparse solutions. We apply our method to analyze subject-verb number agreement and gender bias detection in LSTMs. We observe that it is fast and finds better solutions than the alternative (REINFORCE). Our experiments confirm that each of these phenomenons is mediated through a small subset of neurons that do not play any other discernible role.

Key facts. Cited by 34 papers in the map (data through 14 Oct 2026). Builds on 36 of its 51 references that are in the map. First posted on arXiv 13 Dec 2021.

Builds on

36 of 51 references.

References · 36

  1. 2017Axiomatic Attribution for Deep Networkscited by 3,384
  2. 2016Categorical Reparameterization with Gumbel-Softmaxcited by 2,739
  3. 2021Convex optimizationcited by 2,515
  4. 2016Understanding intermediate layers using linear classifier probescited by 1,353
  5. 2016Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddingscited by 1,312
  6. 2016The Concrete Distribution: A Continuous Relaxation of Discrete Random Variablescited by 1,148
  7. 2019Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Prunedcited by 856
  8. 2021Knowledge Neurons in Pretrained Transformerscited by 601
  9. 2020Towards Faithfully Interpretable NLP Systems: How should we define and evaluate faithfulness?cited by 599
  10. 2017Learning Sparse Neural Networks through L0 Regularizationcited by 583
  11. 2016Assessing the Ability of LSTMs to Learn Syntax-Sensitive Dependenciescited by 354
  12. 2013Direct and Indirect Effectscited by 280
24 more
  1. 2018Analysis Methods in Neural Language Processing: A Surveycited by 277
  2. 2020Inductive Biases for Deep Learning of Higher-Level Cognitioncited by 275
  3. 2020Information-Theoretic Probing with Minimum Description Lengthcited by 232
  4. 2016Fine-grained Analysis of Sentence Embeddings Using Auxiliary Prediction Taskscited by 216
  5. 2018Colorless green recurrent networks dream hierarchicallycited by 213
  6. 2018Gender Bias in Neural Natural Language Processingcited by 211
  7. 2020Interpreting Graph Neural Networks for NLP With Differentiable Edge Maskingcited by 153
  8. 2017Learning to Generate Reviews and Discovering Sentimentcited by 148
  9. 2021Causal Analysis of Syntactic Agreement Mechanisms in Neural Language Modelscited by 131
  10. 2019Interpretable Neural Predictions with Differentiable Binary Variablescited by 126
  11. 2020CausaLM: Causal Model Explanation Through Counterfactual Language Modelscited by 117
  12. 2018Identifying and Controlling Important Neurons in Neural Machine Translationcited by 111
  13. 2020Are Neural Nets Modular? Inspecting Functional Modularity Through Differentiable Weight Maskscited by 98
  14. 2018Under the Hood: Using Diagnostic Classifiers to Investigate and Improve how Language Models Track Agreement Informationcited by 97
  15. 2019The emergence of number and syntax units in LSTM language modelscited by 94
  16. 2018What do RNN Language Models Learn about Filler-Gap Dependencies?cited by 86
  17. 2021Neuron-level Interpretation of Deep NLP Models: A Surveycited by 81
  18. 2020How do Decisions Emerge across Layers in Neural Models? Interpretation with Differentiable Maskingcited by 63
  19. 2020On the Transfer of Disentangled Representations in Realistic Settingscited by 63
  20. 2021On the Pitfalls of Analyzing Individual Neurons in Language Modelscited by 62
  21. 2020Mechanisms for Handling Nested Dependencies in Neural-Network Language Models and Humanscited by 42
  22. 2021Language Models Use Monotonicity to Assess NPI Licensingcited by 31
  23. 2020Investigating Transferability in Pretrained Language Modelscited by 28
  24. 2019Analysing Neural Language Models: Contextual Decomposition Reveals Default Reasoning in Number and Gender Assignmentcited by 20

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 6; 2023: 8; 2024: 12; 2025: 4; 2026: 4.

34 papers.

Sort:

By area:

  1. 2023Finding Neurons in a Haystack: Case Studies with Sparse Probingcited by 334
  2. 2023Does Localization Inform Editing? Surprising Differences in Causality-Based Localization vs. Knowledge Editing in Language Modelscited by 243
  3. 2023Finding Alignments Between Interpretable Causal Variables and Distributed Neural Representationscited by 209
  4. 2023Causal Abstraction: A Theoretical Foundation for Mechanistic Interpretabilitycited by 205
  5. 2022Towards Faithful Model Explanation in NLP: A Surveycited by 130
  6. 2024AtP*: An efficient and scalable method for localizing LLM behaviour to componentscited by 111
  7. 2024RAVEL: Evaluating Interpretability Methods on Disentangling Language Model Representationscited by 79
  8. 2025MIB: A Mechanistic Interpretability Benchmarkcited by 56
  9. 2023Break It Down: Evidence for Structural Compositionality in Neural Networkscited by 44
  10. 2024Activation Scaling for Steering and Interpreting Language Modelscited by 35
  11. 2024Evaluating Open-Source Sparse Autoencoders on Disentangling Factual Knowledge in GPT-2 Smallcited by 34
  12. 2022Causal Proxy Models for Concept-Based Model Explanationscited by 29
22 more
  1. 2024Mechanistic?cited by 25
  2. 2022Probing for Incremental Parse States in Autoregressive Language Modelscited by 24
  3. 2024On the Similarity of Circuits across Languages: a Case Study on the Subject-verb Agreement Taskcited by 24
  4. 2025How Do Transformers Learn Variable Binding in Symbolic Programs?cited by 21
  5. 2024Decomposing and Editing Predictions by Modeling Model Computationcited by 19
  6. 2024A Reply to Makelov et al. (2023)'s "Interpretability Illusion" Argumentscited by 17
  7. 2022The Birth of Bias: A case study on the evolution of gender bias in an English language modelcited by 16
  8. 2022Causal Analysis of Syntactic Agreement Neurons in Multilingual Language Modelscited by 14
  9. 2024Sheaf Discovery with Joint Computation Graph Pruning and Flexible Granularitycited by 14
  10. 2024Towards Compositional Interpretability for XAIcited by 12
  11. 2026From Directions to Regions: Decomposing Activations in Language Models via Local Geometrycited by 12
  12. 2025Rethinking Circuit Completeness in Language Models: AND, OR, and ADDER Gatescited by 10
  13. 2023NeuroSurgeon: A Toolkit for Subnetwork Analysiscited by 9
  14. 2023Causal interventions expose implicit situation models for commonsense language understandingcited by 7
  15. 2022Undesirable Biases in NLP: Addressing Challenges of Measurementcited by 4
  16. 2024Interpreting Bias in Large Language Models: A Feature-Based Approachcited by 4
  17. 2025From Concepts to Components: Concept-Agnostic Attention Module Discovery in Transformerscited by 4
  18. 2026Farther the Shift, Sparser the Representation: Analyzing OOD Mechanisms in LLMscited by 4
  19. 2024Understanding Language Model Circuits through Knowledge Editingcited by 2
  20. 2026Circuit Claims Depend on What Is Extracted and How It Is Comparedcited by 2
  21. 2023Flexible Model Interpretability through Natural Language Model Editingcited by 0
  22. 2026Activation-Guided Neuron Intervention to Induce Alzheimer's-Related Computational Language Phenotypes in a Large Language Modelcited by 0