mlsanityData to 7 Oct 2026

Causal Abstractions of Neural Networks

arXiv 2106.02997 ↗First posted 6 Jun 2021cs.AI cs.LGcited by 401

Not on the map. Why?

Cited by 22 in LLM reasoning, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Structural analysis methods (e.g., probing and feature attribution) are increasingly important tools for neural network analysis. We propose a new structural analysis method grounded in a formal theory of causal abstraction that provides rich characterizations of model-internal representations and their roles in input/output behavior. In this method, neural representations are aligned with variables in interpretable causal models, and then interchange interventions are used to experimentally verify that the neural representations have the causal properties of their aligned variables. We apply this method in a case study to analyze neural models trained on Multiply Quantified Natural Language Inference (MQNLI) corpus, a highly complex NLI dataset that was constructed with a tree-structured natural logic causal model. We discover that a BERT-based model with state-of-the-art performance successfully realizes parts of the natural logic model's causal structure, whereas a simpler baseline model fails to show any such structure, demonstrating that BERT representations encode the compositional structure of MQNLI.

Key facts. Cited by 401 papers in the map (data through 7 Oct 2026). Builds on 30 of its 43 references that are in the map. First posted on arXiv 6 Jun 2021.

Builds on

30 of 43 references.

References · 30

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2017A Unified Approach to Interpreting Model Predictionscited by 6,488
  4. 2016"Why Should I Trust You?": Explaining the Predictions of Any Classifiercited by 5,040
  5. 2017Axiomatic Attribution for Deep Networkscited by 3,350
  6. 2013Visualizing and Understanding Convolutional Networkscited by 2,278
  7. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,201
  8. 2019BERT Rediscovers the Classical NLP Pipelinecited by 1,046
  9. 2019What Does BERT Look At? An Analysis of BERT's Attentioncited by 994
  10. 2014Striving for Simplicity: The All Convolutional Netcited by 983
  11. 2019Designing and Interpreting Probes with Control Taskscited by 608
  12. 2017The (Un)reliability of saliency methodscited by 331
18 more
  1. 2016Not Just a Black Box: Learning Important Features Through Propagating Activation Differencescited by 295
  2. 2013Direct and Indirect Effectscited by 276
  3. 2016Layer-wise Relevance Propagation for Neural Networks with Local Renormalization Layerscited by 197
  4. 2020Causal Mediation Analysis for Interpreting Neural NLP: The Case of Gender Biascited by 183
  5. 2018Dissecting Contextual Word Embeddings: Architecture and Representationcited by 146
  6. 2020Probing the Probing Paradigm: Does Probing Accuracy Entail Task Relevance?cited by 144
  7. 2016Foundations of Structural Causal Models with Cycles and Latent Variablescited by 134
  8. 2020CausaLM: Causal Model Explanation Through Counterfactual Language Modelscited by 117
  9. 2017Causal Consistency of Structural Equation Modelscited by 114
  10. 2018Abstracting Causal Modelscited by 104
  11. 2018Under the Hood: Using Diagnostic Classifiers to Investigate and Improve how Language Models Track Agreement Informationcited by 97
  12. 2019Neural Network Attributions: A Causal Perspectivecited by 77
  13. 2019Approximate Causal Abstractioncited by 62
  14. 2019Discovering the Compositional Structure of Vector Representations with Role Learning Networkscited by 33
  15. 2015Multi-Level Cause-Effect Systemscited by 30
  16. 2019Posing Fair Generalization Tasks for Natural Language Inferencecited by 22
  17. 2019What does it mean to understand a neural network?cited by 4
  18. 2018Analysing the potential of seq-to-seq models for incremental interpretation in task-oriented dialoguecited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 5; 2022: 15; 2023: 56; 2024: 89; 2025: 86; 2026: 150.

401 papers.

Sort:

By area:

  1. 2023Sparse Autoencoders Find Highly Interpretable Features in Language Modelscited by 1,359
  2. 2022Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 smallcited by 1,130
  3. 2023Towards Automated Circuit Discovery for Mechanistic Interpretabilitycited by 700
  4. 2023The Geometry of Truth: Emergent Linear Structure in Large Language Model Representations of True/False Datasetscited by 673
  5. 2024Mechanistic Interpretability for AI Safety -- A Reviewcited by 415
  6. 2024Sparse Feature Circuits: Discovering and Editing Interpretable Causal Graphs in Language Modelscited by 412
  7. 2023Towards Best Practices of Activation Patching in Language Models: Metrics and Methodscited by 328
  8. 2023Reasoning or Reciting? Exploring the Capabilities and Limitations of Language Models Through Counterfactual Taskscited by 243
  9. 2024Patchscopes: A Unifying Framework for Inspecting Hidden Representations of Language Modelscited by 225
  10. 2023How does GPT-2 compute greater-than?: Interpreting mathematical abilities in a pre-trained language modelcited by 199
  11. 2025AxBench: Steering LLMs? Even Simple Baselines Outperform Sparse Autoencoderscited by 199
  12. 2023Finding Alignments Between Interpretable Causal Variables and Distributed Neural Representationscited by 197
38 more
  1. 2023Attribution Patching Outperforms Automated Circuit Discoverycited by 196
  2. 2023Causal Abstraction: A Theoretical Foundation for Mechanistic Interpretabilitycited by 194
  3. 2024How to use and interpret activation patchingcited by 191
  4. 2024ReFT: Representation Finetuning for Language Modelscited by 186
  5. 2024Transcoders Find Interpretable LLM Feature Circuitscited by 186
  6. 2023Localizing Model Behavior with Path Patchingcited by 182
  7. 2021Causal Inference in Natural Language Processing: Estimation, Prediction, Interpretation and Beyondcited by 177
  8. 2023Does Circuit Analysis Interpretability Scale? Evidence from Multiple Choice Capabilities in Chinchillacited by 145
  9. 2023The Hydra Effect: Emergent Self-repair in Language Model Computationscited by 145
  10. 2024Have Faith in Faithfulness: Going Beyond Circuit Overlap When Finding Model Mechanismscited by 144
  11. 2022Language Models as Agent Modelscited by 131
  12. 2022Towards Faithful Model Explanation in NLP: A Surveycited by 129
  13. 2022Toward Transparent AI: A Survey on Interpreting the Inner Structures of Deep Neural Networkscited by 126
  14. 2025Thought Anchors: Which LLM Reasoning Steps Matter?cited by 126
  15. 2024A Practical Review of Mechanistic Interpretability for Transformer-Based Language Modelscited by 113
  16. 2021Inducing Causal Structure for Interpretable Neural Networkscited by 107
  17. 2023Interpretability at Scale: Identifying Causal Mechanisms in Alpacacited by 102
  18. 2023How do Language Models Bind Entities in Context?cited by 102
  19. 2024AtP*: An efficient and scalable method for localizing LLM behaviour to componentscited by 100
  20. 2024Arithmetic Without Algorithms: Language Models Solve Math With a Bag of Heuristicscited by 94
  21. 2023A Mechanistic Interpretation of Arithmetic Reasoning in Language Models using Causal Mediation Analysiscited by 88
  22. 2024A Primer on the Inner Workings of Transformer-based Language Modelscited by 83
  23. 2023Overthinking the Truth: Understanding how Language Models Process False Demonstrationscited by 78
  24. 2024RAVEL: Evaluating Interpretability Methods on Disentangling Language Model Representationscited by 77
  25. 2023Is This the Subspace You Are Looking for? An Interpretability Illusion for Subspace Activation Patchingcited by 75
  26. 2023How Do Transformers Learn In-Context Beyond Simple Functions? A Case Study on Learning with Representationscited by 72
  27. 2024On the Role of Attention Heads in Large Language Model Safetycited by 70
  28. 2024Attention Heads of Large Language Models: A Surveycited by 65
  29. 2023Towards Vision-Language Mechanistic Interpretability: A Causal Tracing Tool for BLIPcited by 62
  30. 2024LLM Circuit Analyses Are Consistent Across Training and Scalecited by 60
  31. 2024Demystifying Verbatim Memorization in Large Language Modelscited by 55
  32. 2022CEBaB: Estimating the Causal Effects of Real-World Concepts on NLP Model Behaviorcited by 54
  33. 2024pyvene: A Library for Understanding and Improving PyTorch Models via Interventionscited by 53
  34. 2022Leveraging Explanations in Interactive Machine Learning: An Overviewcited by 52
  35. 2025MIB: A Mechanistic Interpretability Benchmarkcited by 49
  36. 2023Codebook Features: Sparse and Discrete Interpretability for Neural Networkscited by 45
  37. 2024CausalGym: Benchmarking causal interpretability methods on linguistic taskscited by 45
  38. 2024Usable XAI: 10 Strategies Towards Exploiting Explainability in the LLM Eracited by 45

50 most cited of 401 All 401 (JSON)