mlsanityData to 7 Oct 2026

Inducing Causal Structure for Interpretable Neural Networks

arXiv 2112.00826 ↗First posted 1 Dec 2021cs.LGcited by 107

Not on the map. Why?

Abstract (arXiv)

In many areas, we have well-founded insights about causal structure that would be useful to bring into our trained models while still allowing them to learn in a data-driven fashion. To achieve this, we present the new method of interchange intervention training (IIT). In IIT, we (1) align variables in a causal model (e.g., a deterministic program or Bayesian network) with representations in a neural model and (2) train the neural model to match the counterfactual behavior of the causal model on a base input when aligned representations in both models are set to be the value they would be for a source input. IIT is fully differentiable, flexibly combines with other objectives, and guarantees that the target causal model is a causal abstraction of the neural model when its loss is zero. We evaluate IIT on a structural vision task (MNIST-PVR), a navigational language task (ReaSCAN), and a natural language inference task (MQNLI). We compare IIT against multi-task training objectives and data augmentation. In all our experiments, IIT achieves the best results and produces neural models that are more interpretable in the sense that they more successfully realize the target causal model.

Key facts. Cited by 107 papers in the map (data through 7 Oct 2026). Builds on 39 of its 60 references that are in the map. First posted on arXiv 1 Dec 2021.

Builds on

39 of 60 references.

References · 39

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2017Axiomatic Attribution for Deep Networkscited by 3,350
  4. 2013Visualizing and Understanding Convolutional Networkscited by 2,278
  5. 2019BERT Rediscovers the Classical NLP Pipelinecited by 1,046
  6. 2019What Does BERT Look At? An Analysis of BERT's Attentioncited by 994
  7. 2014Striving for Simplicity: The All Convolutional Netcited by 983
  8. 2017A Survey on Multi-Task Learningcited by 838
  9. 2017An Overview of Multi-Task Learning in Deep Neural Networkscited by 791
  10. 2019Designing and Interpreting Probes with Control Taskscited by 608
  11. 2020Towards Faithfully Interpretable NLP Systems: How should we define and evaluate faithfulness?cited by 590
  12. 2020Null It Out: Guarding Protected Attributes by Iterative Nullspace Projectioncited by 442
27 more
  1. 2019Learning the Difference that Makes a Difference with Counterfactually-Augmented Datacited by 428
  2. 2021Causal Abstractions of Neural Networkscited by 401
  3. 2020Multi-Task Learning with Deep Neural Networks: A Surveycited by 374
  4. 2017The Effectiveness of Data Augmentation in Image Classification using Deep Learningcited by 368
  5. 2016Not Just a Black Box: Learning Important Features Through Propagating Activation Differencescited by 295
  6. 2013Direct and Indirect Effectscited by 276
  7. 2016Layer-wise Relevance Propagation for Neural Networks with Local Renormalization Layerscited by 197
  8. 2018Compositional Attention Networks for Machine Reasoningcited by 183
  9. 2020Causal Mediation Analysis for Interpreting Neural NLP: The Case of Gender Biascited by 183
  10. 2018Dissecting Contextual Word Embeddings: Architecture and Representationcited by 146
  11. 2020Probing the Probing Paradigm: Does Probing Accuracy Entail Task Relevance?cited by 144
  12. 2020CausaLM: Causal Model Explanation Through Counterfactual Language Modelscited by 117
  13. 2017Causal Consistency of Structural Equation Modelscited by 114
  14. 2020A Benchmark for Systematic Generalization in Grounded Language Understandingcited by 108
  15. 2018Abstracting Causal Modelscited by 104
  16. 2018Under the Hood: Using Diagnostic Classifiers to Investigate and Improve how Language Models Track Agreement Informationcited by 97
  17. 2020Are Neural Nets Modular? Inspecting Functional Modularity Through Differentiable Weight Maskscited by 97
  18. 2019Neural Network Attributions: A Causal Perspectivecited by 77
  19. 2019Approximate Causal Abstractioncited by 62
  20. 2018Counterfactuals uncover the modular structure of deep generative modelscited by 51
  21. 2020Causal Effects of Linguistic Propertiescited by 39
  22. 2019Discovering the Compositional Structure of Vector Representations with Role Learning Networkscited by 33
  23. 2015Multi-Level Cause-Effect Systemscited by 30
  24. 2021Pointer Value Retrieval: A new benchmark for understanding the limits of neural network generalizationcited by 29
  25. 2019Posing Fair Generalization Tasks for Natural Language Inferencecited by 22
  26. 2021ReaSCAN: Compositional Reasoning in Language Groundingcited by 19
  27. 2018Analysing the potential of seq-to-seq models for incremental interpretation in task-oriented dialoguecited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 1; 2022: 10; 2023: 22; 2024: 28; 2025: 20; 2026: 26.

107 papers.

Sort:

By area:

  1. 2024Mechanistic Interpretability for AI Safety -- A Reviewcited by 415
  2. 2024Sparse Feature Circuits: Discovering and Editing Interpretable Causal Graphs in Language Modelscited by 412
  3. 2023Towards Best Practices of Activation Patching in Language Models: Metrics and Methodscited by 328
  4. 2023LEACE: Perfect linear concept erasure in closed formcited by 254
  5. 2023Reasoning or Reciting? Exploring the Capabilities and Limitations of Language Models Through Counterfactual Taskscited by 243
  6. 2025AxBench: Steering LLMs? Even Simple Baselines Outperform Sparse Autoencoderscited by 199
  7. 2023Finding Alignments Between Interpretable Causal Variables and Distributed Neural Representationscited by 197
  8. 2023Causal Abstraction: A Theoretical Foundation for Mechanistic Interpretabilitycited by 194
  9. 2024How to use and interpret activation patchingcited by 191
  10. 2024ReFT: Representation Finetuning for Language Modelscited by 186
  11. 2023Localizing Model Behavior with Path Patchingcited by 182
  12. 2023The Hydra Effect: Emergent Self-repair in Language Model Computationscited by 145
38 more
  1. 2022Towards Faithful Model Explanation in NLP: A Surveycited by 129
  2. 2022Toward Transparent AI: A Survey on Interpreting the Inner Structures of Deep Neural Networkscited by 126
  3. 2023Interpretability at Scale: Identifying Causal Mechanisms in Alpacacited by 102
  4. 2024AtP*: An efficient and scalable method for localizing LLM behaviour to componentscited by 100
  5. 2024A Primer on the Inner Workings of Transformer-based Language Modelscited by 83
  6. 2024pyvene: A Library for Understanding and Improving PyTorch Models via Interventionscited by 53
  7. 2024Knowledge Mechanisms in Large Language Models: A Survey and Perspectivecited by 50
  8. 2024CausalGym: Benchmarking causal interpretability methods on linguistic taskscited by 45
  9. 2023Rigorously Assessing Natural Language Explanations of Neuronscited by 44
  10. 2024Summing Up the Facts: Additive Mechanisms Behind Factual Recall in LLMscited by 39
  11. 2025Language Models use Lookbacks to Track Beliefscited by 36
  12. 2024Locating and Editing Factual Associations in Mambacited by 35
  13. 2025The Non-Linear Representation Dilemma: Is Causal Abstraction Enough for Mechanistic Interpretability?cited by 34
  14. 2023Discovering Variable Binding Circuitry with Desideratacited by 32
  15. 2025Everything, Everywhere, All at Once: Is Mechanistic Interpretability Identifiable?cited by 32
  16. 2023Towards Trustworthy and Aligned Machine Learning: A Data-centric Survey with Causality Perspectivescited by 30
  17. 2022Causal Proxy Models for Concept-Based Model Explanationscited by 29
  18. 2024Gradient Routing: Masking Gradients to Localize Computation in Neural Networkscited by 28
  19. 2023Circuit Breaking: Removing Model Behaviors with Targeted Ablationcited by 27
  20. 2023A Glitch in the Matrix? Locating and Detecting Language Model Grounding with Fakepediacited by 24
  21. 2024InterpBench: Semi-Synthetic Transformers for Evaluating Mechanistic Interpretability Techniquescited by 24
  22. 2024On Behalf of the Stakeholders: Trends in NLP Model Interpretability in the Era of LLMscited by 24
  23. 2023Analyzing And Editing Inner Mechanisms Of Backdoored Language Modelscited by 22
  24. 2021Causal Distillation for Language Modelscited by 19
  25. 2024The Landscape of Causal Discovery Data: Grounding Causal Discovery in Real-World Applicationscited by 17
  26. 2022Inducing Character-level Structure in Subword-based Language Models with Type-level Interchange Intervention Trainingcited by 16
  27. 2024A Reply to Makelov et al. (2023)'s "Interpretability Illusion" Argumentscited by 15
  28. 2024Do LLMs dream of elephants (when told not to)? Latent concept association and associative memory in transformerscited by 14
  29. 2024Scope Ambiguities in Large Language Modelscited by 11
  30. 2025RelP: Faithful and Efficient Circuit Discovery in Language Models via Relevance Patchingcited by 10
  31. 2025The Dead Salmons of AI Interpretabilitycited by 10
  32. 2025Capturing Polysemanticity with PRISM: A Multi-Concept Feature Description Frameworkcited by 9
  33. 2024Graph Neural Network Causal Explanation via Neural Causal Modelscited by 8
  34. 2026Causality is Key for Interpretability Claims to Generalisecited by 8
  35. 2023Has It All Been Solved? Open NLP Research Questions Not Solved by Large Language Modelscited by 7
  36. 2025Mechanisms vs. Outcomes: Probing for Syntax Fails to Explain Performance on Targeted Syntactic Evaluationscited by 7
  37. 2022Causal Discovery and Knowledge Injection for Contestable Neural Networks (with Appendices)cited by 6
  38. 2022Breakpoint Transformers for Modeling and Tracking Intermediate Beliefscited by 6

50 most cited of 107 All 107 (JSON)