mlsanityData to 7 Oct 2026

Finding Alignments Between Interpretable Causal Variables and Distributed Neural Representations

arXiv 2303.02536 ↗First posted 5 Mar 2023cs.AIcited by 197

Not on the map. Why?

Abstract (arXiv)

Causal abstraction is a promising theoretical framework for explainable artificial intelligence that defines when an interpretable high-level causal model is a faithful simplification of a low-level deep learning system. However, existing causal abstraction methods have two major limitations: they require a brute-force search over alignments between the high-level model and the low-level one, and they presuppose that variables in the high-level model will align with disjoint sets of neurons in the low-level one. In this paper, we present distributed alignment search (DAS), which overcomes these limitations. In DAS, we find the alignment between high-level and low-level models using gradient descent rather than conducting a brute-force search, and we allow individual neurons to play multiple distinct roles by analyzing representations in non-standard bases-distributed representations. Our experiments show that DAS can discover internal structure that prior approaches miss. Overall, DAS removes previous obstacles to conducting causal abstraction analyses and allows us to find conceptual structure in trained neural nets.

Key facts. Cited by 197 papers in the map (data through 7 Oct 2026). Builds on 31 of its 48 references that are in the map. First posted on arXiv 5 Mar 2023.

Builds on

31 of 48 references.

References · 31

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2016"Why Should I Trust You?": Explaining the Predictions of Any Classifiercited by 5,040
  4. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
  5. 2022Locating and Editing Factual Associations in GPTcited by 2,609
  6. 2022Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 smallcited by 1,130
  7. 2022In-context Learning and Induction Headscited by 930
  8. 2016The Mythos of Model Interpretabilitycited by 716
  9. 2020Towards Faithfully Interpretable NLP Systems: How should we define and evaluate faithfulness?cited by 590
  10. 2020Null It Out: Guarding Protected Attributes by Iterative Nullspace Projectioncited by 442
  11. 2021Causal Abstractions of Neural Networkscited by 401
  12. 2013Direct and Indirect Effectscited by 276
19 more
  1. 2020Causal Mediation Analysis for Interpreting Neural NLP: The Case of Gender Biascited by 183
  2. 2021Implicit Representations of Meaning in Neural Language Modelscited by 179
  3. 2022Towards Faithful Model Explanation in NLP: A Surveycited by 129
  4. 2021Causal Analysis of Syntactic Agreement Mechanisms in Neural Language Modelscited by 127
  5. 2020CausaLM: Causal Model Explanation Through Counterfactual Language Modelscited by 117
  6. 2017Causal Consistency of Structural Equation Modelscited by 114
  7. 2021Inducing Causal Structure for Interpretable Neural Networkscited by 107
  8. 2022Linear Adversarial Concept Erasurecited by 98
  9. 2020Are Neural Nets Modular? Inspecting Functional Modularity Through Differentiable Weight Maskscited by 97
  10. 2019Approximate Causal Abstractioncited by 62
  11. 2022CEBaB: Estimating the Causal Effects of Real-World Concepts on NLP Model Behaviorcited by 54
  12. 2021Sparse Interventions in Language Models with Differentiable Maskingcited by 34
  13. 2021Compositional Abstraction Error and a Category of Causal Modelscited by 32
  14. 2022Causal Proxy Models for Concept-Based Model Explanationscited by 29
  15. 2022Causal Abstraction with Soft Interventionscited by 23
  16. 2019Posing Fair Generalization Tasks for Natural Language Inferencecited by 22
  17. 2022Inducing Character-level Structure in Subword-based Language Models with Type-level Interchange Intervention Trainingcited by 16
  18. 2020Relational reasoning and generalization using non-symbolic neural networkscited by 15
  19. 2018On the Conditional Logic of Simulation Modelscited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 19; 2024: 55; 2025: 46; 2026: 77.

197 papers.

Sort:

By area:

  1. 2024Refusal in Language Models Is Mediated by a Single Directioncited by 960
  2. 2024Mechanistic Interpretability for AI Safety -- A Reviewcited by 415
  3. 2023Towards Best Practices of Activation Patching in Language Models: Metrics and Methodscited by 328
  4. 2024ReFT: Representation Finetuning for Language Modelscited by 186
  5. 2023Localizing Model Behavior with Path Patchingcited by 182
  6. 2023Does Circuit Analysis Interpretability Scale? Evidence from Multiple Choice Capabilities in Chinchillacited by 145
  7. 2023The Hydra Effect: Emergent Self-repair in Language Model Computationscited by 145
  8. 2024A Practical Review of Mechanistic Interpretability for Transformer-Based Language Modelscited by 113
  9. 2023Interpretability at Scale: Identifying Causal Mechanisms in Alpacacited by 102
  10. 2024A Primer on the Inner Workings of Transformer-based Language Modelscited by 83
  11. 2024Towards Principled Evaluations of Sparse Autoencoders for Interpretability and Controlcited by 83
  12. 2024RAVEL: Evaluating Interpretability Methods on Disentangling Language Model Representationscited by 77
38 more
  1. 2023Is This the Subspace You Are Looking for? An Interpretability Illusion for Subspace Activation Patchingcited by 75
  2. 2024Attention Heads of Large Language Models: A Surveycited by 65
  3. 2024Aligning Large Language Models with Representation Editing: A Control Perspectivecited by 62
  4. 2024Controlling Language and Diffusion Models by Transporting Activationscited by 59
  5. 2024Finding Transformer Circuits with Edge Pruningcited by 58
  6. 2024pyvene: A Library for Understanding and Improving PyTorch Models via Interventionscited by 53
  7. 2024LLaVA Steering: Visual Instruction Tuning with 500x Fewer Parameters through Modality Linear Representation-Steeringcited by 50
  8. 2025MIB: A Mechanistic Interpretability Benchmarkcited by 49
  9. 2025Do Language Models Use Their Depth Efficiently?cited by 48
  10. 2024Towards Understanding Safety Alignment: A Mechanistic Perspective from Safety Neuronscited by 47
  11. 2023Codebook Features: Sparse and Discrete Interpretability for Neural Networkscited by 45
  12. 2024CausalGym: Benchmarking causal interpretability methods on linguistic taskscited by 45
  13. 2024Recurrent Neural Networks Learn to Store and Generate Sequences using Non-Linear Representationscited by 45
  14. 2023Rigorously Assessing Natural Language Explanations of Neuronscited by 44
  15. 2023Learning Transformer Programscited by 40
  16. 2025A Unified Understanding and Evaluation of Steering Methodscited by 40
  17. 2024Monitoring Latent World States in Language Models with Propositional Probescited by 39
  18. 2025Enhancing Automated Interpretability with Output-Centric Feature Descriptionscited by 39
  19. 2024Evaluating Open-Source Sparse Autoencoders on Disentangling Factual Knowledge in GPT-2 Smallcited by 34
  20. 2025The Non-Linear Representation Dilemma: Is Causal Abstraction Enough for Mechanistic Interpretability?cited by 34
  21. 2023Discovering Variable Binding Circuitry with Desideratacited by 32
  22. 2025Everything, Everywhere, All at Once: Is Mechanistic Interpretability Identifiable?cited by 32
  23. 2024Mission: Impossible Language Modelscited by 31
  24. 2024Hypothesis Testing the Circuit Hypothesis in LLMscited by 29
  25. 2023Circuit Breaking: Removing Model Behaviors with Targeted Ablationcited by 27
  26. 2024Large Language Models and Causal Inference in Collaboration: A Surveycited by 26
  27. 2024Mechanistic?cited by 25
  28. 2024InterpBench: Semi-Synthetic Transformers for Evaluating Mechanistic Interpretability Techniquescited by 24
  29. 2024The Quest for the Right Mediator: Surveying Mechanistic Interpretability Through the Lens of Causal Mediation Analysiscited by 24
  30. 2023A Geometric Notion of Causal Probingcited by 23
  31. 2023Interpretability is in the Mind of the Beholder: A Causal Framework for Human-interpretable Representation Learningcited by 22
  32. 2024Towards Uncovering How Large Language Model Works: An Explainability Perspectivecited by 22
  33. 2024Learned feature representations are biased by complexity, learning order, position, and morecited by 22
  34. 2024Controllable Context Sensitivity and the Knob Behind Itcited by 22
  35. 2024Missed Causes and Ambiguous Effects: Counterfactuals Pose Challenges for Interpreting Neural Networkscited by 21
  36. 2025EAP-GP: Mitigating Saturation Effect in Gradient-based Automated Circuit Identificationcited by 21
  37. 2025Causal Interventions Reveal Shared Structure Across English Filler-Gap Constructionscited by 21
  38. 2025Prompt-based Adaptation in Large-scale Vision Models: A Surveycited by 21

50 most cited of 197 All 197 (JSON)