# Finding Alignments Between Interpretable Causal Variables and Distributed Neural Representations

arXiv: https://arxiv.org/abs/2303.02536 · HTML: https://mlsanity.com/paper/2303.02536/

## Key facts

- Cited by 209 papers in the map (data through 14 Oct 2026).
- Builds on 31 of its 48 references that are in the map.
- First posted on arXiv 5 Mar 2023.

## Builds on (most cited first, 30 of 31)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- ["Why Should I Trust You?": Explaining the Predictions of Any Classifier](https://mlsanity.com/paper/1602.04938/) (2016)
- [Locating and Editing Factual Associations in GPT](https://mlsanity.com/paper/2202.05262/) (2022)
- [A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference](https://mlsanity.com/paper/1704.05426/) (2017)
- [Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 small](https://mlsanity.com/paper/2211.00593/) (2022)
- [In-context Learning and Induction Heads](https://mlsanity.com/paper/2209.11895/) (2022)
- [The Mythos of Model Interpretability](https://mlsanity.com/paper/1606.03490/) (2016)
- [Towards Faithfully Interpretable NLP Systems: How should we define and evaluate faithfulness?](https://mlsanity.com/paper/2004.03685/) (2020)
- [Null It Out: Guarding Protected Attributes by Iterative Nullspace Projection](https://mlsanity.com/paper/2004.07667/) (2020)
- [Causal Abstractions of Neural Networks](https://mlsanity.com/paper/2106.02997/) (2021)
- [Direct and Indirect Effects](https://mlsanity.com/paper/1301.2300/) (2013)
- [Causal Mediation Analysis for Interpreting Neural NLP: The Case of Gender Bias](https://mlsanity.com/paper/2004.12265/) (2020)
- [Implicit Representations of Meaning in Neural Language Models](https://mlsanity.com/paper/2106.00737/) (2021)
- [Causal Analysis of Syntactic Agreement Mechanisms in Neural Language Models](https://mlsanity.com/paper/2106.06087/) (2021)
- [Towards Faithful Model Explanation in NLP: A Survey](https://mlsanity.com/paper/2209.11326/) (2022)
- [Causal Consistency of Structural Equation Models](https://mlsanity.com/paper/1707.00819/) (2017)
- [CausaLM: Causal Model Explanation Through Counterfactual Language Models](https://mlsanity.com/paper/2005.13407/) (2020)
- [Inducing Causal Structure for Interpretable Neural Networks](https://mlsanity.com/paper/2112.00826/) (2021)
- [Linear Adversarial Concept Erasure](https://mlsanity.com/paper/2201.12091/) (2022)
- [Are Neural Nets Modular? Inspecting Functional Modularity Through Differentiable Weight Masks](https://mlsanity.com/paper/2010.02066/) (2020)
- [Approximate Causal Abstraction](https://mlsanity.com/paper/1906.11583/) (2019)
- [CEBaB: Estimating the Causal Effects of Real-World Concepts on NLP Model Behavior](https://mlsanity.com/paper/2205.14140/) (2022)
- [Sparse Interventions in Language Models with Differentiable Masking](https://mlsanity.com/paper/2112.06837/) (2021)
- [Compositional Abstraction Error and a Category of Causal Models](https://mlsanity.com/paper/2103.15758/) (2021)
- [Causal Proxy Models for Concept-Based Model Explanations](https://mlsanity.com/paper/2209.14279/) (2022)
- [Causal Abstraction with Soft Interventions](https://mlsanity.com/paper/2211.12270/) (2022)
- [Posing Fair Generalization Tasks for Natural Language Inference](https://mlsanity.com/paper/1911.00811/) (2019)
- [Inducing Character-level Structure in Subword-based Language Models with Type-level Interchange Intervention Training](https://mlsanity.com/paper/2212.09897/) (2022)
- [Relational reasoning and generalization using non-symbolic neural networks](https://mlsanity.com/paper/2006.07968/) (2020)

## Built on by (most cited first, 30 of 209)

- [Refusal in Language Models Is Mediated by a Single Direction](https://mlsanity.com/paper/2406.11717/) (2024)
- [Mechanistic Interpretability for AI Safety -- A Review](https://mlsanity.com/paper/2404.14082/) (2024)
- [Towards Best Practices of Activation Patching in Language Models: Metrics and Methods](https://mlsanity.com/paper/2309.16042/) (2023)
- [ReFT: Representation Finetuning for Language Models](https://mlsanity.com/paper/2404.03592/) (2024)
- [Localizing Model Behavior with Path Patching](https://mlsanity.com/paper/2304.05969/) (2023)
- [The Hydra Effect: Emergent Self-repair in Language Model Computations](https://mlsanity.com/paper/2307.15771/) (2023)
- [Does Circuit Analysis Interpretability Scale? Evidence from Multiple Choice Capabilities in Chinchilla](https://mlsanity.com/paper/2307.09458/) (2023)
- [A Practical Review of Mechanistic Interpretability for Transformer-Based Language Models](https://mlsanity.com/paper/2407.02646/) (2024)
- [Interpretability at Scale: Identifying Causal Mechanisms in Alpaca](https://mlsanity.com/paper/2305.08809/) (2023)
- [Towards Principled Evaluations of Sparse Autoencoders for Interpretability and Control](https://mlsanity.com/paper/2405.08366/) (2024)
- [A Primer on the Inner Workings of Transformer-based Language Models](https://mlsanity.com/paper/2405.00208/) (2024)
- [Is This the Subspace You Are Looking for? An Interpretability Illusion for Subspace Activation Patching](https://mlsanity.com/paper/2311.17030/) (2023)
- [RAVEL: Evaluating Interpretability Methods on Disentangling Language Model Representations](https://mlsanity.com/paper/2402.17700/) (2024)
- [Controlling Language and Diffusion Models by Transporting Activations](https://mlsanity.com/paper/2410.23054/) (2024)
- [Attention Heads of Large Language Models: A Survey](https://mlsanity.com/paper/2409.03752/) (2024)
- [Aligning Large Language Models with Representation Editing: A Control Perspective](https://mlsanity.com/paper/2406.05954/) (2024)
- [Finding Transformer Circuits with Edge Pruning](https://mlsanity.com/paper/2406.16778/) (2024)
- [MIB: A Mechanistic Interpretability Benchmark](https://mlsanity.com/paper/2504.13151/) (2025)
- [pyvene: A Library for Understanding and Improving PyTorch Models via Interventions](https://mlsanity.com/paper/2403.07809/) (2024)
- [Towards Understanding Safety Alignment: A Mechanistic Perspective from Safety Neurons](https://mlsanity.com/paper/2406.14144/) (2024)
- [LLaVA Steering: Visual Instruction Tuning with 500x Fewer Parameters through Modality Linear Representation-Steering](https://mlsanity.com/paper/2412.12359/) (2024)
- [Do Language Models Use Their Depth Efficiently?](https://mlsanity.com/paper/2505.13898/) (2025)
- [Codebook Features: Sparse and Discrete Interpretability for Neural Networks](https://mlsanity.com/paper/2310.17230/) (2023)
- [Recurrent Neural Networks Learn to Store and Generate Sequences using Non-Linear Representations](https://mlsanity.com/paper/2408.10920/) (2024)
- [CausalGym: Benchmarking causal interpretability methods on linguistic tasks](https://mlsanity.com/paper/2402.12560/) (2024)
- [A Unified Understanding and Evaluation of Steering Methods](https://mlsanity.com/paper/2502.02716/) (2025)
- [Rigorously Assessing Natural Language Explanations of Neurons](https://mlsanity.com/paper/2309.10312/) (2023)
- [Monitoring Latent World States in Language Models with Propositional Probes](https://mlsanity.com/paper/2406.19501/) (2024)
- [Learning Transformer Programs](https://mlsanity.com/paper/2306.01128/) (2023)
- [Enhancing Automated Interpretability with Output-Centric Feature Descriptions](https://mlsanity.com/paper/2501.08319/) (2025)

## Abstract (arXiv)

Causal abstraction is a promising theoretical framework for explainable artificial intelligence that defines when an interpretable high-level causal model is a faithful simplification of a low-level deep learning system. However, existing causal abstraction methods have two major limitations: they require a brute-force search over alignments between the high-level model and the low-level one, and they presuppose that variables in the high-level model will align with disjoint sets of neurons in the low-level one. In this paper, we present distributed alignment search (DAS), which overcomes these limitations. In DAS, we find the alignment between high-level and low-level models using gradient descent rather than conducting a brute-force search, and we allow individual neurons to play multiple distinct roles by analyzing representations in non-standard bases-distributed representations. Our experiments show that DAS can discover internal structure that prior approaches miss. Overall, DAS removes previous obstacles to conducting causal abstraction analyses and allows us to find conceptual structure in trained neural nets.