# CausalGym: Benchmarking causal interpretability methods on linguistic tasks

arXiv: https://arxiv.org/abs/2402.12560 · HTML: https://mlsanity.com/paper/2402.12560/

## Key facts

- Cited by 46 papers in the map (data through 14 Oct 2026).
- Builds on 36 of its 58 references that are in the map.
- First posted on arXiv 19 Feb 2024.

## Builds on (most cited first, 30 of 36)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Locating and Editing Factual Associations in GPT](https://mlsanity.com/paper/2202.05262/) (2022)
- [Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling](https://mlsanity.com/paper/2304.01373/) (2023)
- [Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddings](https://mlsanity.com/paper/1607.06520/) (2016)
- [Inference-Time Intervention: Eliciting Truthful Answers from a Language Model](https://mlsanity.com/paper/2306.03341/) (2023)
- [Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 small](https://mlsanity.com/paper/2211.00593/) (2022)
- [Probing Classifiers: Promises, Shortcomings, and Advances](https://mlsanity.com/paper/2102.12452/) (2021)
- [The Geometry of Truth: Emergent Linear Structure in Large Language Model Representations of True/False Datasets](https://mlsanity.com/paper/2310.06824/) (2023)
- [Designing and Interpreting Probes with Control Tasks](https://mlsanity.com/paper/1909.03368/) (2019)
- [BLiMP: The Benchmark of Linguistic Minimal Pairs for English](https://mlsanity.com/paper/1912.00582/) (2019)
- [Causal Abstractions of Neural Networks](https://mlsanity.com/paper/2106.02997/) (2021)
- [SAGA: A Fast Incremental Gradient Method With Support for Non-Strongly Convex Composite Objectives](https://mlsanity.com/paper/1407.0202/) (2014)
- [Assessing the Ability of LSTMs to Learn Syntax-Sensitive Dependencies](https://mlsanity.com/paper/1611.01368/) (2016)
- [Emergent Linear Representations in World Models of Self-Supervised Sequence Models](https://mlsanity.com/paper/2309.00941/) (2023)
- [LEACE: Perfect linear concept erasure in closed form](https://mlsanity.com/paper/2306.03819/) (2023)
- [Targeted Syntactic Evaluation of Language Models](https://mlsanity.com/paper/1808.09031/) (2018)
- [Fine-grained Analysis of Sentence Embeddings Using Auxiliary Prediction Tasks](https://mlsanity.com/paper/1608.04207/) (2016)
- [Colorless green recurrent networks dream hierarchically](https://mlsanity.com/paper/1803.11138/) (2018)
- [Finding Alignments Between Interpretable Causal Variables and Distributed Neural Representations](https://mlsanity.com/paper/2303.02536/) (2023)
- [Causal Abstraction: A Theoretical Foundation for Mechanistic Interpretability](https://mlsanity.com/paper/2301.04709/) (2023)
- [Localizing Model Behavior with Path Patching](https://mlsanity.com/paper/2304.05969/) (2023)
- [A Systematic Assessment of Syntactic Generalization in Neural Language Models](https://mlsanity.com/paper/2005.03692/) (2020)
- [Neural Language Models as Psycholinguistic Subjects: Representations of Syntactic State](https://mlsanity.com/paper/1903.03260/) (2019)
- [Sudden Drops in the Loss: Syntax Acquisition, Phase Transitions, and Simplicity Bias in MLMs](https://mlsanity.com/paper/2309.07311/) (2023)
- [Inducing Causal Structure for Interpretable Neural Networks](https://mlsanity.com/paper/2112.00826/) (2021)
- [Interpretability at Scale: Identifying Causal Mechanisms in Alpaca](https://mlsanity.com/paper/2305.08809/) (2023)
- [What Artificial Neural Networks Can Tell Us About Human Language Acquisition](https://mlsanity.com/paper/2208.07998/) (2022)
- [Probing for the Usage of Grammatical Number](https://mlsanity.com/paper/2204.08831/) (2022)
- [Language models align with human judgments on key grammatical constructions](https://mlsanity.com/paper/2402.01676/) (2024)

## Built on by (most cited first, 30 of 46)

- [Mechanistic Interpretability for AI Safety -- A Review](https://mlsanity.com/paper/2404.14082/) (2024)
- [ReFT: Representation Finetuning for Language Models](https://mlsanity.com/paper/2404.03592/) (2024)
- [A Primer on the Inner Workings of Transformer-based Language Models](https://mlsanity.com/paper/2405.00208/) (2024)
- [Large Language Models Share Representations of Latent Grammatical Concepts Across Typologically Diverse Languages](https://mlsanity.com/paper/2501.06346/) (2025)
- [MIB: A Mechanistic Interpretability Benchmark](https://mlsanity.com/paper/2504.13151/) (2025)
- [The Non-Linear Representation Dilemma: Is Causal Abstraction Enough for Mechanistic Interpretability?](https://mlsanity.com/paper/2507.08802/) (2025)
- [Manifold Steering Reveals the Shared Geometry of Neural Network Representation and Behavior](https://mlsanity.com/paper/2605.05115/) (2026)
- [InterpBench: Semi-Synthetic Transformers for Evaluating Mechanistic Interpretability Techniques](https://mlsanity.com/paper/2407.14494/) (2024)
- [Mechanistic?](https://mlsanity.com/paper/2410.09087/) (2024)
- [Language Model Circuits Are Sparse in the Neuron Basis](https://mlsanity.com/paper/2601.22594/) (2026)
- [The Quest for the Right Mediator: Surveying Mechanistic Interpretability Through the Lens of Causal Mediation Analysis](https://mlsanity.com/paper/2408.01416/) (2024)
- [On the Similarity of Circuits across Languages: a Case Study on the Subject-verb Agreement Task](https://mlsanity.com/paper/2410.06496/) (2024)
- [Causal Interventions Reveal Shared Structure Across English Filler-Gap Constructions](https://mlsanity.com/paper/2505.16002/) (2025)
- [A Psycholinguistic Evaluation of Language Models' Sensitivity to Argument Roles](https://mlsanity.com/paper/2410.16139/) (2024)
- [A Systematic Assessment of Language Models with Linguistic Minimal Pairs in Chinese](https://mlsanity.com/paper/2411.06096/) (2024)
- [Steering Vector Fields for Context-Aware Inference-Time Control in Large Language Models](https://mlsanity.com/paper/2602.01654/) (2026)
- [How Reliable are Causal Probing Interventions?](https://mlsanity.com/paper/2408.15510/) (2024)
- [Characterizing the Role of Similarity in the Property Inferences of Language Models](https://mlsanity.com/paper/2410.22590/) (2024)
- [A Close Look at Decomposition-based XAI-Methods for Transformer Language Models](https://mlsanity.com/paper/2502.15886/) (2025)
- [Internal Causal Mechanisms Robustly Predict Language Model Out-of-Distribution Behaviors](https://mlsanity.com/paper/2505.11770/) (2025)
- [How Causal Abstraction Underpins Computational Explanation](https://mlsanity.com/paper/2508.11214/) (2025)
- [Mechanisms vs. Outcomes: Probing for Syntax Fails to Explain Performance on Targeted Syntactic Evaluations](https://mlsanity.com/paper/2506.16678/) (2025)
- [Analyzing the Inner Workings of Transformers in Compositional Generalization](https://mlsanity.com/paper/2502.15277/) (2025)
- [Learning Distribution-Wise Control in Representation Space for Language Models](https://mlsanity.com/paper/2506.06686/) (2025)
- [Pando: Do Interpretability Methods Work When Models Won't Explain Themselves?](https://mlsanity.com/paper/2604.11061/) (2026)
- [Challenges and Opportunities in Text Generation Explainability](https://mlsanity.com/paper/2405.08468/) (2024)
- [Steering Language Models in Multi-Token Generation: A Case Study on Tense and Aspect](https://mlsanity.com/paper/2509.12065/) (2025)
- [Humans and transformer LMs: Abstraction drives language learning](https://mlsanity.com/paper/2603.17475/) (2026)
- [Using Shapley interactions to understand how models use structure](https://mlsanity.com/paper/2403.13106/) (2024)
- [Linguistic Minimal Pairs Elicit Linguistic Similarity in Large Language Models](https://mlsanity.com/paper/2409.12435/) (2024)

## Abstract (arXiv)

Language models (LMs) have proven to be powerful tools for psycholinguistic research, but most prior work has focused on purely behavioural measures (e.g., surprisal comparisons). At the same time, research in model interpretability has begun to illuminate the abstract causal mechanisms shaping LM behavior. To help bring these strands of research closer together, we introduce CausalGym. We adapt and expand the SyntaxGym suite of tasks to benchmark the ability of interpretability methods to causally affect model behaviour. To illustrate how CausalGym can be used, we study the pythia models (14M--6.9B) and assess the causal efficacy of a wide range of interpretability methods, including linear probing and distributed alignment search (DAS). We find that DAS outperforms the other methods, and so we use it to study the learning trajectory of two difficult linguistic phenomena in pythia-1b: negative polarity item licensing and filler--gap dependencies. Our analysis shows that the mechanism implementing both of these tasks is learned in discrete stages, not gradually.