# RAVEL: Evaluating Interpretability Methods on Disentangling Language Model Representations

arXiv: https://arxiv.org/abs/2402.17700 · HTML: https://mlsanity.com/paper/2402.17700/

## Key facts

- Cited by 79 papers in the map (data through 14 Oct 2026).
- Placed in Knowledge and representations inside LLMs, one of 13 directions in Mechanistic interpretability.
- Builds on 46 of its 74 references that are in the map.
- First posted on arXiv 27 Feb 2024.

## Placed in

- Mechanistic interpretability / [Knowledge and representations inside LLMs](https://mlsanity.com/mechinterp/knowledge-and-representations-inside-llms/)

## Builds on (most cited first, 30 of 46)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Locating and Editing Factual Associations in GPT](https://mlsanity.com/paper/2202.05262/) (2022)
- [Sparse Autoencoders Find Highly Interpretable Features in Language Models](https://mlsanity.com/paper/2309.08600/) (2023)
- [Transformer Feed-Forward Layers Are Key-Value Memories](https://mlsanity.com/paper/2012.14913/) (2020)
- [Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 small](https://mlsanity.com/paper/2211.00593/) (2022)
- [BERT Rediscovers the Classical NLP Pipeline](https://mlsanity.com/paper/1905.05950/) (2019)
- [What Does BERT Look At? An Analysis of BERT's Attention](https://mlsanity.com/paper/1906.04341/) (2019)
- [Challenging Common Assumptions in the Unsupervised Learning of Disentangled Representations](https://mlsanity.com/paper/1811.12359/) (2018)
- [Towards Automated Circuit Discovery for Mechanistic Interpretability](https://mlsanity.com/paper/2304.14997/) (2023)
- [The Geometry of Truth: Emergent Linear Structure in Large Language Model Representations of True/False Datasets](https://mlsanity.com/paper/2310.06824/) (2023)
- [Dissecting Recall of Factual Associations in Auto-Regressive Language Models](https://mlsanity.com/paper/2304.14767/) (2023)
- [What you can cram into a single vector: Probing sentence embeddings for linguistic properties](https://mlsanity.com/paper/1805.01070/) (2018)
- [Null It Out: Guarding Protected Attributes by Iterative Nullspace Projection](https://mlsanity.com/paper/2004.07667/) (2020)
- [Causal Abstractions of Neural Networks](https://mlsanity.com/paper/2106.02997/) (2021)
- [Finding Neurons in a Haystack: Case Studies with Sparse Probing](https://mlsanity.com/paper/2305.01610/) (2023)
- [In-Context Learning Creates Task Vectors](https://mlsanity.com/paper/2310.15916/) (2023)
- [MQuAKE: Assessing Knowledge Editing in Language Models via Multi-Hop Questions](https://mlsanity.com/paper/2305.14795/) (2023)
- [Direct and Indirect Effects](https://mlsanity.com/paper/1301.2300/) (2013)
- [LEACE: Perfect linear concept erasure in closed form](https://mlsanity.com/paper/2306.03819/) (2023)
- [Does Localization Inform Editing? Surprising Differences in Causality-Based Localization vs. Knowledge Editing in Language Models](https://mlsanity.com/paper/2301.04213/) (2023)
- [Evaluating the Ripple Effects of Knowledge Editing in Language Models](https://mlsanity.com/paper/2307.12976/) (2023)
- [Patchscopes: A Unifying Framework for Inspecting Hidden Representations of Language Models](https://mlsanity.com/paper/2401.06102/) (2024)
- [Finding Alignments Between Interpretable Causal Variables and Distributed Neural Representations](https://mlsanity.com/paper/2303.02536/) (2023)
- [Information-Theoretic Probing for Linguistic Structure](https://mlsanity.com/paper/2004.03061/) (2020)
- [Linearity of Relation Decoding in Transformer Language Models](https://mlsanity.com/paper/2308.09124/) (2023)
- [Localizing Model Behavior with Path Patching](https://mlsanity.com/paper/2304.05969/) (2023)
- [Implicit Representations of Meaning in Neural Language Models](https://mlsanity.com/paper/2106.00737/) (2021)
- [Linear Algebraic Structure of Word Senses, with Applications to Polysemy](https://mlsanity.com/paper/1601.03764/) (2016)
- [Linear Representations of Sentiment in Large Language Models](https://mlsanity.com/paper/2310.15154/) (2023)
- [Does Circuit Analysis Interpretability Scale? Evidence from Multiple Choice Capabilities in Chinchilla](https://mlsanity.com/paper/2307.09458/) (2023)

## Built on by (most cited first, 30 of 79)

- [Mechanistic Interpretability for AI Safety -- A Review](https://mlsanity.com/paper/2404.14082/) (2024)
- [ReFT: Representation Finetuning for Language Models](https://mlsanity.com/paper/2404.03592/) (2024)
- [Llama Scope: Extracting Millions of Features from Llama-3.1-8B with Sparse Autoencoders](https://mlsanity.com/paper/2410.20526/) (2024)
- [SAEBench: A Comprehensive Benchmark for Sparse Autoencoders in Language Model Interpretability](https://mlsanity.com/paper/2503.09532/) (2025)
- [A Practical Review of Mechanistic Interpretability for Transformer-Based Language Models](https://mlsanity.com/paper/2407.02646/) (2024)
- [Are Sparse Autoencoders Useful? A Case Study in Sparse Probing](https://mlsanity.com/paper/2502.16681/) (2025)
- [A Primer on the Inner Workings of Transformer-based Language Models](https://mlsanity.com/paper/2405.00208/) (2024)
- [MIB: A Mechanistic Interpretability Benchmark](https://mlsanity.com/paper/2504.13151/) (2025)
- [On Large Language Model Continual Unlearning](https://mlsanity.com/paper/2407.10223/) (2024)
- [Knowledge Mechanisms in Large Language Models: A Survey and Perspective](https://mlsanity.com/paper/2407.15017/) (2024)
- [Do Language Models Use Their Depth Efficiently?](https://mlsanity.com/paper/2505.13898/) (2025)
- [Recurrent Neural Networks Learn to Store and Generate Sequences using Non-Linear Representations](https://mlsanity.com/paper/2408.10920/) (2024)
- [Enhancing Automated Interpretability with Output-Centric Feature Descriptions](https://mlsanity.com/paper/2501.08319/) (2025)
- [Estimating Knowledge in Large Language Models Without Generating a Single Token](https://mlsanity.com/paper/2406.12673/) (2024)
- [A Survey on Mechanistic Interpretability for Multi-Modal Foundation Models](https://mlsanity.com/paper/2502.17516/) (2025)
- [Intrinsic Test of Unlearning Using Parametric Knowledge Traces](https://mlsanity.com/paper/2406.11614/) (2024)
- [Manifold Steering Reveals the Shared Geometry of Neural Network Representation and Behavior](https://mlsanity.com/paper/2605.05115/) (2026)
- [InterpBench: Semi-Synthetic Transformers for Evaluating Mechanistic Interpretability Techniques](https://mlsanity.com/paper/2407.14494/) (2024)
- [The Quest for the Right Mediator: Surveying Mechanistic Interpretability Through the Lens of Causal Mediation Analysis](https://mlsanity.com/paper/2408.01416/) (2024)
- [Residual Stream Analysis with Multi-Layer SAEs](https://mlsanity.com/paper/2409.04185/) (2024)
- [Sanity Checks for Sparse Autoencoders: Do SAEs Beat Random Baselines?](https://mlsanity.com/paper/2602.14111/) (2026)
- [Evaluating Adversarial Robustness of Concept Representations in Sparse Autoencoders](https://mlsanity.com/paper/2505.16004/) (2025)
- [RepIt: Steering Language Models with Concept-Specific Refusal Vectors](https://mlsanity.com/paper/2509.13281/) (2025)
- [From Directions to Regions: Decomposing Activations in Language Models via Local Geometry](https://mlsanity.com/paper/2602.02464/) (2026)
- [How Reliable are Causal Probing Interventions?](https://mlsanity.com/paper/2408.15510/) (2024)
- [Characterizing the Role of Similarity in the Property Inferences of Language Models](https://mlsanity.com/paper/2410.22590/) (2024)
- [Internal Causal Mechanisms Robustly Predict Language Model Out-of-Distribution Behaviors](https://mlsanity.com/paper/2505.11770/) (2025)
- [Precise In-Parameter Concept Erasure in Large Language Models](https://mlsanity.com/paper/2505.22586/) (2025)
- [How Causal Abstraction Underpins Computational Explanation](https://mlsanity.com/paper/2508.11214/) (2025)
- [SALT: Steering Activations towards Leakage-free Thinking in Chain of Thought](https://mlsanity.com/paper/2511.07772/) (2025)

## Abstract (arXiv)

Individual neurons participate in the representation of multiple high-level concepts. To what extent can different interpretability methods successfully disentangle these roles? To help address this question, we introduce RAVEL (Resolving Attribute-Value Entanglements in Language Models), a dataset that enables tightly controlled, quantitative comparisons between a variety of existing interpretability methods. We use the resulting conceptual framework to define the new method of Multi-task Distributed Alignment Search (MDAS), which allows us to find distributed representations satisfying multiple causal criteria. With Llama2-7B as the target language model, MDAS achieves state-of-the-art results on RAVEL, demonstrating the importance of going beyond neuron-level analyses to identify features distributed across activations. We release our benchmark at https://github.com/explanare/ravel.