mlsanityData to 7 Oct 2026

RAVEL: Evaluating Interpretability Methods on Disentangling Language Model Representations

arXiv 2402.17700 ↗First posted 27 Feb 2024cs.CL cs.LGcited by 77

Not on the map. Why?

Abstract (arXiv)

Individual neurons participate in the representation of multiple high-level concepts. To what extent can different interpretability methods successfully disentangle these roles? To help address this question, we introduce RAVEL (Resolving Attribute-Value Entanglements in Language Models), a dataset that enables tightly controlled, quantitative comparisons between a variety of existing interpretability methods. We use the resulting conceptual framework to define the new method of Multi-task Distributed Alignment Search (MDAS), which allows us to find distributed representations satisfying multiple causal criteria. With Llama2-7B as the target language model, MDAS achieves state-of-the-art results on RAVEL, demonstrating the importance of going beyond neuron-level analyses to identify features distributed across activations. We release our benchmark at https://github.com/explanare/ravel.

Key facts. Cited by 77 papers in the map (data through 7 Oct 2026). Builds on 46 of its 74 references that are in the map. First posted on arXiv 27 Feb 2024.

Builds on

46 of 74 references.

References · 46

  1. 2017Attention Is All You Needcited by 59,932
  2. 2022Locating and Editing Factual Associations in GPTcited by 2,609
  3. 2023Sparse Autoencoders Find Highly Interpretable Features in Language Modelscited by 1,359
  4. 2020Transformer Feed-Forward Layers Are Key-Value Memoriescited by 1,307
  5. 2022Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 smallcited by 1,130
  6. 2019BERT Rediscovers the Classical NLP Pipelinecited by 1,046
  7. 2019What Does BERT Look At? An Analysis of BERT's Attentioncited by 994
  8. 2018Challenging Common Assumptions in the Unsupervised Learning of Disentangled Representationscited by 929
  9. 2023Towards Automated Circuit Discovery for Mechanistic Interpretabilitycited by 700
  10. 2023The Geometry of Truth: Emergent Linear Structure in Large Language Model Representations of True/False Datasetscited by 673
  11. 2023Dissecting Recall of Factual Associations in Auto-Regressive Language Modelscited by 512
  12. 2018What you can cram into a single vector: Probing sentence embeddings for linguistic propertiescited by 452
34 more
  1. 2020Null It Out: Guarding Protected Attributes by Iterative Nullspace Projectioncited by 442
  2. 2021Causal Abstractions of Neural Networkscited by 401
  3. 2023Finding Neurons in a Haystack: Case Studies with Sparse Probingcited by 328
  4. 2023In-Context Learning Creates Task Vectorscited by 302
  5. 2023MQuAKE: Assessing Knowledge Editing in Language Models via Multi-Hop Questionscited by 292
  6. 2013Direct and Indirect Effectscited by 276
  7. 2023LEACE: Perfect linear concept erasure in closed formcited by 254
  8. 2023Evaluating the Ripple Effects of Knowledge Editing in Language Modelscited by 238
  9. 2023Does Localization Inform Editing? Surprising Differences in Causality-Based Localization vs. Knowledge Editing in Language Modelscited by 237
  10. 2024Patchscopes: A Unifying Framework for Inspecting Hidden Representations of Language Modelscited by 225
  11. 2020Information-Theoretic Probing for Linguistic Structurecited by 204
  12. 2023Finding Alignments Between Interpretable Causal Variables and Distributed Neural Representationscited by 197
  13. 2023Linearity of Relation Decoding in Transformer Language Modelscited by 190
  14. 2023Localizing Model Behavior with Path Patchingcited by 182
  15. 2021Implicit Representations of Meaning in Neural Language Modelscited by 179
  16. 2016Linear Algebraic Structure of Word Senses, with Applications to Polysemycited by 165
  17. 2023Linear Representations of Sentiment in Large Language Modelscited by 157
  18. 2018Dissecting Contextual Word Embeddings: Architecture and Representationcited by 146
  19. 2023Does Circuit Analysis Interpretability Scale? Evidence from Multiple Choice Capabilities in Chinchillacited by 145
  20. 2021Causal Analysis of Syntactic Agreement Mechanisms in Neural Language Modelscited by 127
  21. 2023Faithfulness Tests for Natural Language Explanationscited by 107
  22. 2023How do Language Models Bind Entities in Context?cited by 102
  23. 2020Are Neural Nets Modular? Inspecting Functional Modularity Through Differentiable Weight Maskscited by 97
  24. 2021An Interpretability Illusion for BERTcited by 94
  25. 2020How do Decisions Emerge across Layers in Neural Models? Interpretation with Differentiable Maskingcited by 63
  26. 2019Approximate Causal Abstractioncited by 62
  27. 2021Conditional probing: measuring usable information beyond a baselinecited by 61
  28. 2022CEBaB: Estimating the Causal Effects of Real-World Concepts on NLP Model Behaviorcited by 54
  29. 2023Codebook Features: Sparse and Discrete Interpretability for Neural Networkscited by 45
  30. 2021Sparse Interventions in Language Models with Differentiable Maskingcited by 34
  31. 2023Discovering Variable Binding Circuitry with Desideratacited by 32
  32. 2023FIND: A Function Description Benchmark for Evaluating Interpretability Methodscited by 32
  33. 2022Disentangled Explanations of Neural Network Predictions by Finding Relevant Subspacescited by 25
  34. 2023ALMANACS: A Simulatability Benchmark for Language Model Explainabilitycited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 20; 2025: 31; 2026: 26.

77 papers.

Sort:

By area:

  1. 2024Mechanistic Interpretability for AI Safety -- A Reviewcited by 415
  2. 2024ReFT: Representation Finetuning for Language Modelscited by 186
  3. 2024Llama Scope: Extracting Millions of Features from Llama-3.1-8B with Sparse Autoencoderscited by 133
  4. 2025SAEBench: A Comprehensive Benchmark for Sparse Autoencoders in Language Model Interpretabilitycited by 118
  5. 2024A Practical Review of Mechanistic Interpretability for Transformer-Based Language Modelscited by 113
  6. 2025Are Sparse Autoencoders Useful? A Case Study in Sparse Probingcited by 109
  7. 2024A Primer on the Inner Workings of Transformer-based Language Modelscited by 83
  8. 2024On Large Language Model Continual Unlearningcited by 51
  9. 2024Knowledge Mechanisms in Large Language Models: A Survey and Perspectivecited by 50
  10. 2025MIB: A Mechanistic Interpretability Benchmarkcited by 49
  11. 2025Do Language Models Use Their Depth Efficiently?cited by 48
  12. 2024Recurrent Neural Networks Learn to Store and Generate Sequences using Non-Linear Representationscited by 45
38 more
  1. 2024Estimating Knowledge in Large Language Models Without Generating a Single Tokencited by 39
  2. 2025Enhancing Automated Interpretability with Output-Centric Feature Descriptionscited by 39
  3. 2025A Survey on Mechanistic Interpretability for Multi-Modal Foundation Modelscited by 34
  4. 2024Intrinsic Test of Unlearning Using Parametric Knowledge Tracescited by 28
  5. 2024InterpBench: Semi-Synthetic Transformers for Evaluating Mechanistic Interpretability Techniquescited by 24
  6. 2024The Quest for the Right Mediator: Surveying Mechanistic Interpretability Through the Lens of Causal Mediation Analysiscited by 24
  7. 2026Manifold Steering Reveals the Shared Geometry of Neural Network Representation and Behaviorcited by 22
  8. 2024Residual Stream Analysis with Multi-Layer SAEscited by 18
  9. 2025Internal Causal Mechanisms Robustly Predict Language Model Out-of-Distribution Behaviorscited by 11
  10. 2025Evaluating Adversarial Robustness of Concept Representations in Sparse Autoencoderscited by 11
  11. 2025Precise In-Parameter Concept Erasure in Large Language Modelscited by 11
  12. 2025SALT: Steering Activations towards Leakage-free Thinking in Chain of Thoughtcited by 11
  13. 2025How Causal Abstraction Underpins Computational Explanationcited by 10
  14. 2025RepIt: Steering Language Models with Concept-Specific Refusal Vectorscited by 10
  15. 2026From Directions to Regions: Decomposing Activations in Language Models via Local Geometrycited by 10
  16. 2026Sanity Checks for Sparse Autoencoders: Do SAEs Beat Random Baselines?cited by 10
  17. 2024How Reliable are Causal Probing Interventions?cited by 9
  18. 2024Characterizing the Role of Similarity in the Property Inferences of Language Modelscited by 9
  19. 2024Adaptive Sparse Allocation with Mutual Choice & Feature Choice Sparse Autoencoderscited by 9
  20. 2025HyperDAS: Towards Automating Mechanistic Interpretability with Hypernetworkscited by 8
  21. 2026AudioSAE: Towards Understanding of Audio-Processing Models with Sparse AutoEncoderscited by 8
  22. 2025SAFE: A Sparse Autoencoder-Based Framework for Robust Query Enrichment and Hallucination Mitigation in LLMscited by 7
  23. 2025Decomposing Representation Space into Interpretable Subspaces with Unsupervised Learningcited by 7
  24. 2025OrtSAE: Orthogonal Sparse Autoencoders Uncover Atomic Featurescited by 7
  25. 2025Addressing divergent representations from causal interventions on neural networkscited by 6
  26. 2026From Atoms to Trees: Building a Structured Feature Forest with Hierarchical Sparse Autoencoderscited by 6
  27. 2025From Isolation to Entanglement: When Do Interpretability Methods Identify and Disentangle Known Concepts?cited by 5
  28. 2024TracrBench: Generating Interpretability Testbeds with Large Language Modelscited by 4
  29. 2025Learning Distribution-Wise Control in Representation Space for Language Modelscited by 4
  30. 2025Measuring Sparse Autoencoder Feature Sensitivitycited by 4
  31. 2025Constructing Interpretable Features from Compositional Neuron Groupscited by 3
  32. 2025Enhancing Chain-of-Thought Reasoning with Critical Representation Fine-tuningcited by 3
  33. 2026Indications of Belief-Guided Agency and Meta-Cognitive Monitoring in Large Language Modelscited by 3
  34. 2026Are Sparse Autoencoder Benchmarks Reliable?cited by 3
  35. 2024SAGE: Scalable Ground Truth Evaluations for Large Sparse Autoencoderscited by 2
  36. 2025Are Sparse Autoencoders Useful for Java Function Bug Detection?cited by 2
  37. 2025InverseScope: Scalable Activation Inversion for Interpreting Large Language Modelscited by 2
  38. 2025Steering Information Utility in Key-Value Memory for Language Model Post-Trainingcited by 2

50 most cited of 77 All 77 (JSON)