mlsanityData to 14 Oct 2026

Evaluating Neuron Explanations: A Unified Framework with Sanity Checks

arXiv 2506.05774 ↗First posted 6 Jun 2025cs.LGcited by 4

On the map: Mechanistic interpretability → Circuits and mechanistic analysis

Abstract (arXiv)

Understanding the function of individual units in a neural network is an important building block for mechanistic interpretability. This is often done by generating a simple text explanation of the behavior of individual neurons or units. For these explanations to be useful, we must understand how reliable and truthful they are. In this work we unify many existing explanation evaluation methods under one mathematical framework. This allows us to compare existing evaluation metrics, understand the evaluation pipeline with increased clarity and apply existing statistical methods on the evaluation. In addition, we propose two simple sanity checks on the evaluation metrics and show that many commonly used metrics fail these tests and do not change their score after massive changes to the concept labels. Based on our experimental and theoretical results, we propose guidelines that future evaluations should follow and identify a set of reliable evaluation metrics.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Placed in Circuits and mechanistic analysis, one of 13 directions in Mechanistic interpretability. Builds on 40 of its 57 references that are in the map. First posted on arXiv 6 Jun 2025.

Builds on

40 of 57 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023Scale Alone Does not Improve Mechanistic Interpretability in Vision Models11 Jul

Other references · 39

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  3. 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,109
  4. 2023Sparse Autoencoders Find Highly Interpretable Features in Language Modelscited by 1,420
  5. 2017Interpretability Beyond Feature Attribution: Quantitative Testing with Concept Activation Vectors (TCAV)cited by 1,319
  6. 2018Sanity Checks for Saliency Mapscited by 1,156
  7. 2020Concept Bottleneck Modelscited by 1,032
  8. 2018RISE: Randomized Input Sampling for Explanation of Black-box Modelscited by 838
  9. 2017Network Dissection: Quantifying Interpretability of Deep Visual Representationscited by 708
  10. 2018A Benchmark for Interpretability Methods in Deep Neural Networkscited by 476
  11. 2021Interpretable Machine Learning: Fundamental Principles and 10 Grand Challengescited by 367
  12. 2023Finding Neurons in a Haystack: Case Studies with Sparse Probingcited by 334
27 more
  1. 2022Post-hoc Concept Bottleneck Modelscited by 293
  2. 2023Label-Free Concept Bottleneck Modelscited by 285
  3. 2020Understanding the Role of Individual Units in a Deep Neural Networkcited by 255
  4. 2022Extracting Latent Steering Vectors from Pretrained Language Modelscited by 225
  5. 2020Evaluating and Aggregating Feature-based Model Explanationscited by 188
  6. 2014Object Detectors Emerge in Deep Scene CNNscited by 179
  7. 2022CRAFT: Concept Recursive Activation FacTorization for Explainabilitycited by 178
  8. 2020Compositional Explanations of Neuronscited by 172
  9. 2022CLIP-Dissect: Automatic Description of Neuron Representations in Deep Vision Networkscited by 147
  10. 2022Natural Language Descriptions of Deep Visual Featurescited by 137
  11. 2020Invertible Concept-based Explanations for CNN Models with Non-negative Concept Activation Vectorscited by 120
  12. 2021An Interpretability Illusion for BERTcited by 97
  13. 2021Look at the Variance! Efficient Black-box Explanations with Sobol-based Sensitivity Analysiscited by 65
  14. 2024VLG-CBM: Training Concept Bottleneck Models with Vision-Language Guidancecited by 65
  15. 2024A Multimodal Automated Interpretability Agentcited by 63
  16. 2023Explaining black box text modules in natural language with language modelscited by 62
  17. 2023Rigorously Assessing Natural Language Explanations of Neuronscited by 45
  18. 2025Enhancing Automated Interpretability with Output-Centric Feature Descriptionscited by 41
  19. 2024Interpreting the Second-Order Effects of Neurons in CLIPcited by 36
  20. 2023FIND: A Function Description Benchmark for Evaluating Interpretability Methodscited by 32
  21. 2023Labeling Neural Representations with Inverse Recognitioncited by 32
  22. 2024Linear Explanations for Individual Neuronscited by 19
  23. 2024CoSy: Evaluating Textual Explanations of Neuronscited by 17
  24. 2023Towards a fuller understanding of neurons with Clustered Compositional Explanationscited by 15
  25. 2023The Importance of Prompt Tuning for Automated Neuron Explanationscited by 12
  26. 2024Interpreting Neurons in Deep Vision Networks with Language Modelscited by 10
  27. 2024Benchmarking XAI Explanations with Human-Aligned Evaluationscited by 8

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 3; 2026: 1.

4 papers.

  1. 2025CorrSteer: Generation-Time LLM Steering via Correlated Sparse Autoencoder Featurescited by 12
  2. 2025Faithful and Stable Neuron Explanations for Trustworthy Mechanistic Interpretabilitycited by 3
  3. 2026A Geometric View for Understanding Concept Learning and Neuron Interpretation in Sparse Autoencoderscited by 1
  4. 2025Beyond Top Activations: Efficient and Reliable Crowdsourced Evaluation of Automated Interpretabilitycited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Evaluating Explanations: An Explanatory Virtues Framework for Mechanistic Interpretability -- The Strange Science Part I.ii
  2. 2023N2G: A Scalable Approach for Quantifying Interpretable Neuron Representations in Large Language Models
  3. 2026NeuronScope: A Multi-Agent Framework for Explaining Polysemantic Neurons in Language Models
  4. 2023Understanding CNN Hidden Neuron Activations Using Structured Background Knowledge and Deductive Reasoning
  5. 2024Choosing the right basis for interpretability: Psychophysical comparison between neuron-based and dictionary-based representations
  6. 2023Identifying Interpretable Visual Features in Artificial and Biological Neural Systems
  7. 2025A Mathematical Philosophy of Explanations in Mechanistic Interpretability -- The Strange Science Part I.i
  8. 2024LLM-assisted Concept Discovery: Automatically Identifying and Explaining Neuron Functions