# Beyond Top Activations: Efficient and Reliable Crowdsourced Evaluation of Automated Interpretability

arXiv: https://arxiv.org/abs/2506.07985 · HTML: https://mlsanity.com/paper/2506.07985/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Circuits and mechanistic analysis, one of 13 directions in Mechanistic interpretability.
- Builds on 23 of its 30 references that are in the map.
- First posted on arXiv 9 Jun 2025.

## Placed in

- Mechanistic interpretability / [Circuits and mechanistic analysis](https://mlsanity.com/mechinterp/circuits-and-mechanistic-analysis/)

## Builds on (most cited first, 23 of 23)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [GPT-4o System Card](https://mlsanity.com/paper/2410.21276/) (2024)
- [Sigmoid Loss for Language Image Pre-Training](https://mlsanity.com/paper/2303.15343/) (2023)
- [LLaVA-OneVision: Easy Visual Task Transfer](https://mlsanity.com/paper/2408.03326/) (2024)
- [SigLIP 2: Multilingual Vision-Language Encoders with Improved Semantic Understanding, Localization, and Dense Features](https://mlsanity.com/paper/2502.14786/) (2025)
- [Network Dissection: Quantifying Interpretability of Deep Visual Representations](https://mlsanity.com/paper/1704.05796/) (2017)
- [Understanding the Role of Individual Units in a Deep Neural Network](https://mlsanity.com/paper/2009.05041/) (2020)
- [CogVLM2: Visual Language Models for Image and Video Understanding](https://mlsanity.com/paper/2408.16500/) (2024)
- [Compositional Explanations of Neurons](https://mlsanity.com/paper/2006.14032/) (2020)
- [CLIP-Dissect: Automatic Description of Neuron Representations in Deep Vision Networks](https://mlsanity.com/paper/2204.10965/) (2022)
- [Natural Language Descriptions of Deep Visual Features](https://mlsanity.com/paper/2201.11114/) (2022)
- [A Multimodal Automated Interpretability Agent](https://mlsanity.com/paper/2404.14394/) (2024)
- [Enhancing Automated Interpretability with Output-Centric Feature Descriptions](https://mlsanity.com/paper/2501.08319/) (2025)
- [Interpreting the Second-Order Effects of Neurons in CLIP](https://mlsanity.com/paper/2406.04341/) (2024)
- [Identifying Interpretable Subspaces in Image Representations](https://mlsanity.com/paper/2307.10504/) (2023)
- [Labeling Neural Representations with Inverse Recognition](https://mlsanity.com/paper/2311.13594/) (2023)
- [Linear Explanations for Individual Neurons](https://mlsanity.com/paper/2405.06855/) (2024)
- [Scale Alone Does not Improve Mechanistic Interpretability in Vision Models](https://mlsanity.com/paper/2307.05471/) (2023)
- [Towards a fuller understanding of neurons with Clustered Compositional Explanations](https://mlsanity.com/paper/2310.18443/) (2023)
- [Interpreting Neurons in Deep Vision Networks with Language Models](https://mlsanity.com/paper/2403.13771/) (2024)
- [Evaluating Neuron Explanations: A Unified Framework with Sanity Checks](https://mlsanity.com/paper/2506.05774/) (2025)

## Abstract (arXiv)

Interpreting individual neurons or directions in activation space is an important topic in mechanistic interpretability. Numerous automated interpretability methods have been proposed to generate such explanations, but it remains unclear how reliable these explanations are, and which methods produce the most accurate descriptions. While crowd-sourced evaluations are commonly used, existing pipelines are noisy, costly, and typically assess only the highest-activating inputs, leading to unreliable results. In this paper, we introduce two techniques to enable cost-effective and accurate crowdsourced evaluation of automated interpretability methods beyond top activating inputs. First, we propose Model-Guided Importance Sampling (MG-IS) to select the most informative inputs to show human raters. In our experiments, we show this reduces the number of inputs needed to reach the same evaluation accuracy by ~13x. Second, we address label noise in crowd-sourced ratings through Bayesian Rating Aggregation (BRAgg), which allows us to reduce the number of ratings per input required to overcome noise by ~3x. Together, these techniques reduce the evaluation cost by ~40x, making large-scale evaluation feasible. Finally, we use our methods to conduct a large scale crowd-sourced study comparing recent automated interpretability methods for vision networks.