Interpreting Language Models Through Concept Descriptions: A Survey
Not on the map. Why?
Abstract (arXiv)
Understanding the decision-making processes of neural networks is a central goal of mechanistic interpretability. In the context of Large Language Models (LLMs), this involves uncovering the underlying mechanisms and identifying the roles of individual model components such as neurons and attention heads, as well as model abstractions such as the learned sparse features extracted by Sparse Autoencoders (SAEs). A rapidly growing line of work tackles this challenge by using powerful generator models to produce open-vocabulary, natural language concept descriptions for these components. In this paper, we provide the first survey of the emerging field of concept descriptions for model components and abstractions. We chart the key methods for generating these descriptions, the evolving landscape of automated and human metrics for evaluating them, and the datasets that underpin this research. Our synthesis reveals a growing demand for more rigorous, causal evaluation. By outlining the state of the art and identifying key challenges, this survey provides a roadmap for future research toward making models more transparent.
Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 52 of its 79 references that are in the map. First posted on arXiv 1 Oct 2025.
Builds on
52 of 79 references.
References · 52
- 2023Sparse Autoencoders Find Highly Interpretable Features in Language Modelscited by 1,359
- 2017Interpretability Beyond Feature Attribution: Quantitative Testing with Concept Activation Vectors (TCAV)cited by 1,309
- 2022Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 smallcited by 1,130
- 2023Language Models Don't Always Say What They Think: Unfaithful Explanations in Chain-of-Thought Promptingcited by 1,081
- 2019What Does BERT Look At? An Analysis of BERT's Attentioncited by 994
- 2019Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Prunedcited by 844
- 2021Probing Classifiers: Promises, Shortcomings, and Advancescited by 831
- 2023Measuring Faithfulness in Chain-of-Thought Reasoningcited by 668
- 2020Towards Faithfully Interpretable NLP Systems: How should we define and evaluate faithfulness?cited by 590
- 2022Transformer Feed-Forward Layers Build Predictions by Promoting Concepts in the Vocabulary Spacecited by 522
- 2018e-SNLI: Natural Language Inference with Natural Language Explanationscited by 468
- 2018What you can cram into a single vector: Probing sentence embeddings for linguistic propertiescited by 452
38 more
- 2024Sparse Feature Circuits: Discovering and Editing Interpretable Causal Graphs in Language Modelscited by 412
- 2024Jumping Ahead: Improving Reconstruction Fidelity with JumpReLU Sparse Autoencoderscited by 283
- 2016Rationalizing Neural Predictionscited by 281
- 2025AxBench: Steering LLMs? Even Simple Baselines Outperform Sparse Autoencoderscited by 199
- 2020Compositional Explanations of Neuronscited by 172
- 2022CLIP-Dissect: Automatic Description of Neuron Representations in Deep Vision Networkscited by 144
- 2024Llama Scope: Extracting Millions of Features from Llama-3.1-8B with Sparse Autoencoderscited by 133
- 2021Reframing Human-AI Collaboration for Generating Free-Text Explanationscited by 125
- 2024Are self-explanations from Large Language Models faithful?cited by 101
- 2021An Interpretability Illusion for BERTcited by 94
- 2023Do Models Explain Themselves? Counterfactual Simulatability of Natural Language Explanationscited by 85
- 2021Neuron-level Interpretation of Deep NLP Models: A Surveycited by 81
- 2024Concept Induction: Analyzing Unstructured Text with High-Level Concepts Using LLooMcited by 77
- 2023On Measuring Faithfulness or Self-consistency of Natural Language Explanationscited by 71
- 2023Explaining black box text modules in natural language with language modelscited by 61
- 2023Goal-Driven Explainable Clustering via Language Descriptionscited by 58
- 2022Discovering Latent Concepts Learned in BERTcited by 56
- 2022CEBaB: Estimating the Causal Effects of Real-World Concepts on NLP Model Behaviorcited by 54
- 2024Separating Tongue from Thought: Activation Patching Reveals Language-Agnostic Concept Representations in Transformerscited by 42
- 2025Enhancing Automated Interpretability with Output-Centric Feature Descriptionscited by 39
- 2023FIND: A Function Description Benchmark for Evaluating Interpretability Methodscited by 32
- 2023Labeling Neural Representations with Inverse Recognitioncited by 31
- 2025Automated Interpretability Metrics Do Not Distinguish Trained and Random Transformerscited by 30
- 2024Chain-of-Thought Unfaithfulness as Disguised Accuracycited by 27
- 2025We Can't Understand AI Using our Existing Vocabularycited by 25
- 2024CoSy: Evaluating Textual Explanations of Neuronscited by 17
- 2024Inferring Functionality of Attention Heads from their Parameterscited by 17
- 2024Decoding Dark Matter: Specialized Sparse Autoencoders for Interpreting Rare Concepts in Foundation Modelscited by 15
- 2023From Neural Activations to Concepts: A Survey on Explaining Concepts in Neural Networkscited by 13
- 2024Latent Concept-based Explanation of NLP Modelscited by 13
- 2025FADE: Why Bad Descriptions Happen to Good Featurescited by 13
- 2023The Importance of Prompt Tuning for Automated Neuron Explanationscited by 12
- 2023Self-Consistency of Large Language Models under Ambiguitycited by 12
- 2024Interpreting Context Look-ups in Transformers: Investigating Attention-MLP Interactionscited by 11
- 2025The Atlas of In-Context Learning: How Attention Heads Shape In-Context Retrieval Augmentationcited by 11
- 2024Evaluating Readability and Faithfulness of Concept-based Explanationscited by 10
- 2025ConSim: Measuring Concept-Based Explanations' Effectiveness with Automated Simulatabilitycited by 9
- 2025Capturing Polysemanticity with PRISM: A Multi-Concept Feature Description Frameworkcited by 9
50 most cited of 52 All 52 (JSON)
Built on by
Citing papers per year: 2026: 3.
3 papers.