mlsanityData to 14 Oct 2026

ConceptViz: A Visual Analytics Approach for Exploring Concepts in Large Language Models

arXiv 2509.20376 ↗First posted 20 Sep 2025cs.CL cs.AIcited by 1

On the map: Mechanistic interpretability → SAE features and their applications

Abstract (arXiv)

Large language models (LLMs) have achieved remarkable performance across a wide range of natural language tasks. Understanding how LLMs internally represent knowledge remains a significant challenge. Despite Sparse Autoencoders (SAEs) have emerged as a promising technique for extracting interpretable features from LLMs, SAE features do not inherently align with human-understandable concepts, making their interpretation cumbersome and labor-intensive. To bridge the gap between SAE features and human concepts, we present ConceptViz, a visual analytics system designed for exploring concepts in LLMs. ConceptViz implements a novel dentification => Interpretation => Validation pipeline, enabling users to query SAEs using concepts of interest, interactively explore concept-to-feature alignments, and validate the correspondences through model behavior verification. We demonstrate the effectiveness of ConceptViz through two usage scenarios and a user study. Our results show that ConceptViz enhances interpretability research by streamlining the discovery and validation of meaningful concept representations in LLMs, ultimately aiding researchers in building more accurate mental models of LLM features. Our code and user guide are publicly available at https://github.com/Happy-Hippo209/ConceptViz.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in SAE features and their applications, one of 13 directions in Mechanistic interpretability. Builds on 52 of its 73 references that are in the map. First posted on arXiv 20 Sep 2025.

Builds on

52 of 73 references.

Closest ancestors · 13

Same direction, newest first

  1. 2025Beyond Input Activations: Identifying Influential Latents by Gradient Sparse Autoencoders12 May
  2. 2025SAEBench: A Comprehensive Benchmark for Sparse Autoencoders in Language Model Interpretability12 Mar
  3. 2025A Survey on Sparse Autoencoders: Interpreting the Internal Mechanisms of Large Language Models7 Mar
  4. 2025Interpreting and Steering LLMs with Mutual Information-based Explanations on Sparse Autoencoders21 Feb
  5. 2024Llama Scope: Extracting Millions of Features from Llama-3.1-8B with Sparse Autoencoders27 Oct
  6. 2024Applying sparse autoencoders to unlearn knowledge in language models25 Oct
  7. 2024Automatically Interpreting Millions of Features in Large Language Models17 Oct
  8. 2024A is for Absorption: Studying Feature Splitting and Absorption in Sparse Autoencoders22 Sep
  9. 2024Measuring Progress in Dictionary Learning for Language Model Interpretability with Board Game Models31 Jul
  10. 2024Jumping Ahead: Improving Reconstruction Fidelity with JumpReLU Sparse Autoencoders19 Jul
  11. 2024Scaling and evaluating sparse autoencoders6 Jun
  12. 2024Improving Dictionary Learning with Gated Sparse Autoencoders24 Apr
1 more
  1. 2023Sparse Autoencoders Find Highly Interpretable Features in Language Models15 Sep

Other references · 39

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2022Locating and Editing Factual Associations in GPTcited by 2,691
  3. 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,998
  4. 2022Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 smallcited by 1,165
  5. 2019What Does BERT Look At? An Analysis of BERT's Attentioncited by 1,003
  6. 2022Toy Models of Superpositioncited by 971
  7. 2022In-context Learning and Induction Headscited by 964
  8. 2019Attention is not not Explanationcited by 636
  9. 2024Sparse Feature Circuits: Discovering and Editing Interpretable Causal Graphs in Language Modelscited by 435
  10. 2024Gemma Scope: Open Sparse Autoencoders Everywhere All At Once on Gemma 2cited by 394
  11. 2019A Multiscale Visualization of Attention in the Transformer Modelcited by 292
  12. 2024Do Llamas Work in English? On the Latent Language of Multilingual Transformerscited by 281
27 more
  1. 2020A Diagnostic Study of Explainability Techniques for Text Classificationcited by 159
  2. 2018Visual Analytics in Deep Learning: An Interrogative Survey for the Next Frontierscited by 117
  3. 2024Information Flow Routes: Automatically Interpreting Language Models at Scalecited by 95
  4. 2024Discover-then-Name: Task-Agnostic Concept Bottlenecks via Automated Concept Discoverycited by 91
  5. 2022Interactive and Visual Prompt Engineering for Ad-hoc Task Adaptation with Large Language Modelscited by 70
  6. 2018Seq2Seq-Vis: A Visual Debugging Tool for Sequence-to-Sequence Modelscited by 69
  7. 2024Sparse autoencoders reveal selective remapping of visual concepts during adaptationcited by 68
  8. 2023PromptMagician: Interactive Prompt Engineering for Text-to-Image Creationcited by 58
  9. 2020CNN Explainer: Learning Convolutional Neural Networks with Interactive Visualizationcited by 57
  10. 2023From Language Modeling to Instruction Following: Understanding the Behavior Shift in LLMs after Instruction Tuningcited by 56
  11. 2023AttentionViz: A Global View of Transformer Attentioncited by 50
  12. 2020Attention Flows: Analyzing and Comparing Attention Mechanisms in Language Modelscited by 49
  13. 2023Neuron-Level Knowledge Attribution in Large Language Modelscited by 37
  14. 2017Towards Better Analysis of Machine Learning Models: A Visual Analytics Perspectivecited by 36
  15. 2024Evaluating Open-Source Sparse Autoencoders on Disentangling Factual Knowledge in GPT-2 Smallcited by 34
  16. 2023Foundation Models Meet Visualizations: Challenges and Opportunitiescited by 31
  17. 2021Dodrio: Exploring Transformer Models with Interactive Visualizationcited by 28
  18. 2022ConceptExplainer: Interactive Explanation for Deep Neural Networks from a Concept Perspectivecited by 18
  19. 2021Human-in-the-loop Extraction of Interpretable Concepts in Deep Learning Modelscited by 17
  20. 2021LMdiff: A Visual Diff Tool to Compare Language Modelscited by 17
  21. 2024LM Transparency Tool: Interactive Tool for Analyzing Transformer Language Modelscited by 16
  22. 2024Transformer Explainer: Learning LLM Transformers with Interactive Visual Explanation and Experimentationcited by 10
  23. 2024KnowledgeVIS: Interpreting Language Models by Comparing Fill-in-the-Blank Promptscited by 7
  24. 2024iScore: Visual Analytics for Interpreting How Language Models Automatically Score Summariescited by 6
  25. 2023N2G: A Scalable Approach for Quantifying Interpretable Neuron Representations in Large Language Modelscited by 4
  26. 2024Compress and Compare: Interactively Evaluating Efficiency and Behavior Across ML Model Compression Experimentscited by 4
  27. 2024Dynamic Color Assignment for Hierarchical Datacited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026VisualScratchpad: Inference-time Visual Concepts Analysis in Vision Language Modelscited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2026Navigating the Concept Space of Language Models
  2. 2026Concept Component Analysis: A Principled Approach for Concept Extraction in LLMs
  3. 2025Interpreting Language Models Through Concept Descriptions: A Survey
  4. 2026Beyond Linear Concepts: Discovering and Aligning Non-Linear Concept Manifolds in Large Language Models
  5. 2024Crafting Large Language Models for Enhanced Interpretability
  6. 2024Exploring Concept Depth: How Large Language Models Acquire Knowledge and Concept at Different Layers?
  7. 2025Concept Layers: Enhancing Interpretability and Intervenability via LLM Conceptualization
  8. 2025Visual Exploration of Feature Relationships in Sparse Autoencoders with Curated Concepts