# ConceptViz: A Visual Analytics Approach for Exploring Concepts in Large Language Models

arXiv: https://arxiv.org/abs/2509.20376 · HTML: https://mlsanity.com/paper/2509.20376/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in SAE features and their applications, one of 13 directions in Mechanistic interpretability.
- Builds on 52 of its 73 references that are in the map.
- First posted on arXiv 20 Sep 2025.

## Placed in

- Mechanistic interpretability / [SAE features and their applications](https://mlsanity.com/mechinterp/sae-features-and-their-applications/)

## Builds on (most cited first, 30 of 52)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Locating and Editing Factual Associations in GPT](https://mlsanity.com/paper/2202.05262/) (2022)
- [Gemma 2: Improving Open Language Models at a Practical Size](https://mlsanity.com/paper/2408.00118/) (2024)
- [Sparse Autoencoders Find Highly Interpretable Features in Language Models](https://mlsanity.com/paper/2309.08600/) (2023)
- [Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 small](https://mlsanity.com/paper/2211.00593/) (2022)
- [What Does BERT Look At? An Analysis of BERT's Attention](https://mlsanity.com/paper/1906.04341/) (2019)
- [Toy Models of Superposition](https://mlsanity.com/paper/2209.10652/) (2022)
- [In-context Learning and Induction Heads](https://mlsanity.com/paper/2209.11895/) (2022)
- [Attention is not not Explanation](https://mlsanity.com/paper/1908.04626/) (2019)
- [Scaling and evaluating sparse autoencoders](https://mlsanity.com/paper/2406.04093/) (2024)
- [Sparse Feature Circuits: Discovering and Editing Interpretable Causal Graphs in Language Models](https://mlsanity.com/paper/2403.19647/) (2024)
- [Gemma Scope: Open Sparse Autoencoders Everywhere All At Once on Gemma 2](https://mlsanity.com/paper/2408.05147/) (2024)
- [Jumping Ahead: Improving Reconstruction Fidelity with JumpReLU Sparse Autoencoders](https://mlsanity.com/paper/2407.14435/) (2024)
- [A Multiscale Visualization of Attention in the Transformer Model](https://mlsanity.com/paper/1906.05714/) (2019)
- [Do Llamas Work in English? On the Latent Language of Multilingual Transformers](https://mlsanity.com/paper/2402.10588/) (2024)
- [Improving Dictionary Learning with Gated Sparse Autoencoders](https://mlsanity.com/paper/2404.16014/) (2024)
- [A is for Absorption: Studying Feature Splitting and Absorption in Sparse Autoencoders](https://mlsanity.com/paper/2409.14507/) (2024)
- [A Diagnostic Study of Explainability Techniques for Text Classification](https://mlsanity.com/paper/2009.13295/) (2020)
- [Llama Scope: Extracting Millions of Features from Llama-3.1-8B with Sparse Autoencoders](https://mlsanity.com/paper/2410.20526/) (2024)
- [SAEBench: A Comprehensive Benchmark for Sparse Autoencoders in Language Model Interpretability](https://mlsanity.com/paper/2503.09532/) (2025)
- [Visual Analytics in Deep Learning: An Interrogative Survey for the Next Frontiers](https://mlsanity.com/paper/1801.06889/) (2018)
- [Automatically Interpreting Millions of Features in Large Language Models](https://mlsanity.com/paper/2410.13928/) (2024)
- [Information Flow Routes: Automatically Interpreting Language Models at Scale](https://mlsanity.com/paper/2403.00824/) (2024)
- [Discover-then-Name: Task-Agnostic Concept Bottlenecks via Automated Concept Discovery](https://mlsanity.com/paper/2407.14499/) (2024)
- [A Survey on Sparse Autoencoders: Interpreting the Internal Mechanisms of Large Language Models](https://mlsanity.com/paper/2503.05613/) (2025)
- [Interactive and Visual Prompt Engineering for Ad-hoc Task Adaptation with Large Language Models](https://mlsanity.com/paper/2208.07852/) (2022)
- [Seq2Seq-Vis: A Visual Debugging Tool for Sequence-to-Sequence Models](https://mlsanity.com/paper/1804.09299/) (2018)
- [Sparse autoencoders reveal selective remapping of visual concepts during adaptation](https://mlsanity.com/paper/2412.05276/) (2024)
- [Applying sparse autoencoders to unlearn knowledge in language models](https://mlsanity.com/paper/2410.19278/) (2024)
- [PromptMagician: Interactive Prompt Engineering for Text-to-Image Creation](https://mlsanity.com/paper/2307.09036/) (2023)

## Built on by (most cited first, 1 of 1)

- [VisualScratchpad: Inference-time Visual Concepts Analysis in Vision Language Models](https://mlsanity.com/paper/2603.07335/) (2026)

## Abstract (arXiv)

Large language models (LLMs) have achieved remarkable performance across a wide range of natural language tasks. Understanding how LLMs internally represent knowledge remains a significant challenge. Despite Sparse Autoencoders (SAEs) have emerged as a promising technique for extracting interpretable features from LLMs, SAE features do not inherently align with human-understandable concepts, making their interpretation cumbersome and labor-intensive. To bridge the gap between SAE features and human concepts, we present ConceptViz, a visual analytics system designed for exploring concepts in LLMs. ConceptViz implements a novel dentification => Interpretation => Validation pipeline, enabling users to query SAEs using concepts of interest, interactively explore concept-to-feature alignments, and validate the correspondences through model behavior verification. We demonstrate the effectiveness of ConceptViz through two usage scenarios and a user study. Our results show that ConceptViz enhances interpretability research by streamlining the discovery and validation of meaningful concept representations in LLMs, ultimately aiding researchers in building more accurate mental models of LLM features. Our code and user guide are publicly available at https://github.com/Happy-Hippo209/ConceptViz.