# Codebook Features: Sparse and Discrete Interpretability for Neural Networks

arXiv: https://arxiv.org/abs/2310.17230 · HTML: https://mlsanity.com/paper/2310.17230/

## Key facts

- Cited by 48 papers in the map (data through 14 Oct 2026).
- Builds on 45 of its 86 references that are in the map.
- First posted on arXiv 26 Oct 2023.

## Builds on (most cited first, 30 of 45)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Explaining and Harnessing Adversarial Examples](https://mlsanity.com/paper/1412.6572/) (2014)
- [Neural Discrete Representation Learning](https://mlsanity.com/paper/1711.00937/) (2017)
- [On the Opportunities and Risks of Foundation Models](https://mlsanity.com/paper/2108.07258/) (2021)
- [Pointer Sentinel Mixture Models](https://mlsanity.com/paper/1609.07843/) (2016)
- [Locating and Editing Factual Associations in GPT](https://mlsanity.com/paper/2202.05262/) (2022)
- [Billion-scale similarity search with GPUs](https://mlsanity.com/paper/1702.08734/) (2017)
- [Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computation](https://mlsanity.com/paper/1308.3432/) (2013)
- [Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling](https://mlsanity.com/paper/2304.01373/) (2023)
- [Sparse Autoencoders Find Highly Interpretable Features in Language Models](https://mlsanity.com/paper/2309.08600/) (2023)
- [Understanding intermediate layers using linear classifier probes](https://mlsanity.com/paper/1610.01644/) (2016)
- [Transformer Feed-Forward Layers Are Key-Value Memories](https://mlsanity.com/paper/2012.14913/) (2020)
- [Interpretability Beyond Feature Attribution: Quantitative Testing with Concept Activation Vectors (TCAV)](https://mlsanity.com/paper/1711.11279/) (2017)
- [Interpretability in the Wild: a Circuit for Indirect Object Identification in GPT-2 small](https://mlsanity.com/paper/2211.00593/) (2022)
- [Concept Bottleneck Models](https://mlsanity.com/paper/2007.04612/) (2020)
- [What Does BERT Look At? An Analysis of BERT's Attention](https://mlsanity.com/paper/1906.04341/) (2019)
- [Mass-Editing Memory in a Transformer](https://mlsanity.com/paper/2210.07229/) (2022)
- [Steering Language Models With Activation Engineering](https://mlsanity.com/paper/2308.10248/) (2023)
- [A Primer in BERTology: What we know about how BERT works](https://mlsanity.com/paper/2002.12327/) (2020)
- [CTRL: A Conditional Transformer Language Model for Controllable Generation](https://mlsanity.com/paper/1909.05858/) (2019)
- [GPT-NeoX-20B: An Open-Source Autoregressive Language Model](https://mlsanity.com/paper/2204.06745/) (2022)
- [Vector-quantized Image Modeling with Improved VQGAN](https://mlsanity.com/paper/2110.04627/) (2021)
- [Fast Model Editing at Scale](https://mlsanity.com/paper/2110.11309/) (2021)
- [Google's Multilingual Neural Machine Translation System: Enabling Zero-Shot Translation](https://mlsanity.com/paper/1611.04558/) (2016)
- [Causal Abstractions of Neural Networks](https://mlsanity.com/paper/2106.02997/) (2021)
- [Neural Module Networks](https://mlsanity.com/paper/1511.02799/) (2015)
- [The Neuro-Symbolic Concept Learner: Interpreting Scenes, Words, and Sentences From Natural Supervision](https://mlsanity.com/paper/1904.12584/) (2019)
- [Post-hoc Concept Bottleneck Models](https://mlsanity.com/paper/2205.15480/) (2022)
- [Understanding the Role of Individual Units in a Deep Neural Network](https://mlsanity.com/paper/2009.05041/) (2020)

## Built on by (most cited first, 30 of 48)

- [Scaling Monosemanticity: Extracting Interpretable Features from Claude 3 Sonnet](https://mlsanity.com/paper/2605.29358/) (2026)
- [Improving Dictionary Learning with Gated Sparse Autoencoders](https://mlsanity.com/paper/2404.16014/) (2024)
- [A Survey on Uncertainty Quantification of Large Language Models: Taxonomy, Open Research Challenges, and Future Directions](https://mlsanity.com/paper/2412.05563/) (2024)
- [Towards Principled Evaluations of Sparse Autoencoders for Interpretability and Control](https://mlsanity.com/paper/2405.08366/) (2024)
- [RAVEL: Evaluating Interpretability Methods on Disentangling Language Model Representations](https://mlsanity.com/paper/2402.17700/) (2024)
- [Archetypal SAE: Adaptive and Stable Dictionary Learning for Concept Extraction in Large Vision Models](https://mlsanity.com/paper/2502.12892/) (2025)
- [Identifying Functionally Important Features with End-to-End Sparse Dictionary Learning](https://mlsanity.com/paper/2405.12241/) (2024)
- [Universal Sparse Autoencoders: Interpretable Cross-Model Concept Alignment](https://mlsanity.com/paper/2502.03714/) (2025)
- [Knowledge Mechanisms in Large Language Models: A Survey and Perspective](https://mlsanity.com/paper/2407.15017/) (2024)
- [Reasoning about Uncertainty: Do Reasoning Models Know When They Don't Know?](https://mlsanity.com/paper/2506.18183/) (2025)
- [Mathematical Models of Computation in Superposition](https://mlsanity.com/paper/2408.05451/) (2024)
- [Backward Lens: Projecting Language Model Gradients into the Vocabulary Space](https://mlsanity.com/paper/2402.12865/) (2024)
- [Interpreting the linear structure of vision-language model embedding spaces](https://mlsanity.com/paper/2504.11695/) (2025)
- [The Quest for the Right Mediator: Surveying Mechanistic Interpretability Through the Lens of Causal Mediation Analysis](https://mlsanity.com/paper/2408.01416/) (2024)
- [NeuroStrike: Neuron-Level Attacks on Aligned LLMs](https://mlsanity.com/paper/2509.11864/) (2025)
- [Planning in a recurrent neural network that plays Sokoban](https://mlsanity.com/paper/2407.15421/) (2024)
- [Sparse Autoencoders Enable Scalable and Reliable Circuit Identification in Language Models](https://mlsanity.com/paper/2405.12522/) (2024)
- [Neural Concept Binder](https://mlsanity.com/paper/2406.09949/) (2024)
- [Monet: Mixture of Monosemantic Experts for Transformers](https://mlsanity.com/paper/2412.04139/) (2024)
- [Model Directions, Not Words: Mechanistic Topic Models Using Sparse Autoencoders](https://mlsanity.com/paper/2507.23220/) (2025)
- [InversionView: A General-Purpose Method for Reading Information from Neural Activations](https://mlsanity.com/paper/2405.17653/) (2024)
- [Residual vector quantization for KV cache compression in large language model](https://mlsanity.com/paper/2410.15704/) (2024)
- [Improving Semantic Control in Discrete Latent Spaces with Transformer Quantized Variational Autoencoders](https://mlsanity.com/paper/2402.00723/) (2024)
- [Towards scientific discovery with dictionary learning: Extracting biological concepts from microscopy foundation models](https://mlsanity.com/paper/2412.16247/) (2024)
- [Towards Intrinsic Interpretability of Large Language Models:A Survey of Design Principles and Architectures](https://mlsanity.com/paper/2604.16042/) (2026)
- [Understanding the role of FFNs in driving multilingual behaviour in LLMs](https://mlsanity.com/paper/2404.13855/) (2024)
- [Human Motion Unlearning](https://mlsanity.com/paper/2503.18674/) (2025)
- [ProtSAE: Disentangling and Interpreting Protein Language Models via Semantically-Guided Sparse Autoencoders](https://mlsanity.com/paper/2509.05309/) (2025)
- [Binary Sparse Coding for Interpretability](https://mlsanity.com/paper/2509.25596/) (2025)
- [Explicit Fuzzy Logic in the Feed-Forward Layer: Self-Forgetting Quantifiers Discover Legible Grammatical-Licensing Detectors](https://mlsanity.com/paper/2606.31845/) (2026)

## Abstract (arXiv)

Understanding neural networks is challenging in part because of the dense, continuous nature of their hidden states. We explore whether we can train neural networks to have hidden states that are sparse, discrete, and more interpretable by quantizing their continuous features into what we call codebook features. Codebook features are produced by finetuning neural networks with vector quantization bottlenecks at each layer, producing a network whose hidden features are the sum of a small number of discrete vector codes chosen from a larger codebook. Surprisingly, we find that neural networks can operate under this extreme bottleneck with only modest degradation in performance. This sparse, discrete bottleneck also provides an intuitive way of controlling neural network behavior: first, find codes that activate when the desired behavior is present, then activate those same codes during generation to elicit that behavior. We validate our approach by training codebook Transformers on several different datasets. First, we explore a finite state machine dataset with far more hidden states than neurons. In this setting, our approach overcomes the superposition problem by assigning states to distinct codes, and we find that we can make the neural network behave as if it is in a different state by activating the code for that state. Second, we train Transformer language models with up to 410M parameters on two natural language datasets. We identify codes in these models representing diverse, disentangled concepts (ranging from negative emotions to months of the year) and find that we can guide the model to generate different topics by activating the appropriate codes during inference. Overall, codebook features appear to be a promising unit of analysis and control for neural networks and interpretability. Our codebase and models are open-sourced at https://github.com/taufeeque9/codebook-features.