# Backpack Language Models

arXiv: https://arxiv.org/abs/2305.16765 · HTML: https://mlsanity.com/paper/2305.16765/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Builds on 33 of its 59 references that are in the map.
- First posted on arXiv 26 May 2023.

## Builds on (most cited first, 30 of 33)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Efficient Estimation of Word Representations in Vector Space](https://mlsanity.com/paper/1301.3781/) (2013)
- [FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness](https://mlsanity.com/paper/2205.14135/) (2022)
- [Pointer Sentinel Mixture Models](https://mlsanity.com/paper/1609.07843/) (2016)
- [Locating and Editing Factual Associations in GPT](https://mlsanity.com/paper/2202.05262/) (2022)
- [Efficiently Modeling Long Sequences with Structured State Spaces](https://mlsanity.com/paper/2111.00396/) (2021)
- [Deep contextualized word representations](https://mlsanity.com/paper/1802.05365/) (2018)
- [Enriching Word Vectors with Subword Information](https://mlsanity.com/paper/1607.04606/) (2016)
- [Interpretability Beyond Feature Attribution: Quantitative Testing with Concept Activation Vectors (TCAV)](https://mlsanity.com/paper/1711.11279/) (2017)
- [Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddings](https://mlsanity.com/paper/1607.06520/) (2016)
- [Concept Bottleneck Models](https://mlsanity.com/paper/2007.04612/) (2020)
- [Mass-Editing Memory in a Transformer](https://mlsanity.com/paper/2210.07229/) (2022)
- [The LAMBADA dataset: Word prediction requiring a broad discourse context](https://mlsanity.com/paper/1606.06031/) (2016)
- [In-context Learning and Induction Heads](https://mlsanity.com/paper/2209.11895/) (2022)
- [Plug and Play Language Models: A Simple Approach to Controlled Text Generation](https://mlsanity.com/paper/1912.02164/) (2019)
- [Gender Bias in Coreference Resolution: Evaluation and Debiasing Methods](https://mlsanity.com/paper/1804.06876/) (2018)
- [BLiMP: The Benchmark of Linguistic Minimal Pairs for English](https://mlsanity.com/paper/1912.00582/) (2019)
- [Editing Factual Knowledge in Language Models](https://mlsanity.com/paper/2104.08164/) (2021)
- [MAUVE: Measuring the Gap Between Neural Text and Human Text using Divergence Frontiers](https://mlsanity.com/paper/2102.01454/) (2021)
- [Linguistic Knowledge and Transferability of Contextual Representations](https://mlsanity.com/paper/1903.08855/) (2019)
- [Neural Additive Models: Interpretable Machine Learning with Neural Nets](https://mlsanity.com/paper/2004.13912/) (2020)
- [Using the Output Embedding to Improve Language Models](https://mlsanity.com/paper/1608.05859/) (2016)
- [SimLex-999: Evaluating Semantic Models with (Genuine) Similarity Estimation](https://mlsanity.com/paper/1408.3456/) (2014)
- [The emergence of number and syntax units in LSTM language models](https://mlsanity.com/paper/1903.07435/) (2019)
- [NODE-GAM: Neural Generalized Additive Model for Interpretable Deep Learning](https://mlsanity.com/paper/2106.01613/) (2021)
- [GAMI-Net: An Explainable Neural Network based on Generalized Additive Models with Structured Interactions](https://mlsanity.com/paper/2003.07132/) (2020)
- [Neural Basis Models for Interpretability](https://mlsanity.com/paper/2205.14120/) (2022)
- [SimVerb-3500: A Large-Scale Evaluation Set of Verb Similarity](https://mlsanity.com/paper/1608.00869/) (2016)
- [Scalable Interpretability via Polynomials](https://mlsanity.com/paper/2205.14108/) (2022)
- [Probing for Incremental Parse States in Autoregressive Language Models](https://mlsanity.com/paper/2211.09748/) (2022)

## Built on by (most cited first, 15 of 15)

- [Quiet-STaR: Language Models Can Teach Themselves to Think Before Speaking](https://mlsanity.com/paper/2403.09629/) (2024)
- [Flipping the Dialogue: Training and Evaluating User Language Models](https://mlsanity.com/paper/2510.06552/) (2025)
- [A Survey on Knowledge Editing of Neural Networks](https://mlsanity.com/paper/2310.19704/) (2023)
- [Gradient Routing: Masking Gradients to Localize Computation in Neural Networks](https://mlsanity.com/paper/2410.04332/) (2024)
- [The Quest for the Right Mediator: Surveying Mechanistic Interpretability Through the Lens of Causal Mediation Analysis](https://mlsanity.com/paper/2408.01416/) (2024)
- [PaCE: Parsimonious Concept Engineering for Large Language Models](https://mlsanity.com/paper/2406.04331/) (2024)
- [Monet: Mixture of Monosemantic Experts for Transformers](https://mlsanity.com/paper/2412.04139/) (2024)
- [Semformer: Transformer Language Models with Semantic Planning](https://mlsanity.com/paper/2409.11143/) (2024)
- [Towards Intrinsic Interpretability of Large Language Models:A Survey of Design Principles and Architectures](https://mlsanity.com/paper/2604.16042/) (2026)
- [Mini-Giants: "Small" Language Models and Open Source Win-Win](https://mlsanity.com/paper/2307.08189/) (2023)
- [Model Editing with Canonical Examples](https://mlsanity.com/paper/2402.06155/) (2024)
- [HANS, are you clever? Clever Hans Effect Analysis of Neural Systems](https://mlsanity.com/paper/2309.12481/) (2023)
- [Representing Rule-based Chatbots with Transformers](https://mlsanity.com/paper/2407.10949/) (2024)
- [Training, Reading, and Editing Legible Transformers](https://mlsanity.com/paper/2607.08946/) (2026)
- [Programmatic Representation Learning with Language Models](https://mlsanity.com/paper/2510.14825/) (2025)

## Abstract (arXiv)

We present Backpacks: a new neural architecture that marries strong modeling performance with an interface for interpretability and control. Backpacks learn multiple non-contextual sense vectors for each word in a vocabulary, and represent a word in a sequence as a context-dependent, non-negative linear combination of sense vectors in this sequence. We find that, after training, sense vectors specialize, each encoding a different aspect of a word. We can interpret a sense vector by inspecting its (non-contextual, linear) projection onto the output space, and intervene on these interpretable hooks to change the model's behavior in predictable ways. We train a 170M-parameter Backpack language model on OpenWebText, matching the loss of a GPT-2 small (124Mparameter) Transformer. On lexical similarity evaluations, we find that Backpack sense vectors outperform even a 6B-parameter Transformer LM's word embeddings. Finally, we present simple algorithms that intervene on sense vectors to perform controllable text generation and debiasing. For example, we can edit the sense vocabulary to tend more towards a topic, or localize a source of gender bias to a sense vector and globally suppress that sense.