# On the Origins of Linear Representations in Large Language Models

arXiv: https://arxiv.org/abs/2403.03867 · HTML: https://mlsanity.com/paper/2403.03867/

## Key facts

- Cited by 79 papers in the map (data through 14 Oct 2026).
- Builds on 45 of its 64 references that are in the map.
- First posted on arXiv 6 Mar 2024.

## Builds on (most cited first, 30 of 45)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networks](https://mlsanity.com/paper/1511.06434/) (2015)
- [Interpretability Beyond Feature Attribution: Quantitative Testing with Concept Activation Vectors (TCAV)](https://mlsanity.com/paper/1711.11279/) (2017)
- [Inference-Time Intervention: Eliciting Truthful Answers from a Language Model](https://mlsanity.com/paper/2306.03341/) (2023)
- [Toy Models of Superposition](https://mlsanity.com/paper/2209.10652/) (2022)
- [Discovering Latent Knowledge in Language Models Without Supervision](https://mlsanity.com/paper/2212.03827/) (2022)
- [Network Dissection: Quantifying Interpretability of Deep Visual Representations](https://mlsanity.com/paper/1704.05796/) (2017)
- [The Implicit Bias of Gradient Descent on Separable Data](https://mlsanity.com/paper/1710.10345/) (2017)
- [Mind the Gap: Understanding the Modality Gap in Multi-modal Contrastive Representation Learning](https://mlsanity.com/paper/2203.02053/) (2022)
- [Variational Autoencoders and Nonlinear ICA: A Unifying Framework](https://mlsanity.com/paper/1907.04809/) (2019)
- [Finding Neurons in a Haystack: Case Studies with Sparse Probing](https://mlsanity.com/paper/2305.01610/) (2023)
- [Emergent Linear Representations in World Models of Self-Supervised Sequence Models](https://mlsanity.com/paper/2309.00941/) (2023)
- [Contrastive Learning Inverts the Data Generating Process](https://mlsanity.com/paper/2102.08850/) (2021)
- [Visualizing and Measuring the Geometry of BERT](https://mlsanity.com/paper/1906.02715/) (2019)
- [Relative representations enable zero-shot latent space communication](https://mlsanity.com/paper/2209.15430/) (2022)
- [Linear Algebraic Structure of Word Senses, with Applications to Polysemy](https://mlsanity.com/paper/1601.03764/) (2016)
- [Linear Representations of Sentiment in Large Language Models](https://mlsanity.com/paper/2310.15154/) (2023)
- [Disentanglement via Mechanism Sparsity Regularization: A New Principle for Nonlinear ICA](https://mlsanity.com/paper/2107.10098/) (2021)
- [Acquisition of Chess Knowledge in AlphaZero](https://mlsanity.com/paper/2111.09259/) (2021)
- [The Geometry of Multilingual Language Model Representations](https://mlsanity.com/paper/2205.10964/) (2022)
- [Learning Linear Causal Representations from Interventions under General Nonlinear Mixing](https://mlsanity.com/paper/2306.02235/) (2023)
- [Identifiability of deep generative models without auxiliary information](https://mlsanity.com/paper/2206.10044/) (2022)
- [Analogies Explained: Towards Understanding Word Embeddings](https://mlsanity.com/paper/1901.09813/) (2019)
- [Identifiability of latent-variable and structural-equation models: from linear to nonlinear](https://mlsanity.com/paper/2302.02672/) (2023)
- [Towards Understanding Linear Word Analogies](https://mlsanity.com/paper/1810.04882/) (2018)
- [Learning latent causal graphs via mixture oracles](https://mlsanity.com/paper/2106.15563/) (2021)
- [Probing BERT in Hyperbolic Spaces](https://mlsanity.com/paper/2104.03869/) (2021)
- [Score-based Causal Representation Learning with Interventions](https://mlsanity.com/paper/2301.08230/) (2023)
- [Linear Spaces of Meanings: Compositional Structures in Vision-Language Models](https://mlsanity.com/paper/2302.14383/) (2023)

## Built on by (most cited first, 30 of 79)

- [Not All Language Model Features Are One-Dimensionally Linear](https://mlsanity.com/paper/2405.14860/) (2024)
- [The Geometry of Categorical and Hierarchical Concepts in Large Language Models](https://mlsanity.com/paper/2406.01506/) (2024)
- [A Primer on the Inner Workings of Transformer-based Language Models](https://mlsanity.com/paper/2405.00208/) (2024)
- [From Flat to Hierarchical: Extracting Sparse Representations with Matching Pursuit](https://mlsanity.com/paper/2506.03093/) (2025)
- [I Have Covered All the Bases Here: Interpreting Reasoning Features in Large Language Models via Sparse Autoencoders](https://mlsanity.com/paper/2503.18878/) (2025)
- [Causal Language Modeling Can Elicit Search and Reasoning Capabilities on Logic Puzzles](https://mlsanity.com/paper/2409.10502/) (2024)
- [Do Sparse Autoencoders Capture Concept Manifolds?](https://mlsanity.com/paper/2604.28119/) (2026)
- [There Will Be a Scientific Theory of Deep Learning](https://mlsanity.com/paper/2604.21691/) (2026)
- [On Linear Representations and Pretraining Data Frequency in Language Models](https://mlsanity.com/paper/2504.12459/) (2025)
- [Text Generation Beyond Discrete Token Sampling](https://mlsanity.com/paper/2505.14827/) (2025)
- [The Geometry of Reasoning: Flowing Logics in Representation Space](https://mlsanity.com/paper/2510.09782/) (2025)
- [PaCE: Parsimonious Concept Engineering for Large Language Models](https://mlsanity.com/paper/2406.04331/) (2024)
- [Head Pursuit: Probing Attention Specialization in Multimodal Transformers](https://mlsanity.com/paper/2510.21518/) (2025)
- [Monotonic Representation of Numeric Properties in Language Models](https://mlsanity.com/paper/2403.10381/) (2024)
- [Do LLMs dream of elephants (when told not to)? Latent concept association and associative memory in transformers](https://mlsanity.com/paper/2406.18400/) (2024)
- [All or None: Identifiable Linear Properties of Next-token Predictors in Language Modeling](https://mlsanity.com/paper/2410.23501/) (2024)
- [I Predict Therefore I Am: Is Next Token Prediction Enough to Learn Human-Interpretable Concepts from Data?](https://mlsanity.com/paper/2503.08980/) (2025)
- [On the Emergence of Linear Analogies in Word Embeddings](https://mlsanity.com/paper/2505.18651/) (2025)
- [Emergence of Linear Truth Encodings in Language Models](https://mlsanity.com/paper/2510.15804/) (2025)
- [The Information Geometry of Softmax: Probing and Steering](https://mlsanity.com/paper/2602.15293/) (2026)
- [The Blessing and Curse of Dimensionality in Safety Alignment](https://mlsanity.com/paper/2507.20333/) (2025)
- [Linear representations in language models can change dramatically over a conversation](https://mlsanity.com/paper/2601.20834/) (2026)
- [L2M: Mutual Information Scaling Law for Long-Context Language Modeling](https://mlsanity.com/paper/2503.04725/) (2025)
- [Closed-Form Training Dynamics Reveal Learned Features and Linear Structure in Word2Vec-like Models](https://mlsanity.com/paper/2502.09863/) (2025)
- [Efficient and accurate steering of Large Language Models through attention-guided feature learning](https://mlsanity.com/paper/2602.00333/) (2026)
- [Can DPO Learn Diverse Human Values? A Theoretical Scaling Law](https://mlsanity.com/paper/2408.03459/) (2024)
- [The Geometry of Harmfulness in LLMs through Subconcept Probing](https://mlsanity.com/paper/2507.21141/) (2025)
- [Transformer Normalisation Layers and the Independence of Semantic Subspaces](https://mlsanity.com/paper/2406.17837/) (2024)
- [ResiDual Transformer Alignment with Spectral Decomposition](https://mlsanity.com/paper/2411.00246/) (2024)
- [Stop Probing, Start Coding: Why Linear Probes and Sparse Autoencoders Fail at Compositional Generalisation](https://mlsanity.com/paper/2603.28744/) (2026)

## Abstract (arXiv)

Recent works have argued that high-level semantic concepts are encoded "linearly" in the representation space of large language models. In this work, we study the origins of such linear representations. To that end, we introduce a simple latent variable model to abstract and formalize the concept dynamics of the next token prediction. We use this formalism to show that the next token prediction objective (softmax with cross-entropy) and the implicit bias of gradient descent together promote the linear representation of concepts. Experiments show that linear representations emerge when learning from data matching the latent variable model, confirming that this simple structure already suffices to yield linear representations. We additionally confirm some predictions of the theory using the LLaMA-2 large language model, giving evidence that the simplified model yields generalizable insights.