mlsanityData to 14 Oct 2026

On the Origins of Linear Representations in Large Language Models

arXiv 2403.03867 ↗First posted 6 Mar 2024cs.CL cs.LG stat.MLcited by 79

Not on the map. Why?

Cited by 20 in Mechanistic interpretability, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Recent works have argued that high-level semantic concepts are encoded "linearly" in the representation space of large language models. In this work, we study the origins of such linear representations. To that end, we introduce a simple latent variable model to abstract and formalize the concept dynamics of the next token prediction. We use this formalism to show that the next token prediction objective (softmax with cross-entropy) and the implicit bias of gradient descent together promote the linear representation of concepts. Experiments show that linear representations emerge when learning from data matching the latent variable model, confirming that this simple structure already suffices to yield linear representations. We additionally confirm some predictions of the theory using the LLaMA-2 large language model, giving evidence that the simplified model yields generalizable insights.

Key facts. Cited by 79 papers in the map (data through 14 Oct 2026). Builds on 45 of its 64 references that are in the map. First posted on arXiv 6 Mar 2024.

Builds on

45 of 64 references.

References · 45

  1. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  3. 2015Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networkscited by 2,054
  4. 2017Interpretability Beyond Feature Attribution: Quantitative Testing with Concept Activation Vectors (TCAV)cited by 1,319
  5. 2023Inference-Time Intervention: Eliciting Truthful Answers from a Language Modelcited by 1,302
  6. 2022Toy Models of Superpositioncited by 971
  7. 2022Discovering Latent Knowledge in Language Models Without Supervisioncited by 767
  8. 2017Network Dissection: Quantifying Interpretability of Deep Visual Representationscited by 708
  9. 2017The Implicit Bias of Gradient Descent on Separable Datacited by 655
  10. 2022Mind the Gap: Understanding the Modality Gap in Multi-modal Contrastive Representation Learningcited by 561
  11. 2019Variational Autoencoders and Nonlinear ICA: A Unifying Frameworkcited by 503
  12. 2023Finding Neurons in a Haystack: Case Studies with Sparse Probingcited by 334
33 more
  1. 2023Emergent Linear Representations in World Models of Self-Supervised Sequence Modelscited by 319
  2. 2021Contrastive Learning Inverts the Data Generating Processcited by 221
  3. 2019Visualizing and Measuring the Geometry of BERTcited by 216
  4. 2022Relative representations enable zero-shot latent space communicationcited by 191
  5. 2016Linear Algebraic Structure of Word Senses, with Applications to Polysemycited by 165
  6. 2023Linear Representations of Sentiment in Large Language Modelscited by 163
  7. 2021Disentanglement via Mechanism Sparsity Regularization: A New Principle for Nonlinear ICAcited by 160
  8. 2021Acquisition of Chess Knowledge in AlphaZerocited by 123
  9. 2022The Geometry of Multilingual Language Model Representationscited by 95
  10. 2023Learning Linear Causal Representations from Interventions under General Nonlinear Mixingcited by 92
  11. 2022Identifiability of deep generative models without auxiliary informationcited by 70
  12. 2019Analogies Explained: Towards Understanding Word Embeddingscited by 64
  13. 2023Identifiability of latent-variable and structural-equation models: from linear to nonlinearcited by 58
  14. 2018Towards Understanding Linear Word Analogiescited by 57
  15. 2021Learning latent causal graphs via mixture oraclescited by 55
  16. 2021Probing BERT in Hyperbolic Spacescited by 50
  17. 2023Score-based Causal Representation Learning with Interventionscited by 49
  18. 2023Linear Spaces of Meanings: Compositional Structures in Vision-Language Modelscited by 48
  19. 2022Causal Structure Learning: a Combinatorial Perspectivecited by 46
  20. 2023Bridging the Human-AI Knowledge Gap: Concept Discovery and Transfer in AlphaZerocited by 42
  21. 2017Latent Constraints: Learning to Generate Conditionally from Unconditional Generative Modelscited by 39
  22. 2024Learning Interpretable Concepts: Unifying Causal Representation Learning and Foundation Modelscited by 29
  23. 2021Multi-Facet Clustering Variational Autoencoderscited by 28
  24. 2017SVCCA: Singular Vector Canonical Correlation Analysis for Deep Learning Dynamics and Interpretabilitycited by 16
  25. 2023Uncovering Meanings of Embeddings via Partial Orthogonalitycited by 15
  26. 2016Exponential Family Embeddingscited by 13
  27. 2021Structure learning in polynomial time: Greedy algorithms, Bregman information, and exponential familiescited by 11
  28. 2023An Interventional Perspective on Identifiability in Gaussian LTI Systems with Independent Component Analysiscited by 9
  29. 2015A Latent Variable Model Approach to PMI-based Word Embeddingscited by 7
  30. 2017Dynamic Bernoulli Embeddings for Language Evolutioncited by 6
  31. 2019Understanding Composition of Word Embeddings via Tensor Decompositioncited by 4
  32. 2019Natural Alpha Embeddingscited by 4
  33. 2023Contrastive Loss is All You Need to Recover Analogies as Parallel Linescited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 17; 2025: 27; 2026: 35.

79 papers.

Sort:

By area:

  1. 2024Not All Language Model Features Are One-Dimensionally Linearcited by 212
  2. 2024The Geometry of Categorical and Hierarchical Concepts in Large Language Modelscited by 159
  3. 2024A Primer on the Inner Workings of Transformer-based Language Modelscited by 85
  4. 2025From Flat to Hierarchical: Extracting Sparse Representations with Matching Pursuitcited by 49
  5. 2025I Have Covered All the Bases Here: Interpreting Reasoning Features in Large Language Models via Sparse Autoencoderscited by 45
  6. 2024Causal Language Modeling Can Elicit Search and Reasoning Capabilities on Logic Puzzlescited by 38
  7. 2026Do Sparse Autoencoders Capture Concept Manifolds?cited by 31
  8. 2026There Will Be a Scientific Theory of Deep Learningcited by 29
  9. 2025On Linear Representations and Pretraining Data Frequency in Language Modelscited by 23
  10. 2025Text Generation Beyond Discrete Token Samplingcited by 21
  11. 2025The Geometry of Reasoning: Flowing Logics in Representation Spacecited by 21
  12. 2024PaCE: Parsimonious Concept Engineering for Large Language Modelscited by 19
38 more
  1. 2025Head Pursuit: Probing Attention Specialization in Multimodal Transformerscited by 17
  2. 2024Monotonic Representation of Numeric Properties in Language Modelscited by 15
  3. 2024Do LLMs dream of elephants (when told not to)? Latent concept association and associative memory in transformerscited by 15
  4. 2024All or None: Identifiable Linear Properties of Next-token Predictors in Language Modelingcited by 14
  5. 2025I Predict Therefore I Am: Is Next Token Prediction Enough to Learn Human-Interpretable Concepts from Data?cited by 13
  6. 2025On the Emergence of Linear Analogies in Word Embeddingscited by 13
  7. 2025Emergence of Linear Truth Encodings in Language Modelscited by 12
  8. 2026The Information Geometry of Softmax: Probing and Steeringcited by 11
  9. 2025The Blessing and Curse of Dimensionality in Safety Alignmentcited by 10
  10. 2026Linear representations in language models can change dramatically over a conversationcited by 10
  11. 2025L2M: Mutual Information Scaling Law for Long-Context Language Modelingcited by 9
  12. 2025Closed-Form Training Dynamics Reveal Learned Features and Linear Structure in Word2Vec-like Modelscited by 8
  13. 2026Efficient and accurate steering of Large Language Models through attention-guided feature learningcited by 8
  14. 2024Can DPO Learn Diverse Human Values? A Theoretical Scaling Lawcited by 7
  15. 2025The Geometry of Harmfulness in LLMs through Subconcept Probingcited by 7
  16. 2024Transformer Normalisation Layers and the Independence of Semantic Subspacescited by 6
  17. 2024ResiDual Transformer Alignment with Spectral Decompositioncited by 6
  18. 2026Stop Probing, Start Coding: Why Linear Probes and Sparse Autoencoders Fail at Compositional Generalisationcited by 6
  19. 2024Provably Neural Active Learning Succeeds via Prioritizing Perplexing Samplescited by 5
  20. 2024Lines of Thought in Large Language Modelscited by 5
  21. 2025Attention-Only Transformers via Unrolled Subspace Denoisingcited by 5
  22. 2026Minimizing Collateral Damage in Activation Steeringcited by 5
  23. 2024Superposed Decoding: Multiple Generations from a Single Autoregressive Inference Passcited by 4
  24. 2024Towards scientific discovery with dictionary learning: Extracting biological concepts from microscopy foundation modelscited by 4
  25. 2025Combining Causal Models for More Accurate Abstractions of Neural Networkscited by 4
  26. 2024GPT-2 Through the Lens of Vector Symbolic Architecturescited by 3
  27. 2025Toward a Flexible Framework for Linear Representation Hypothesis Using Maximum Likelihood Estimationcited by 3
  28. 2025Nonparametric Identification of Latent Conceptscited by 3
  29. 2025A Unified Theory of Sparse Dictionary Learning in Mechanistic Interpretability: Piecewise Biconvexity and Spurious Minimacited by 3
  30. 2026How Do Transformers Learn to Associate Tokens: Gradient Leading Terms Bring Mechanistic Interpretabilitycited by 3
  31. 2026Why Linear Interpretability Works: Invariant Subspaces as a Result of Architectural Constraintscited by 3
  32. 2026The Lattice Representation Hypothesis of Large Language Modelscited by 3
  33. 2026Probing Visual Concepts in Lightweight Vision-Language Models for Automated Drivingcited by 3
  34. 2025Causality Is Key to Understand and Balance Multiple Goals in Trustworthy ML and Foundation Modelscited by 2
  35. 2025Unveiling the Latent Directions of Reflection in Large Language Modelscited by 2
  36. 2026Superposition Without Interference? Towards Isolated Interventions via Almost Orthogonal Features in Language Modelscited by 2
  37. 2026Structure Before Collapse: Transient semantic geometry in next-token predictioncited by 2
  38. 2024Compositional Structures in Neural Embedding and Interaction Decompositionscited by 1

50 most cited of 79 All 79 (JSON)