mlsanityData to 7 Oct 2026

All or None: Identifiable Linear Properties of Next-token Predictors in Language Modeling

arXiv 2410.23501 ↗First posted 30 Oct 2024stat.ML cs.AI cs.CLcited by 14

Not on the map. Why?

Abstract (arXiv)

We analyze identifiability as a possible explanation for the ubiquity of linear properties across language models, such as the vector difference between the representations of "easy" and "easiest" being parallel to that between "lucky" and "luckiest". For this, we ask whether finding a linear property in one model implies that any model that induces the same distribution has that property, too. To answer that, we first prove an identifiability result to characterize distribution-equivalent next-token predictors, lifting a diversity requirement of previous results. Second, based on a refinement of relational linearity [Paccanaro and Hinton, 2001; Hernandez et al., 2024], we show how many notions of linearity are amenable to our analysis. Finally, we show that under suitable conditions, these linear properties either hold in all or none distribution-equivalent next-token predictors.

Key facts. Cited by 14 papers in the map (data through 7 Oct 2026). Builds on 46 of its 81 references that are in the map. First posted on arXiv 30 Oct 2024.

Builds on

46 of 81 references.

References · 46

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2013Efficient Estimation of Word Representations in Vector Spacecited by 4,346
  3. 2013Distributed Representations of Words and Phrases and their Compositionalitycited by 3,462
  4. 2017Interpretability Beyond Feature Attribution: Quantitative Testing with Concept Activation Vectors (TCAV)cited by 1,309
  5. 2016Understanding intermediate layers using linear classifier probescited by 1,279
  6. 2024Refusal in Language Models Is Mediated by a Single Directioncited by 960
  7. 2022Toy Models of Superpositioncited by 928
  8. 2022Discovering Latent Knowledge in Language Models Without Supervisioncited by 733
  9. 2023The Linear Representation Hypothesis and the Geometry of Large Language Modelscited by 679
  10. 2023The Geometry of Truth: Emergent Linear Structure in Large Language Model Representations of True/False Datasetscited by 673
  11. 2019Variational Autoencoders and Nonlinear ICA: A Unifying Frameworkcited by 494
  12. 2022Emergent World Representations: Exploring a Sequence Model Trained on a Synthetic Taskcited by 417
34 more
  1. 2018Generating Wikipedia by Summarizing Long Sequencescited by 333
  2. 2023Trained Transformers Learn Linear Models In-Contextcited by 331
  3. 2023Emergent Linear Representations in World Models of Self-Supervised Sequence Modelscited by 313
  4. 2023Language Models Represent Space and Timecited by 306
  5. 2018Nonlinear ICA Using Auxiliary Variables and Generalized Contrastive Learningcited by 278
  6. 2023Does Localization Inform Editing? Surprising Differences in Causality-Based Localization vs. Knowledge Editing in Language Modelscited by 237
  7. 2023Linearity of Relation Decoding in Transformer Language Modelscited by 190
  8. 2023Linear Representations of Sentiment in Large Language Modelscited by 157
  9. 2024The Geometry of Categorical and Hierarchical Concepts in Large Language Modelscited by 151
  10. 2024Improving Instruction-Following in Language Models through Activation Steeringcited by 149
  11. 2022Interventional Causal Representation Learningcited by 140
  12. 2022CITRIS: Causal Identifiability from Temporal Intervened Sequencescited by 132
  13. 2023Future Lens: Anticipating Subsequent Tokens from a Single Hidden Statecited by 116
  14. 2023Nonparametric Identifiability of Causal Representations from Unknown Interventionscited by 105
  15. 2020On Linear Identifiability of Learned Representationscited by 103
  16. 2020ICE-BeeM: Identifiable Conditional Energy-Based Deep Models Based on Nonlinear ICAcited by 102
  17. 2023Language Models Implement Simple Word2Vec-style Vector Arithmeticcited by 97
  18. 2022The Geometry of Multilingual Language Model Representationscited by 94
  19. 2019The Incomplete Rosetta Stone Problem: Identifiability Results for Multi-View Nonlinear ICAcited by 86
  20. 2023Identifiability Guarantees for Causal Disentanglement from Soft Interventionscited by 84
  21. 2020Hidden Markov Nonlinear ICA: Unsupervised Learning from Nonstationary Time Seriescited by 76
  22. 2024On the Origins of Linear Representations in Large Language Modelscited by 76
  23. 2019Analogies Explained: Towards Understanding Word Embeddingscited by 63
  24. 2023Causal Component Analysiscited by 56
  25. 2021Identifiable Deep Generative Models via Sparse Decodingcited by 54
  26. 2022Function Classes for Identifiable Nonlinear Independent Component Analysiscited by 53
  27. 2022Synergies between Disentanglement and Sparsity: Generalization and Identifiability in Multi-Task Learningcited by 44
  28. 2023General Identifiability and Achievability for Causal Representation Learningcited by 34
  29. 2022Indeterminacy in Generative Models: Characterization and Strong Identifiabilitycited by 33
  30. 2024Learning Interpretable Concepts: Unifying Causal Representation Learning and Foundation Modelscited by 29
  31. 2024Position: Understanding LLMs Requires More Than Statistical Generalizationcited by 20
  32. 2024Monotonic Representation of Numeric Properties in Language Modelscited by 15
  33. 2023Towards fully covariant machine learningcited by 10
  34. 2021Binary Independent Component Analysis: A Non-stationarity-based Approachcited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 6; 2026: 8.

14 papers.

Sort:

  1. 2025I Predict Therefore I Am: Is Next Token Prediction Enough to Learn Human-Interpretable Concepts from Data?cited by 13
  2. 2025Position: An Empirically Grounded Identifiability Theory Will Accelerate Self-Supervised Learning Researchcited by 9
  3. 2025What is causal about causal models and representations?cited by 6
  4. 2026Stop Probing, Start Coding: Why Linear Probes and Sparse Autoencoders Fail at Compositional Generalisationcited by 5
  5. 2025When Does Closeness in Distribution Imply Representational Similarity? An Identifiability Perspectivecited by 4
  6. 2026Statistical and structural identifiability in representation learningcited by 4
  7. 2025Nonparametric Identification of Latent Conceptscited by 3
  8. 2025Beyond the Black Box: Identifiable Interpretation and Control in Generative Models via Causal Minimalitycited by 0
  9. 2026Concept Component Analysis: A Principled Approach for Concept Extraction in LLMscited by 0
  10. 2026Logit Distance Bounds Representational Similaritycited by 0
  11. 2026Who Guards the Guardians? The Challenges of Evaluating Identifiability of Learned Representationscited by 0
  12. 2026On the Blessing of Pre-training in Weak-to-Strong Generalizationcited by 0
2 more
  1. 2026Relational Linear Properties in Language Models: An Empirical Investigationcited by 0
  2. 2026Not all solutions are created equal: An analytical dissociation of functional and representational similarity in deep linear neural networkscited by 0