mlsanityData to 7 Oct 2026

Closed-Form Training Dynamics Reveal Learned Features and Linear Structure in Word2Vec-like Models

arXiv 2502.09863 ↗First posted 14 Feb 2025cs.LG cs.CL stat.MLcited by 7

Not on the map. Why?

Abstract (arXiv)

Self-supervised word embedding algorithms such as word2vec provide a minimal setting for studying representation learning in language modeling. We examine the quartic Taylor approximation of the word2vec loss around the origin, and we show that both the resulting training dynamics and the final performance on downstream tasks are empirically very similar to those of word2vec. Our main contribution is to analytically solve for both the gradient flow training dynamics and the final word embeddings in terms of only the corpus statistics and training hyperparameters. The solutions reveal that these models learn orthogonal linear subspaces one at a time, each one incrementing the effective rank of the embeddings until model capacity is saturated. Training on Wikipedia, we find that each of the top linear subspaces represents an interpretable topic-level concept. Finally, we apply our theory to describe how linear representations of more abstract semantic concepts emerge during training; these can be used to complete analogies via vector addition.

Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Builds on 36 of its 59 references that are in the map. First posted on arXiv 14 Feb 2025.

Builds on

36 of 59 references.

References · 36

  1. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  2. 2013Distributed Representations of Words and Phrases and their Compositionalitycited by 3,462
  3. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  4. 2023Inference-Time Intervention: Eliciting Truthful Answers from a Language Modelcited by 1,259
  5. 2023Representation Engineering: A Top-Down Approach to AI Transparencycited by 1,239
  6. 2022Editing Models with Task Arithmeticcited by 1,076
  7. 2013Exact solutions to the nonlinear dynamics of learning in deep linear neural networkscited by 748
  8. 2019Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descentcited by 732
  9. 2023The Linear Representation Hypothesis and the Geometry of Large Language Modelscited by 679
  10. 2018On Lazy Training in Differentiable Programmingcited by 671
  11. 2023Are Emergent Abilities of Large Language Models a Mirage?cited by 515
  12. 2019Implicit Regularization in Deep Matrix Factorizationcited by 402
24 more
  1. 2023Emergent Linear Representations in World Models of Self-Supervised Sequence Modelscited by 313
  2. 2021Provable Guarantees for Self-Supervised Deep Learning with Spectral Contrastive Losscited by 311
  3. 2021Understanding self-supervised Learning Dynamics without Contrastive Pairscited by 254
  4. 2020Spectrum Dependent Learning Curves in Kernel Regression and Wide Neural Networkscited by 200
  5. 2018A mathematical theory of semantic development in deep neural networkscited by 182
  6. 2024A Mechanistic Understanding of Alignment Algorithms: A Case Study on DPO and Toxicitycited by 179
  7. 2022Neural Networks can Learn Representations with Gradient Descentcited by 161
  8. 2020Towards Resolving the Implicit Bias of Gradient Descent for Matrix Factorization: Greedy Low-Rank Learningcited by 148
  9. 2019Implicit Regularization of Discrete Gradient Dynamics in Linear Neural Networkscited by 122
  10. 2021Neural Networks as Kernel Learners: The Silent Alignment Effectcited by 106
  11. 2019Dynamics of Deep Neural Networks and Neural Tangent Hierarchycited by 96
  12. 2022Understanding Contrastive Learning Requires Incorporating Inductive Biasescited by 96
  13. 2023How Do Transformers Learn Topic Structure: Towards a Mechanistic Understandingcited by 85
  14. 2021Saddle-to-Saddle Dynamics in Deep Linear Networks: Small Initialization Training, Symmetry, and Sparsitycited by 83
  15. 2024On the Origins of Linear Representations in Large Language Modelscited by 76
  16. 2019The Implicit Bias of Depth: How Incremental Learning Drives Generalizationcited by 64
  17. 2023Concept Algebra for (Score-Based) Text-Controlled Generative Modelscited by 59
  18. 2020Gradient Descent for Deep Matrix Factorization: Dynamics and Implicit Bias towards Low Rankcited by 42
  19. 2023Feature-Learning Networks Are Consistent Across Widths At Realistic Scalescited by 40
  20. 2019A General Framework for Implicit and Explicit Debiasing of Distributional Word Vector Spacescited by 36
  21. 2023On the Stepwise Nature of Self-Supervised Learningcited by 36
  22. 2022Algorithmic Regularization in Model-free Overparametrized Asymmetric Matrix Factorizationcited by 34
  23. 2022Quadratic models for understanding catapult dynamics of neural networkscited by 17
  24. 2024The Optimization Landscape of SGD Across the Feature Learning Strengthcited by 13

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 1; 2026: 6.

7 papers.

  1. 2026Do Sparse Autoencoders Capture Concept Manifolds?cited by 25
  2. 2025Saddle-to-Saddle Dynamics Explains A Simplicity Bias Across Neural Network Architecturescited by 14
  3. 2026Jailbreak Scaling Laws for Large Language Models: Polynomial-Exponential Crossovercited by 2
  4. 2026Structure Before Collapse: Transient semantic geometry in next-token predictioncited by 2
  5. 2026Hierarchical Concept Geometry in Language Models Emerges from Word Co-occurrencecited by 0
  6. 2026Muon learns balanced solutions in matrix factorization without slow saddle-to-saddle dynamicscited by 0
  7. 2026Group-Invariant Statistics Determine Embedding Geometry: Harmonic Analysis of Representations from Bach to the Night Skycited by 0