mlsanityData to 7 Oct 2026

Transformers learn through gradual rank increase

arXiv 2306.07042 ↗First posted 12 Jun 2023cs.LGcited by 52

Not on the map. Why?

Abstract (arXiv)

We identify incremental learning dynamics in transformers, where the difference between trained and initial weights progressively increases in rank. We rigorously prove this occurs under the simplifying assumptions of diagonal weight matrices and small initialization. Our experiments support the theory and also show that phenomenon can occur in practice without the simplifying assumptions.

Key facts. Cited by 52 papers in the map (data through 7 Oct 2026). Builds on 40 of its 56 references that are in the map. First posted on arXiv 12 Jun 2023.

Builds on

40 of 56 references.

References · 40

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  6. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  7. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  8. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  9. 2013Exact solutions to the nonlinear dynamics of learning in deep linear neural networkscited by 748
  10. 2018On Lazy Training in Differentiable Programmingcited by 671
  11. 2020Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuningcited by 658
  12. 2019Implicit Regularization in Deep Matrix Factorizationcited by 402
28 more
  1. 2018Measuring the Intrinsic Dimension of Objective Landscapescited by 364
  2. 2020Kernel and Rich Regimes in Overparametrized Modelscited by 302
  3. 2018On the Optimization of Deep Networks: Implicit Acceleration by Overparameterizationcited by 280
  4. 2018Algorithmic Regularization in Learning Deep Homogeneous Models: Layers are Automatically Balancedcited by 172
  5. 2022Neural Networks can Learn Representations with Gradient Descentcited by 161
  6. 2020Towards Resolving the Implicit Bias of Gradient Descent for Matrix Factorization: Greedy Low-Rank Learningcited by 148
  7. 2023SGD learning on neural networks: leap complexity and saddle-to-saddle dynamicscited by 142
  8. 2022The merged-staircase property: a necessary and nearly sufficient condition for SGD learning of sparse functions on two-layer neural networkscited by 141
  9. 2019Implicit Regularization of Discrete Gradient Dynamics in Linear Neural Networkscited by 122
  10. 2020Implicit Regularization in Deep Learning May Not Be Explainable by Normscited by 118
  11. 2022Learning Single-Index Models with Shallow Neural Networkscited by 92
  12. 2019Limitations of Lazy Training of Two-layers Neural Networkscited by 87
  13. 2021Saddle-to-Saddle Dynamics in Deep Linear Networks: Small Initialization Training, Symmetry, and Sparsitycited by 83
  14. 2021Small random initialization is akin to spectral learning: Optimization and generalization guarantees for overparameterized low-rank matrix reconstructioncited by 77
  15. 2022Gradient flow dynamics of shallow ReLU networks for square loss and orthogonal inputscited by 74
  16. 2020Implicit Bias in Deep Linear Classification: Initialization Scale vs Training Accuracycited by 70
  17. 2019The Implicit Bias of Depth: How Incremental Learning Drives Generalizationcited by 64
  18. 2022Neural Networks Efficiently Learn Low-Dimensional Representations with SGDcited by 61
  19. 2023Saddle-to-Saddle Dynamics in Diagonal Linear Networkscited by 54
  20. 2022Implicit Regularization Towards Rank Minimization in ReLU Networkscited by 53
  21. 2022Implicit Bias in Leaky ReLU Networks Trained on High-Dimensional Datacited by 46
  22. 2021Quantifying the Benefit of Using Differentiable Learning over Tangent Kernelscited by 41
  23. 2023Understanding Incremental Learning of Gradient Descent: A Fine-grained Analysis of Matrix Sensingcited by 40
  24. 2023On the Stepwise Nature of Self-Supervised Learningcited by 36
  25. 2022Incremental Learning in Diagonal Linear Networkscited by 27
  26. 2019Disentangling feature and lazy training in deep neural networkscited by 10
  27. 2021Implicit Regularization in Deep Tensor Factorizationcited by 5
  28. 2022Rank-1 Matrix Completion with Gradient Descent and Small Random Initializationcited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 5; 2024: 18; 2025: 17; 2026: 12.

52 papers.

Sort:

By area:

  1. 2024How Transformers Learn Causal Structure with Gradient Descentcited by 114
  2. 2023Transformers as Support Vector Machinescited by 67
  3. 2023JoMA: Demystifying Multilayer Transformers via JOint Dynamics of MLP and Attentioncited by 52
  4. 2024LoRA Training in the NTK Regime has No Spurious Local Minimacited by 48
  5. 2025Training Dynamics of In-Context Learning in Linear Attentioncited by 44
  6. 2025When is Task Vector Provably Effective for Model Editing? A Generalization Analysis of Nonlinear Transformerscited by 38
  7. 2024Beyond Anti-Forgetting: Multimodal Continual Instruction Tuning with Positive Forward Transfercited by 31
  8. 2024A phase transition between positional and semantic learning in a solvable model of dot-product attentioncited by 30
  9. 2024How JEPA Avoids Noisy Features: The Implicit Bias of Deep Linear Self Distillation Networkscited by 29
  10. 2024Approaching Deep Learning through the Spectral Dynamics of Weightscited by 29
  11. 2024Towards a Theoretical Understanding of the 'Reversal Curse' via Training Dynamicscited by 27
  12. 2024Understanding the Expressive Power and Mechanisms of Transformer for Sequence Modelingcited by 25
38 more
  1. 2024A distributional simplicity bias in the learning dynamics of transformerscited by 23
  2. 2025PoLAR: Polar-Decomposed Low-Rank Adapter Representationcited by 19
  3. 2023Uncovering hidden geometry in Transformers via disentangling position and contextcited by 15
  4. 2025Learning In-context n-grams with Transformers: Sub-n-grams Are Near-stationary Pointscited by 15
  5. 2024Towards a theory of model distillationcited by 14
  6. 2024Geometric Signatures of Compositionality Across a Language Model's Lifetimecited by 14
  7. 2025Saddle-to-Saddle Dynamics Explains A Simplicity Bias Across Neural Network Architecturescited by 14
  8. 2023Characterizing Large Language Model Geometry Helps Solve Toxicity Detection and Generationcited by 11
  9. 2025Alternating Gradient Flows: A Theory of Feature Learning in Two-layer Neural Networkscited by 11
  10. 2025How Transformers Learn Regular Language Recognition: A Theoretical Study on Training Dynamics and Implicit Biascited by 10
  11. 2025Subspace Networks: Scaling Decentralized Training with Communication-Efficient Model Parallelismcited by 10
  12. 2025Generalization or Hallucination? Understanding Out-of-Context Reasoning in Transformerscited by 10
  13. 2024Dissecting the Interplay of Attention Paths in a Statistical Mechanics Theory of Transformerscited by 9
  14. 2024Learning on Transformers is Provable Low-Rank and Sparse: A One-layer Analysiscited by 8
  15. 2024How Transformers Get Rich: Approximation and Dynamics Analysiscited by 7
  16. 2025DiTASK: Multi-Task Fine-Tuning with Diffeomorphic Transformationscited by 7
  17. 2025Optimizing Rank for High-Fidelity Implicit Neural Representationscited by 7
  18. 2026To Use or not to Use Muon: How Simplicity Bias in Optimizers Matterscited by 7
  19. 2024Can Implicit Bias Imply Adversarial Robustness?cited by 5
  20. 2025What Happens During the Loss Plateau? Understanding Abrupt Learning in Transformerscited by 5
  21. 2026KV-CoRE: Benchmarking Data-Dependent Low-Rank Compressibility of KV-Caches in LLMscited by 5
  22. 2024Compression of Structured Data with Autoencoders: Provable Benefit of Nonlinearities and Depthcited by 4
  23. 2024Reasoning in Large Language Models: A Geometric Perspectivecited by 4
  24. 2025Exact Learning Dynamics of In-Context Learning in Linear Transformers and Its Application to Non-Linear Transformerscited by 3
  25. 2023Implicit Regularization Makes Overparameterized Asymmetric Matrix Sensing Robust to Perturbationscited by 2
  26. 2026Breaking the Reversal Curse in Autoregressive Language Models via Identity Bridgecited by 2
  27. 2026Swift-SVD: Theoretical Optimality Meets Practical Efficiency in Low-Rank LLM Compressioncited by 2
  28. 2026Algebraic Dead Directions in LayerNorm Transformers: A Forward-Pass-Only Diagnostic at LLM Scalecited by 2
  29. 2024A Mechanistic Study of Transformers Training Dynamicscited by 1
  30. 2025Spectral Architecture Search for Neural Network Modelscited by 1
  31. 2025Understanding Incremental Learning with Closed-form Solution to Gradient Flow on Overparamerterized Matrix Factorizationcited by 1
  32. 2025Towards Understanding Transformers in Learning Random Walkscited by 1
  33. 2026Incremental Learning of Sparse Attention Patterns in Transformerscited by 1
  34. 2026Transformers Provably Learn to Internalize Chain-of-Thoughtcited by 1
  35. 2026Conservation Laws for Modern Neural Architecturescited by 1
  36. 2026Dead-Direction Signatures: A Cheap Spectral Reading of Singular Complexitycited by 1
  37. 2025SCALE: Upscaled Continual Learning of Large Language Modelscited by 0
  38. 2026Local EGOP for Continuous Index Learningcited by 0

50 most cited of 52 All 52 (JSON)