mlsanityData to 7 Oct 2026

Emergent Low-Rank Training Dynamics in MLPs with Smooth Activations

arXiv 2602.06208 ↗First posted 5 Feb 2026cs.LG cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

Recent empirical evidence has demonstrated that the training dynamics of large-scale deep neural networks occur within low-dimensional subspaces. While this has inspired new research into low-rank training, compression, and adaptation, theoretical justification for these dynamics in nonlinear networks remains limited. %compared to deep linear settings. To address this gap, this paper analyzes the learning dynamics of multi-layer perceptrons (MLPs) under gradient descent (GD). We demonstrate that the weight dynamics concentrate within invariant low-dimensional subspaces throughout training. Theoretically, we precisely characterize these invariant subspaces for two-layer networks with smooth nonlinear activations, providing insight into their emergence. Experimentally, we validate that this phenomenon extends beyond our theoretical assumptions. Leveraging these insights, we empirically show there exists a low-rank MLP parameterization that, when initialized within the appropriate subspaces, matches the classification performance of fully-parameterized counterparts on a variety of classification tasks.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 40 of its 53 references that are in the map. First posted on arXiv 5 Feb 2026.

Builds on

40 of 53 references.

References · 40

  1. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  2. 2017Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithmscited by 4,283
  3. 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,250
  4. 2020Prevalence of Neural Collapse during the terminal phase of deep learning trainingcited by 713
  5. 2024GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projectioncited by 447
  6. 2015An Introduction to Matrix Concentration Inequalitiescited by 391
  7. 2018Measuring the Intrinsic Dimension of Objective Landscapescited by 364
  8. 2017Implicit Regularization in Matrix Factorizationcited by 318
  9. 2018Gradient Descent Happens in a Tiny Subspacecited by 208
  10. 2018A Convergence Analysis of Gradient Descent for Deep Linear Neural Networkscited by 180
  11. 2019Towards moderate overparameterization: global convergence guarantees for training shallow neural networkscited by 176
  12. 2020Towards Resolving the Implicit Bias of Gradient Descent for Matrix Factorization: Greedy Low-Rank Learningcited by 148
28 more
  1. 2022High-dimensional Asymptotics of Feature Learning: How One Gradient Step Improves the Representationcited by 146
  2. 2020Evaluation of Neural Architectures Trained with Square Loss vs Cross-Entropy in Classification Taskscited by 137
  3. 2019Implicit Regularization of Discrete Gradient Dynamics in Linear Neural Networkscited by 122
  4. 2018Overparameterized Nonlinear Learning: Gradient Descent Takes the Shortest Path?cited by 87
  5. 2022On the Implicit Bias in Deep-Learning Algorithmscited by 84
  6. 2022Are All Losses Created Equal: A Neural Collapse Perspectivecited by 78
  7. 2022Gradient flow dynamics of shallow ReLU networks for square loss and orthogonal inputscited by 74
  8. 2022Implicit Regularization Towards Rank Minimization in ReLU Networkscited by 53
  9. 2024APOLLO: SGD-like Memory, AdamW-level Performancecited by 50
  10. 2022Implicit Bias in Leaky ReLU Networks Trained on High-Dimensional Datacited by 46
  11. 2024From Lazy to Rich: Exact Learning Dynamics in Deep Linear Networkscited by 46
  12. 2022Low-rank lottery tickets: finding efficient low-rank neural networks via matrix differential equationscited by 42
  13. 2024LDAdam: Adaptive Optimization from Low-Dimensional Gradient Statisticscited by 35
  14. 2023Understanding Deep Representation Learning via Layerwise Feature Compression and Discriminationcited by 31
  15. 2021How many degrees of freedom do we need to train deep networks: a loss landscape perspectivecited by 28
  16. 2023Early Neuron Alignment in Two-layer ReLU Networks with Small Initializationcited by 28
  17. 2024Compressible Dynamics in Deep Overparameterized Low-Rank Learning & Adaptationcited by 26
  18. 2024From Low Rank Gradient Subspace Stabilization to Low-Rank Weights: Observations, Theories, and Applicationscited by 24
  19. 2024The Prevalence of Neural Collapse in Neural Multivariate Regressioncited by 21
  20. 2023Fine-tuning Happens in Tiny Subspaces: Exploring Intrinsic Task-specific Subspaces of Pre-trained Language Modelscited by 19
  21. 2023The Law of Parsimony in Gradient Descent for Learning Deep Linear Networkscited by 19
  22. 2023Implicit Bias of Gradient Descent for Two-layer ReLU and Leaky ReLU Networks on Nearly-orthogonal Datacited by 18
  23. 2023Learning a Neuron by a Shallow ReLU Network: Dynamics and Implicit Bias for Correlated Inputscited by 16
  24. 2025SubTrack++ : Gradient Subspace Tracking for Scalable LLM Trainingcited by 14
  25. 2025An Overview of Low-Rank Structures in the Training and Adaptation of Large Modelscited by 7
  26. 2022Global Convergence of SGD On Two Layer Neural Netscited by 5
  27. 2025Low Rank Gradients and Where to Find Themcited by 4
  28. 2025Linearly Separable Features in Shallow Nonlinear Networks: Width Scales Polynomially with Intrinsic Data Dimensioncited by 1

Built on by

Not cited yet.