mlsanityData to 7 Oct 2026

Low-rank bias, weight decay, and model merging in neural networks

arXiv 2502.17340 ↗First posted 24 Feb 2025cs.LGcited by 4

Not on the map. Why?

Abstract (arXiv)

We explore the low-rank structure of the weight matrices in neural networks at the stationary points (limiting solutions of optimization algorithms) with L2 regularization (also known as weight decay). We show several properties of such deep neural networks, induced by L2 regularization. In particular, for a stationary point we show alignment of the parameters and the gradient, norm preservation across layers, and low-rank bias: properties previously known in the context of solution of gradient descent/flow type algorithms. Experiments show that the assumptions made in the analysis only mildly affect the observations. In addition, we investigate a multitask learning phenomenon enabled by L2 regularization and low-rank bias. In particular, we show that if two networks are trained, such that the inputs in the training set of one network are approximately orthogonal to the inputs in the training set of the other network, the new network obtained by simply summing the weights of the two networks will perform as well on both training sets as the respective individual networks. We demonstrate this for shallow ReLU neural networks trained by gradient descent, as well as deep linear networks trained by gradient flow.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Builds on 40 of its 48 references that are in the map. First posted on arXiv 24 Feb 2025.

Builds on

40 of 48 references.

References · 40

  1. 2017Attention Is All You Needcited by 59,932
  2. 2017Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithmscited by 4,283
  3. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  4. 2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,366
  5. 2018Averaging Weights Leads to Wider Optima and Better Generalizationcited by 1,194
  6. 2022Editing Models with Task Arithmeticcited by 1,076
  7. 2020Prevalence of Neural Collapse during the terminal phase of deep learning trainingcited by 713
  8. 2018Gradient Descent Provably Optimizes Over-parameterized Neural Networkscited by 655
  9. 2019Linear Mode Connectivity and the Lottery Ticket Hypothesiscited by 627
  10. 2019Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networkscited by 562
  11. 2020What is being transferred in transfer learning?cited by 365
  12. 2019Gradient Descent Maximizes the Margin of Homogeneous Neural Networkscited by 308
28 more
  1. 2018On the Optimization of Deep Networks: Implicit Acceleration by Overparameterizationcited by 280
  2. 2023Rewarded soups: towards Pareto-optimal alignment by interpolating weights fine-tuned on diverse rewardscited by 232
  3. 2022Patching open-vocabulary models by interpolating weightscited by 187
  4. 2018Gradient descent aligns the layers of deep linear networkscited by 182
  5. 2018Algorithmic Regularization in Learning Deep Homogeneous Models: Layers are Automatically Balancedcited by 172
  6. 2020Directional convergence and alignment in deep learningcited by 172
  7. 2020A Simple Convergence Proof of Adam and Adagradcited by 169
  8. 2022Diverse Weight Averaging for Out-of-Distribution Generalizationcited by 148
  9. 2024WARM: On the Benefits of Weight Averaged Reward Modelscited by 145
  10. 2018Regularization Matters: Generalization and Optimization of Neural Nets v.s. their Induced Kernelcited by 131
  11. 2019Polylogarithmic width suffices for gradient descent to achieve arbitrarily small test error with shallow ReLU networkscited by 120
  12. 2023DiLoCo: Distributed Low-Communication Training of Language Modelscited by 115
  13. 2022Inducing Neural Collapse in Imbalanced Learning: Do We Really Need a Learnable Classifier at the End of Deep Neural Network?cited by 100
  14. 2023Convergence of Adam Under Relaxed Assumptionscited by 97
  15. 2022On the Implicit Bias in Deep-Learning Algorithmscited by 84
  16. 2021Saddle-to-Saddle Dynamics in Deep Linear Networks: Small Initialization Training, Symmetry, and Sparsitycited by 83
  17. 2022Gradient flow dynamics of shallow ReLU networks for square loss and orthogonal inputscited by 74
  18. 2022Implicit Regularization Towards Rank Minimization in ReLU Networkscited by 53
  19. 2022Implicit Bias in Leaky ReLU Networks Trained on High-Dimensional Datacited by 46
  20. 2024WARP: On the Benefits of Weight Averaged Rewarded Policiescited by 42
  21. 2020Regularization Matters: A Nonparametric Perspective on Overparametrized Neural Networkcited by 38
  22. 2021Oracle Complexity in Nonsmooth Nonconvex Optimizationcited by 38
  23. 2022Implicit Bias of Large Depth Networks: a Notion of Rank for Nonlinear Functionscited by 38
  24. 2024Approaching Deep Learning through the Spectral Dynamics of Weightscited by 29
  25. 2023Early Neuron Alignment in Two-layer ReLU Networks with Small Initializationcited by 28
  26. 2022Momentum-based Weight Interpolation of Strong Zero-Shot Models for Continual Learningcited by 19
  27. 2022SGD and Weight Decay Secretly Minimize the Rank of Your Neural Networkcited by 17
  28. 2024Towards Better Generalization: Weight Decay Induces Low-rank Bias for Neural Networkscited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 4.

4 papers.

  1. 2026Deciphering Two Training Clocks in Grokking via Deep Linear Network Theory with Conditional ReLU Reductioncited by 1
  2. 2026Cascading Through the Hierarchy: Regularizer-Induced Feature Detection as Phase Transitions in Deep Linear Neural Networkscited by 0
  3. 2026Manifold Projection and Iterative Autoencoder Refinement for Masked Language Modelingcited by 0
  4. 2026Learning Functional Subspaces for Neural Network Compressioncited by 0