mlsanityData to 7 Oct 2026

On the Benefits of Weight Normalization for Overparameterized Matrix Sensing

arXiv 2510.01175 ↗First posted 1 Oct 2025cs.LG eess.SP math.OCcited by 3

Not on the map. Why?

Abstract (arXiv)

While normalization techniques are widely used in deep learning, their theoretical understanding remains relatively limited. In this work, we establish the benefits of (generalized) weight normalization (WN) applied to the overparameterized matrix sensing problem. We prove that WN with Riemannian optimization achieves linear convergence, yielding an exponential speedup over standard methods that do not use WN. Our analysis further demonstrates that both iteration and sample complexity improve polynomially as the level of overparameterization increases. To the best of our knowledge, this work provides the first characterization of how WN leverages overparameterization for faster convergence in matrix sensing.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 31 of its 54 references that are in the map. First posted on arXiv 1 Oct 2025.

Builds on

31 of 54 references.

References · 31

  1. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  2. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
  3. 2016Layer Normalizationcited by 5,579
  4. 2024DoRA: Weight-Decomposed Low-Rank Adaptationcited by 740
  5. 2016Weight Normalization: A Simple Reparameterization to Accelerate Training of Deep Neural Networkscited by 541
  6. 2019Implicit Regularization in Deep Matrix Factorizationcited by 402
  7. 2020Towards Resolving the Implicit Bias of Gradient Descent for Matrix Factorization: Greedy Low-Rank Learningcited by 148
  8. 2021Small random initialization is akin to spectral learning: Optimization and generalization guarantees for overparameterized low-rank matrix reconstructioncited by 77
  9. 2020Accelerating Ill-Conditioned Low-Rank Matrix Estimation via Scaled Gradient Descentcited by 70
  10. 2021Global Convergence of Gradient Descent for Asymmetric Low-Rank Matrix Factorizationcited by 44
  11. 2023Understanding Incremental Learning of Gradient Descent: A Fine-grained Analysis of Matrix Sensingcited by 40
  12. 2023The Power of Preconditioning in Overparameterized Low-Rank Matrix Sensingcited by 32
19 more
  1. 2021On the computational and statistical complexity of over-parameterized matrix sensingcited by 31
  2. 2021Scaling and Scalability: Provable Nonconvex Low-Rank Tensor Estimation from Incomplete Measurementscited by 28
  3. 2014Optimization Techniques on Riemannian Manifoldscited by 27
  4. 2023Over-Parameterization Exponentially Slows Down Gradient Descent for Learning a Single Neuroncited by 25
  5. 2019Implicit Regularization and Convergence for Weight Normalizationcited by 20
  6. 2021Understanding Deflation Process in Over-parametrized Tensor Decompositioncited by 20
  7. 2022The effect of smooth parametrizations on nonconvex optimization landscapescited by 20
  8. 2025PoLAR: Polar-Decomposed Low-Rank Adapter Representationcited by 19
  9. 2023How Over-Parameterization Slows Down Gradient Descent in Matrix Sensing: The Curses of Symmetry and Initializationcited by 18
  10. 2024On the Crucial Role of Initialization for Matrix Factorizationcited by 15
  11. 2023Convergence of Alternating Gradient Descent for Matrix Factorizationcited by 12
  12. 2025Preconditioned Gradient Descent for Over-Parameterized Nonconvex Matrix Factorizationcited by 12
  13. 2023Robust Implicit Regularization via Weight Normalizationcited by 10
  14. 2021Factorization Approach for Low-complexity Matrix Completion Problems: Exponential Number of Spurious Solutions and Failure of Gradient Methodscited by 6
  15. 2023Provably Accelerating Ill-Conditioned Low-rank Estimation via Scaled Gradient Descent, Even with Overparameterizationcited by 6
  16. 2024Convergence of Gradient Descent with Small Initialization for Unregularized Matrix Completioncited by 6
  17. 2023Optimizing Orthogonalized Tensor Deflation via Random Tensor Theorycited by 4
  18. 2024Optimization and Generalization Guarantees for Weight Normalizationcited by 2
  19. 2025Global Convergence of Gradient EM for Over-Parameterized Gaussian Mixturescited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026Low-Rank Adaptation Redux for Large Modelscited by 2
  2. 2026Escaping Local Minima Provably in Non-convex Matrix Sensing: A Deterministic Framework via Simulated Liftingcited by 1
  3. 2026Direction-Magnitude Decomposition for Low-Rank Matrix Optimization: Faster Convergence and Saddle-to-saddle Dynamicscited by 0