mlsanityData to 7 Oct 2026

A Complete Loss Landscape Analysis of Regularized Deep Matrix Factorization

arXiv 2506.20344 ↗First posted 25 Jun 2025math.OC cs.LGcited by 1

Not on the map. Why?

Abstract (arXiv)

Despite its wide range of applications across various domains, the optimization foundations of deep matrix factorization (DMF) remain largely open. In this work, we aim to fill this gap by conducting a comprehensive study of the loss landscape of the regularized DMF problem. Toward this goal, we first provide a closed-form characterization of all critical points of the problem. Building on this, we establish precise conditions under which a critical point is a local minimizer, a global minimizer, a strict saddle point, or a non-strict saddle point. Leveraging these results, we derive a necessary and sufficient condition under which every critical point is either a local minimizer or a strict saddle point. This provides insights into why gradient-based methods almost always converge to a local minimizer of the regularized DMF problem. Finally, we conduct numerical experiments to visualize its loss landscape to support our theory.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 46 of its 54 references that are in the map. First posted on arXiv 25 Jun 2025.

Builds on

46 of 54 references.

References · 46

  1. 2017Visualizing the Loss Landscape of Neural Netscited by 1,126
  2. 2017Neural Collaborative Filteringcited by 1,108
  3. 2013Exact solutions to the nonlinear dynamics of learning in deep linear neural networkscited by 748
  4. 2018Gradient Descent Provably Optimizes Over-parameterized Neural Networkscited by 655
  5. 2019Implicit Regularization in Deep Matrix Factorizationcited by 402
  6. 2015Escaping From Saddle Points --- Online Stochastic Gradient for Tensor Decompositioncited by 241
  7. 2016Deep Learning without Poor Local Minimacited by 233
  8. 2021A Geometric Analysis of Neural Collapse with Unconstrained Featurescited by 226
  9. 2014Qualitatively characterizing neural network optimization problemscited by 186
  10. 2018A Convergence Analysis of Gradient Descent for Deep Linear Neural Networkscited by 180
  11. 2017No Spurious Local Minima in Nonconvex Low Rank Problems: A Unified Geometric Analysiscited by 127
  12. 2016Topology and Geometry of Half-Rectified Network Optimizationcited by 102
34 more
  1. 2022On the Optimization Landscape of Neural Collapse under MSE Loss: Global Optimality with Unconstrained Featurescited by 97
  2. 2016A Geometric Analysis of Phase Retrievalcited by 92
  3. 2016Global Optimality of Local Search for Low Rank Matrix Recoverycited by 81
  4. 2017The loss surface of deep and wide neural networkscited by 74
  5. 2018Escaping Saddles with Stochastic Gradientscited by 66
  6. 2017Deep linear neural networks with arbitrary loss: All local minima are globalcited by 64
  7. 2020The Global Landscape of Neural Networks: An Overviewcited by 57
  8. 2022Neural Collapse with Normalized Features: A Geometric Analysis over the Riemannian Manifoldcited by 50
  9. 2015Complete Dictionary Recovery over the Sphere I: Overview and the Geometric Picturecited by 49
  10. 2018Deep Network Interpolation for Continuous Imagery Effect Transitioncited by 47
  11. 2019Loss Landscapes of Regularized Linear Autoencoderscited by 45
  12. 2020Gradient Descent for Deep Matrix Factorization: Dynamics and Implicit Bias towards Low Rankcited by 42
  13. 2017Global Optimality in Low-rank Matrix Optimizationcited by 35
  14. 2019Pure and Spurious Critical Points: a Geometric Study of Linear Networkscited by 33
  15. 2023Understanding Deep Representation Learning via Layerwise Feature Compression and Discriminationcited by 31
  16. 2017Depth Creates No Bad Local Minimacited by 29
  17. 2023Implicit bias of SGD in L2-regularized linear DNNs: One-way jumps from high to low rankcited by 28
  18. 2015A deep matrix factorization method for learning attribute representationscited by 24
  19. 2021The loss landscape of deep linear neural networks: a second-order analysiscited by 24
  20. 2015A Unified Approach to Error Bounds for Structured Convex Optimization Problemscited by 21
  21. 2022On the Effective Number of Linear Regions in Shallow Univariate ReLU Networks: Convergence Guarantees and Implicit Biascited by 21
  22. 2018The loss surface of deep linear networks viewed through the algebraic geometry lenscited by 20
  23. 2020Piecewise linear activations substantially shape the loss surfaces of neural networkscited by 19
  24. 2024A Global Geometric Analysis of Maximal Coding Rate Reductioncited by 13
  25. 2018On the Landscape of Synchronization Networks: A Perspective from Nonconvex Optimizationcited by 11
  26. 2018Learning Deep Models: Critical Points and Local Opennesscited by 10
  27. 2020Solving Orthogonal Group Synchronization via Convex and Low-Rank Optimization: Tightness and Landscape Analysiscited by 10
  28. 2020Optimization Landscape of Tucker Decompositioncited by 10
  29. 2021Spurious Local Minima Are Common for Deep Neural Networks with Piecewise Linear Activationscited by 7
  30. 2023Benign landscapes of low-dimensional relaxations for orthogonal synchronization on general graphscited by 6
  31. 2016A Unified Convex Surrogate for the Schatten-p Normcited by 3
  32. 2019Error bound of critical points and KL property of exponent 1/2 for squared F-norm regularized factorizationcited by 3
  33. 2023Local Geometry Determines Global Landscape in Low-rank Factorization for Synchronizationcited by 3
  34. 2025Error Bound Analysis for the Regularized Loss of Deep Linear Neural Networkscited by 1

Built on by

Cited by year

Citing papers per year: 2025: 1.

1 paper.

  1. 2025Gradient Descent with Large Step Sizes: Chaos and Fractal Convergence Regioncited by 5