A Dynamics Theory of Implicit Regularization in Deep Low-Rank Matrix Factorization
Not on the map. Why?
Abstract (arXiv)
Implicit regularization is an important way to interpret neural networks. Recent theory starts to explain implicit regularization with the model of deep matrix factorization (DMF) and analyze the trajectory of discrete gradient dynamics in the optimization process. These discrete gradient dynamics are relatively small but not infinitesimal, thus fitting well with the practical implementation of neural networks. Currently, discrete gradient dynamics analysis has been successfully applied to shallow networks but encounters the difficulty of complex computation for deep networks. In this work, we introduce another discrete gradient dynamics approach to explain implicit regularization, i.e. landscape analysis. It mainly focuses on gradient regions, such as saddle points and local minima. We theoretically establish the connection between saddle point escaping (SPE) stages and the matrix rank in DMF. We prove that, for a rank-R matrix reconstruction, DMF will converge to a second-order critical point after R stages of SPE. This conclusion is further experimentally verified on a low-rank matrix reconstruction problem. This work provides a new theory to analyze implicit regularization in deep learning.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 40 of its 57 references that are in the map. First posted on arXiv 29 Dec 2022.
Builds on
40 of 57 references.
References · 40
- 2016An overview of gradient descent optimization algorithmscited by 1,021
- 2019On the Convergence of Adam and Beyondcited by 834
- 2013Exact solutions to the nonlinear dynamics of learning in deep linear neural networkscited by 748
- 2020Time Series Forecasting With Deep Learning: A Surveycited by 457
- 2019Implicit Regularization in Deep Matrix Factorizationcited by 402
- 2014In Search of the Real Inductive Bias: On the Role of Implicit Regularization in Deep Learningcited by 339
- 2017How to Escape Saddle Points Efficientlycited by 284
- 2018On the Optimization of Deep Networks: Implicit Acceleration by Overparameterizationcited by 280
- 2017Recent Trends in Deep Learning Based Natural Language Processingcited by 262
- 2015Escaping From Saddle Points --- Online Stochastic Gradient for Tensor Decompositioncited by 241
- 2016Deep Learning without Poor Local Minimacited by 233
- 2018A mathematical theory of semantic development in deep neural networkscited by 182
28 more
- 2018A Convergence Analysis of Gradient Descent for Deep Linear Neural Networkscited by 180
- 2020Towards Resolving the Implicit Bias of Gradient Descent for Matrix Factorization: Greedy Low-Rank Learningcited by 148
- 2019Implicit Regularization of Discrete Gradient Dynamics in Linear Neural Networkscited by 122
- 2020Implicit Regularization in Deep Learning May Not Be Explainable by Normscited by 118
- 2019Attention in Natural Language Processingcited by 100
- 2019Deep Learning vs. Traditional Computer Visioncited by 94
- 2021Saddle-to-Saddle Dynamics in Deep Linear Networks: Small Initialization Training, Symmetry, and Sparsitycited by 83
- 2019Theoretical Issues in Deep Networks: Approximation, Optimization and Generalizationcited by 72
- 2018Escaping Saddles with Stochastic Gradientscited by 66
- 2019The Implicit Bias of Depth: How Incremental Learning Drives Generalizationcited by 64
- 2019Learning deep linear neural networks: Riemannian gradient flows and convergence to global minimizerscited by 61
- 2019Width Provably Matters in Optimization for Deep Linear Neural Networkscited by 52
- 2020Gradient Descent for Deep Matrix Factorization: Dynamics and Implicit Bias towards Low Rankcited by 42
- 2019Escaping Saddle Points with Adaptive Gradient Methodscited by 34
- 2017Global optimality conditions for deep neural networkscited by 31
- 2021One-dimensional Deep Low-rank and Sparse Network for Accelerated MRIcited by 27
- 2021The loss landscape of deep linear neural networks: a second-order analysiscited by 24
- 2016Efficient approaches for escaping higher order saddle points in non-convex optimizationcited by 23
- 2022Physics-driven Synthetic Data Learning for Biomedical Magnetic Resonancecited by 21
- 2020Review and Prospect: Deep Learning in Nuclear Magnetic Resonance Spectroscopycited by 17
- 2021Accelerated MRI Reconstruction with Separable and Enhanced Low-Rank Hankel Regularizationcited by 17
- 2019Accelerated Nuclear Magnetic Resonance Spectroscopy with Deep Learningcited by 16
- 2019Image Reconstruction with Low-rankness and Self-consistency of k-space Data in Parallel MRIcited by 12
- 2020A Sparse Model-inspired Deep Thresholding Network for Exponential Signal Reconstruction -- Application in Fast Biological Spectroscopycited by 12
- 2018Generalization and Expressivity for Deep Netscited by 11
- 2020Exponential Signal Reconstruction with Deep Hankel Matrix Factorizationcited by 8
- 2018Vandermonde Factorization of Hankel Matrix for Complex Exponential Signal Recovery -- Application in Fast NMR Spectroscopycited by 3
- 2017Are Saddles Good Enough for Deep Learning?cited by 2
Built on by
Citing papers per year: 2026: 1.
1 paper.