Gradient flow on extensive-rank positive semi-definite matrix denoising
Not on the map. Why?
Abstract (arXiv)
In this work, we present a new approach to analyze the gradient flow for a positive semi-definite matrix denoising problem in an extensive-rank and high-dimensional regime. We use recent linear pencil techniques of random matrix theory to derive fixed point equations which track the complete time evolution of the matrix-mean-square-error of the problem. The predictions of the resulting fixed point equations are validated by numerical experiments. In this short note we briefly illustrate a few predictions of our formalism by way of examples, and in particular we uncover continuous phase transitions in the extensive-rank and high-dimensional regime, which connect to the classical phase transitions of the low-rank problem in the appropriate limit. The formalism has much wider applicability than shown in this communication.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 18 of its 37 references that are in the map. First posted on arXiv 16 Mar 2023.
Builds on
18 of 37 references.
References · 18
- 2013Exact solutions to the nonlinear dynamics of learning in deep linear neural networkscited by 748
- 2017Implicit Regularization in Matrix Factorizationcited by 318
- 2018Nonconvex Optimization Meets Low-Rank Matrix Factorization: An Overviewcited by 159
- 2020The Neural Tangent Kernel in High Dimensions: Triple Descent and a Multi-Scale Theory of Generalizationcited by 101
- 2014A statistical model for tensor PCAcited by 72
- 2022High-dimensional limit theorems for SGD: Effective dynamics and critical scalingcited by 69
- 2020Gradient Descent for Deep Matrix Factorization: Dynamics and Implicit Bias towards Low Rankcited by 42
- 2017Estimation of Low-Rank Matrices via Approximate Message Passingcited by 41
- 2019Passed & Spurious: Descent Algorithms and Local Minima in Spiked Matrix-Tensor Modelscited by 28
- 2018Harnessing Structures in Big Data via Guaranteed Low-Rank Matrix Estimationcited by 25
- 2014Phase transitions and sample complexity in Bayes-optimal matrix factorizationcited by 22
- 2021Model, sample, and epoch-wise descents: exact solution of gradient flow in the random feature modelcited by 17
6 more
- 2021Statistical limits of dictionary learning: random matrix theory and the spectral replica methodcited by 10
- 2022The price of ignorance: how much does it cost to forget noise structure in low-rank matrix estimation?cited by 9
- 2022Matrix factorization with neural networkscited by 6
- 2021Rank-one matrix estimation: analytic time evolution of gradient descent dynamicscited by 5
- 2022Gradient flow in the gaussian covariate model: exact solution of learning curves and multiple descent structurescited by 5
- 2022High-dimensional Asymptotics of Langevin Dynamics in Spiked Matrix Modelscited by 2
Built on by
Citing papers per year: 2024: 1.
1 paper.