Algorithmic Regularization in Tensor Optimization: Towards a Lifted Approach in Matrix Sensing
Not on the map. Why?
Abstract (arXiv)
Gradient descent (GD) is crucial for generalization in machine learning models, as it induces implicit regularization, promoting compact representations. In this work, we examine the role of GD in inducing implicit regularization for tensor optimization, particularly within the context of the lifted matrix sensing framework. This framework has been recently proposed to address the non-convex matrix sensing problem by transforming spurious solutions into strict saddles when optimizing over symmetric, rank-1 tensors. We show that, with sufficiently small initialization scale, GD applied to this lifted problem results in approximate rank-1 tensors and critical points with escape directions. Our findings underscore the significance of the tensor parametrization of matrix sensing, in combination with first-order methods, in achieving global optimality in such problems.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 22 of its 55 references that are in the map. First posted on arXiv 24 Oct 2023.
Builds on
22 of 55 references.
References · 22
- 2014Adam: A Method for Stochastic Optimizationcited by 36,960
- 2021Small random initialization is akin to spectral learning: Optimization and generalization guarantees for overparameterized low-rank matrix reconstructioncited by 77
- 2019Low-Rank Matrix Completion: A Contemporary Surveycited by 43
- 2021Implicit Regularization in Tensor Factorizationcited by 43
- 2023Understanding Incremental Learning of Gradient Descent: A Fine-grained Analysis of Matrix Sensingcited by 40
- 2021On the computational and statistical complexity of over-parameterized matrix sensingcited by 31
- 2016Nonconvex phase synchronizationcited by 30
- 2022Implicit Regularization in Hierarchical Tensor Factorization and Deep Convolutional Neural Networkscited by 27
- 2019Sharp Restricted Isometry Bounds for the Inexistence of Spurious Local Minima in Nonconvex Matrix Recoverycited by 26
- 2022Global Convergence of Sub-gradient Method for Robust Matrix Recovery: Small Initialization, Noisy Measurements, and Over-parameterizationcited by 21
- 2021Understanding Deflation Process in Over-parametrized Tensor Decompositioncited by 20
- 2022The effect of smooth parametrizations on nonconvex optimization landscapescited by 20
10 more
- 2018Exact Guarantees on the Absence of Spurious Local Minima for Non-negative Rank-1 Robust Principal Component Analysiscited by 13
- 2022Improved Global Guarantees for the Nonconvex Burer--Monteiro Factorization via Rank Overparameterizationcited by 13
- 2021General Low-rank Matrix Optimization: Geometric Analysis and Sharper Boundscited by 12
- 2022Estimation Contracts for Outlier-Robust Geometric Perceptioncited by 11
- 2020How Many Samples is a Good Initial Point Worth in Low-rank Matrix Recovery?cited by 10
- 2021Sharp Restricted Isometry Property Bounds for Low-rank Matrix Recovery Problems with Corrupted Measurementscited by 7
- 2022Noisy Low-rank Matrix Optimization: Geometry of Local Minima and Convergence Ratecited by 7
- 2017Conic Relaxations for Power System State Estimation with Line Measurementscited by 5
- 2022Semidefinite Programming versus Burer-Monteiro Factorization for Matrix Sensingcited by 5
- 2023Over-parametrization via Lifting for Low-rank Matrix Sensing: Conversion of Spurious Solutions to Strict Saddle Pointscited by 5
Built on by
Citing papers per year: 2026: 1.
1 paper.