mlsanityData to 7 Oct 2026

Implicit Regularization in Tensor Factorization

arXiv 2102.09972 ↗First posted 19 Feb 2021cs.LG cs.AI cs.NEcited by 43

Not on the map. Why?

Abstract (arXiv)

Recent efforts to unravel the mystery of implicit regularization in deep learning have led to a theoretical focus on matrix factorization -- matrix completion via linear neural network. As a step further towards practical deep learning, we provide the first theoretical analysis of implicit regularization in tensor factorization -- tensor completion via certain type of non-linear neural network. We circumvent the notorious difficulty of tensor problems by adopting a dynamical systems perspective, and characterizing the evolution induced by gradient descent. The characterization suggests a form of greedy low tensor rank search, which we rigorously prove under certain conditions, and empirically demonstrate under others. Motivated by tensor rank capturing the implicit regularization of a non-linear neural network, we empirically explore it as a measure of complexity, and find that it captures the essence of datasets on which neural networks generalize. This leads us to believe that tensor rank may pave way to explaining both implicit regularization in deep learning, and the properties of real-world data translating this implicit regularization to generalization.

Key facts. Cited by 43 papers in the map (data through 7 Oct 2026). Builds on 54 of its 82 references that are in the map. First posted on arXiv 19 Feb 2021.

Builds on

54 of 82 references.

References · 54

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2017Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithmscited by 4,283
  3. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  4. 2017The Implicit Bias of Gradient Descent on Separable Datacited by 649
  5. 2019Implicit Regularization in Deep Matrix Factorizationcited by 402
  6. 2017Implicit Regularization in Matrix Factorizationcited by 318
  7. 2020Kernel and Rich Regimes in Overparametrized Modelscited by 302
  8. 2018On the Optimization of Deep Networks: Implicit Acceleration by Overparameterizationcited by 280
  9. 2020Implicit Bias of Gradient Descent for Wide Two-layer Neural Networks Trained with the Logistic Losscited by 267
  10. 2018Characterizing Implicit Bias in Terms of Optimization Geometrycited by 259
  11. 2018Implicit Bias of Gradient Descent on Linear Convolutional Networkscited by 251
  12. 2017High-dimensional dynamics of generalization error in neural networkscited by 238
38 more
  1. 2019Mean-field theory of two-layers neural networks: dimension-free bounds and kernel limitcited by 187
  2. 2018Gradient descent aligns the layers of deep linear networkscited by 182
  3. 2018Algorithmic Regularization in Learning Deep Homogeneous Models: Layers are Automatically Balancedcited by 172
  4. 2018Nonconvex Optimization Meets Low-Rank Matrix Factorization: An Overviewcited by 159
  5. 2020Towards Resolving the Implicit Bias of Gradient Descent for Matrix Factorization: Greedy Low-Rank Learningcited by 148
  6. 2019Implicit Regularization of Discrete Gradient Dynamics in Linear Neural Networkscited by 122
  7. 2020Implicit Regularization in Deep Learning May Not Be Explainable by Normscited by 118
  8. 2019Dynamics of stochastic gradient descent for two-layer neural networks in the teacher-student setupcited by 112
  9. 2018Convergence of Gradient Descent on Separable Datacited by 108
  10. 2015Low-rank Solutions of Linear Matrix Equations via Procrustes Flowcited by 97
  11. 2015On the Expressive Power of Deep Learning: A Tensor Analysiscited by 97
  12. 2018Overparameterized Nonlinear Learning: Gradient Descent Takes the Shortest Path?cited by 87
  13. 2018An analytic theory of generalization dynamics and transfer learning in deep linear networkscited by 84
  14. 2020A Unifying View on Implicit Bias in Training Linear Neural Networkscited by 74
  15. 2019The Implicit Bias of Depth: How Incremental Learning Drives Generalizationcited by 64
  16. 2020The Implicit Regularization of Stochastic Gradient Flow for Least Squarescited by 58
  17. 2016Inductive Bias of Deep Convolutional Networks through Pooling Geometrycited by 53
  18. 2018Quantum Entanglement in Deep Learning Architecturescited by 49
  19. 2017Tensor Completion Algorithms in Big Data Analyticscited by 48
  20. 2020Implicit Regularization in ReLU Networks with the Square Losscited by 45
  21. 2018Understanding training and generalization in deep learning by Fourier analysiscited by 43
  22. 2020Gradient Descent for Deep Matrix Factorization: Dynamics and Implicit Bias towards Low Rankcited by 42
  23. 2020The Depth-to-Width Interplay in Self-Attentioncited by 39
  24. 2019Nonconvex Low-Rank Tensor Completion from Noisy Datacited by 35
  25. 2015Smooth PARAFAC Decomposition for Tensor Completioncited by 34
  26. 2016Convolutional Rectifier Networks as Generalized Tensor Decompositionscited by 32
  27. 2014Provable Tensor Factorization with Missing Datacited by 31
  28. 2017Expressive power of recurrent neural networkscited by 30
  29. 2017On Polynomial Time Methods for Exact Low Rank Tensor Completioncited by 27
  30. 2017Deep Learning and Quantum Entanglement: Fundamental Connections with Implications to Network Designcited by 22
  31. 2019Implicit Regularization and Convergence for Weight Normalizationcited by 20
  32. 2020Beyond Lazy Training for Over-parameterized Tensor Decompositioncited by 15
  33. 2017On the Long-Term Memory of Deep Recurrent Networkscited by 14
  34. 2017Analysis and Design of Convolutional Networks via Hierarchical Tensor Decompositionscited by 13
  35. 2020Understanding Generalization in Deep Learning via Tensor Methodscited by 13
  36. 2015Deep SimNetscited by 10
  37. 2019Generalized Tensor Models for Recurrent Neural Networkscited by 8
  38. 2014SimNets: A Generalization of Convolutional Networkscited by 5

50 most cited of 54 All 54 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 5; 2022: 14; 2023: 14; 2024: 6; 2025: 3; 2026: 1.

43 papers.

Sort:

By area:

  1. 2022On the Implicit Bias in Deep-Learning Algorithmscited by 84
  2. 2021Gradient Descent on Two-layer Nets: Margin Maximization and Simplicity Biascited by 82
  3. 2023Saddle-to-Saddle Dynamics in Diagonal Linear Networkscited by 54
  4. 2022Implicit Regularization Towards Rank Minimization in ReLU Networkscited by 53
  5. 2023Understanding Incremental Learning of Gradient Descent: A Fine-grained Analysis of Matrix Sensingcited by 40
  6. 2021Continuous vs. Discrete Optimization of Deep Neural Networkscited by 34
  7. 2021ZerO Initialization: Initializing Neural Networks with only Zeros and Onescited by 28
  8. 2022Implicit Regularization in Hierarchical Tensor Factorization and Deep Convolutional Neural Networkscited by 27
  9. 2022Incremental Learning in Diagonal Linear Networkscited by 27
  10. 2023Graph Neural Networks Use Graphs When They Shouldn'tcited by 25
  11. 2021More is Less: Inducing Sparsity via Overparameterizationcited by 22
  12. 2021Understanding Deflation Process in Over-parametrized Tensor Decompositioncited by 20
31 more
  1. 2024Neural Collapse versus Low-rank Bias: Is Deep Neural Collapse Really Optimal?cited by 16
  2. 2022The Role of Linear Layers in Nonlinear Interpolating Networkscited by 15
  3. 2022Tensor-on-Tensor Regression: Riemannian Optimization, Over-parameterization, Statistical-computational Gap, and Their Interplaycited by 15
  4. 2025Learning In-context n-grams with Transformers: Sub-n-grams Are Near-stationary Pointscited by 15
  5. 2022Permutation Search of Tensor Network Structures via Local Samplingcited by 13
  6. 2022On the Ability of Graph Neural Networks to Model Interactions Between Verticescited by 12
  7. 2023Robust Implicit Regularization via Weight Normalizationcited by 10
  8. 2023InRank: Incremental Low-Rank Learningcited by 10
  9. 2023PRIOR: Personalized Prior for Reactivating the Information Overlooked in Federated Learningcited by 8
  10. 2022Thinking Outside the Ball: Optimal Learning with Gradient Descent for Generalized Linear Stochastic Convex Optimizationcited by 7
  11. 2022Learning Low Dimensional State Spaces with Overparameterized Recurrent Neural Netscited by 7
  12. 2023ReLU Neural Networks with Linear Layers are Biased Towards Single- and Multi-Index Modelscited by 7
  13. 2023The Inductive Bias of Flatness Regularization for Deep Matrix Factorizationcited by 6
  14. 2025Why is Your Language Model a Poor Implicit Reward Model?cited by 6
  15. 2022How You Start Matters for Generalizationcited by 5
  16. 2022Behind the Scenes of Gradient Descent: A Trajectory Analysis via Basis Function Decompositioncited by 4
  17. 2023Implicit Regularization Leads to Benign Overfitting for Sparse Linear Regressioncited by 4
  18. 2023What Makes Data Suitable for a Locally Connected Neural Network? A Necessary and Sufficient Condition Based on Quantum Entanglementcited by 4
  19. 2023Implicit regularization in AI meets generalized hardness of approximation in optimization -- Sharp results for diagonal linear networkscited by 4
  20. 2024Implicit Bias of Policy Gradient in Linear Quadratic Control: Extrapolation to Unseen Initial Statescited by 4
  21. 2022Implicit Regularization with Polynomial Growth in Deep Tensor Factorizationcited by 3
  22. 2022TTML: tensor trains for general supervised machine learningcited by 2
  23. 2023Implicit Regularization Makes Overparameterized Asymmetric Matrix Sensing Robust to Perturbationscited by 2
  24. 2024Improving Adaptivity via Over-Parameterization in Sequence Modelscited by 2
  25. 2024Low-Dimension-to-High-Dimension Generalization And Its Implications for Length Generalizationcited by 2
  26. 2023Algorithmic Regularization in Tensor Optimization: Towards a Lifted Approach in Matrix Sensingcited by 1
  27. 2024Implicit Regularization via Spectral Neural Networks and Non-linear Matrix Sensingcited by 1
  28. 2024Lecture Notes on Linear Neural Networks: A Tale of Optimization and Generalization in Deep Learningcited by 1
  29. 2026Incremental Learning of Sparse Attention Patterns in Transformerscited by 1
  30. 2023Sparse Mean Estimation in Adversarial Settings via Incremental Learningcited by 0
  31. 2025Tuning-Free Structured Sparse Recovery of Multiple Measurement Vectors using Implicit Regularizationcited by 0