mlsanityData to 7 Oct 2026

Understanding Deflation Process in Over-parametrized Tensor Decomposition

arXiv 2106.06573 ↗First posted 11 Jun 2021stat.ML cs.LGcited by 20

Not on the map. Why?

Abstract (arXiv)

In this paper we study the training dynamics for gradient flow on over-parametrized tensor decomposition problems. Empirically, such training process often first fits larger components and then discovers smaller components, which is similar to a tensor deflation process that is commonly used in tensor decomposition algorithms. We prove that for orthogonally decomposable tensor, a slightly modified version of gradient flow would follow a tensor deflation process and recover all the tensor components. Our proof suggests that for orthogonal tensors, gradient flow dynamics works similarly as greedy low-rank learning in the matrix setting, which is a first step towards understanding the implicit regularization effect of over-parametrized models for low-rank tensors.

Key facts. Cited by 20 papers in the map (data through 7 Oct 2026). Builds on 46 of its 56 references that are in the map. First posted on arXiv 11 Jun 2021.

Builds on

46 of 56 references.

References · 46

  1. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  2. 2018A Convergence Theory for Deep Learning via Over-Parameterizationcited by 784
  3. 2018On Lazy Training in Differentiable Programmingcited by 671
  4. 2018Gradient Descent Provably Optimizes Over-parameterized Neural Networkscited by 655
  5. 2017The Implicit Bias of Gradient Descent on Separable Datacited by 649
  6. 2018Gradient Descent Finds Global Minima of Deep Neural Networkscited by 572
  7. 2019On Exact Computation with an Infinitely Wide Neural Netcited by 572
  8. 2019Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networkscited by 562
  9. 2018A Mean Field View of the Landscape of Two-Layers Neural Networkscited by 463
  10. 2018On the Global Convergence of Gradient Descent for Over-parameterized Models using Optimal Transportcited by 422
  11. 2019Implicit Regularization in Deep Matrix Factorizationcited by 402
  12. 2018Learning and Generalization in Overparameterized Neural Networks, Going Beyond Two Layerscited by 340
34 more
  1. 2019Gradient Descent Maximizes the Margin of Homogeneous Neural Networkscited by 308
  2. 2020Kernel and Rich Regimes in Overparametrized Modelscited by 302
  3. 2020Implicit Bias of Gradient Descent for Wide Two-layer Neural Networks Trained with the Logistic Losscited by 267
  4. 2018Learning Overparameterized Neural Networks via Stochastic Gradient Descent on Structured Datacited by 260
  5. 2018Characterizing Implicit Bias in Terms of Optimization Geometrycited by 259
  6. 2018Implicit Bias of Gradient Descent on Linear Convolutional Networkscited by 251
  7. 2018Gradient descent aligns the layers of deep linear networkscited by 182
  8. 2019Towards moderate overparameterization: global convergence guarantees for training shallow neural networkscited by 176
  9. 2020Directional convergence and alignment in deep learningcited by 172
  10. 2019Linearized two-layers neural networks in high dimensioncited by 155
  11. 2020When Do Neural Networks Outperform Kernel Methods?cited by 150
  12. 2020Towards Resolving the Implicit Bias of Gradient Descent for Matrix Factorization: Greedy Low-Rank Learningcited by 148
  13. 2018Stochastic Gradient Descent Optimizes Over-parameterized Deep ReLU Networkscited by 139
  14. 2018Regularization Matters: Generalization and Optimization of Neural Nets v.s. their Induced Kernelcited by 131
  15. 2018Convergence of Gradient Descent on Separable Datacited by 108
  16. 2018Mean Field Analysis of Neural Networks: A Central Limit Theoremcited by 107
  17. 2018Trainability and Accuracy of Neural Networks: An Interacting Particle System Approachcited by 105
  18. 2019On the Power and Limitations of Random Features for Understanding Neural Networkscited by 103
  19. 2019Dynamics of Deep Neural Networks and Neural Tangent Hierarchycited by 96
  20. 2019What Can ResNet Learn Efficiently, Going Beyond Kernels?cited by 91
  21. 2019Limitations of Lazy Training of Two-layers Neural Networkscited by 87
  22. 2018Risk and parameter convergence of logistic regressioncited by 81
  23. 2017Learning One-hidden-layer Neural Networks with Landscape Designcited by 79
  24. 2019Beyond Linearization: On Quadratic and Higher-Order Approximation of Wide Neural Networkscited by 79
  25. 2019Asymptotics of Wide Networks from Feynman Diagramscited by 77
  26. 2020Implicit Bias in Deep Linear Classification: Initialization Scale vs Training Accuracycited by 70
  27. 2020A Rigorous Framework for the Mean Field Limit of Multilayer Neural Networkscited by 68
  28. 2017Stochastic Particle Gradient Descent for Infinite Ensemblescited by 52
  29. 2019Sparse Optimization on Measures with Over-parameterized Gradient Descentcited by 50
  30. 2020Towards Understanding Hierarchical Learning: Benefits of Neural Representationscited by 46
  31. 2021Implicit Regularization in Tensor Factorizationcited by 43
  32. 2020Beyond Lazy Training for Over-parameterized Tensor Decompositioncited by 15
  33. 2020Taylorized Training: Towards Better Approximation of Neural Network Training at Finite Widthcited by 11
  34. 2019Characterizing the implicit bias via a primal-dual analysiscited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 9; 2023: 4; 2024: 3; 2025: 2; 2026: 2.

20 papers.

Sort:

  1. 2022Understanding the Generalization Benefit of Normalization Layers: Sharpness Reductioncited by 86
  2. 2022Implicit Bias of Gradient Descent on Reparametrized Models: On Equivalence to Mirror Descentcited by 38
  3. 2022Implicit Regularization in Hierarchical Tensor Factorization and Deep Convolutional Neural Networkscited by 27
  4. 2022Incremental Learning in Diagonal Linear Networkscited by 27
  5. 2022Tensor-on-Tensor Regression: Riemannian Optimization, Over-parameterization, Statistical-computational Gap, and Their Interplaycited by 15
  6. 2024Simplicity Bias of Two-Layer Networks beyond Linearly Separable Datacited by 11
  7. 2025Alternating Gradient Flows: A Theory of Feature Learning in Two-layer Neural Networkscited by 11
  8. 2022Plateau in Monotonic Linear Interpolation -- A "Biased" View of Loss Landscape for Deep Networkscited by 10
  9. 2024Convergence of Gradient Descent with Small Initialization for Unregularized Matrix Completioncited by 6
  10. 2023A Quadratic Synchronization Rule for Distributed Deep Learningcited by 5
  11. 2022Behind the Scenes of Gradient Descent: A Trajectory Analysis via Basis Function Decompositioncited by 4
  12. 2023Implicit Regularization Leads to Benign Overfitting for Sparse Linear Regressioncited by 4
8 more
  1. 2023What Makes Data Suitable for a Locally Connected Neural Network? A Necessary and Sufficient Condition Based on Quantum Entanglementcited by 4
  2. 2024How Does Gradient Descent Learn Features -- A Local Analysis for Regularized Two-Layer Neural Networkscited by 4
  3. 2022Implicit Regularization with Polynomial Growth in Deep Tensor Factorizationcited by 3
  4. 2025On the Benefits of Weight Normalization for Overparameterized Matrix Sensingcited by 3
  5. 2026Low-Rank Adaptation Redux for Large Modelscited by 2
  6. 2022Multivariate Analysis for Multiple Network Data via Semi-Symmetric Tensor PCAcited by 1
  7. 2023Algorithmic Regularization in Tensor Optimization: Towards a Lifted Approach in Matrix Sensingcited by 1
  8. 2026Direction-Magnitude Decomposition for Low-Rank Matrix Optimization: Faster Convergence and Saddle-to-saddle Dynamicscited by 0