mlsanityData to 14 Oct 2026

Implicit Regularization in Hierarchical Tensor Factorization and Deep Convolutional Neural Networks

arXiv 2201.11729 ↗First posted 27 Jan 2022cs.LG cs.AI cs.NEcited by 27

Not on the map. Why?

Abstract (arXiv)

In the pursuit of explaining implicit regularization in deep learning, prominent focus was given to matrix and tensor factorizations, which correspond to simplified neural networks. It was shown that these models exhibit an implicit tendency towards low matrix and tensor ranks, respectively. Drawing closer to practical deep learning, the current paper theoretically analyzes the implicit regularization in hierarchical tensor factorization, a model equivalent to certain deep convolutional neural networks. Through a dynamical systems lens, we overcome challenges associated with hierarchy, and establish implicit regularization towards low hierarchical tensor rank. This translates to an implicit regularization towards locality for the associated convolutional networks. Inspired by our theory, we design explicit regularization discouraging locality, and demonstrate its ability to improve the performance of modern convolutional networks on non-local tasks, in defiance of conventional wisdom by which architectural changes are needed. Our work highlights the potential of enhancing neural networks via theoretical analysis of their implicit regularization.

Key facts. Cited by 27 papers in the map (data through 14 Oct 2026). Builds on 72 of its 106 references that are in the map. First posted on arXiv 27 Jan 2022.

Builds on

72 of 106 references.

References · 72

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2016Understanding deep learning requires rethinking generalizationcited by 1,674
  3. 2016Temporal Segment Networks: Towards Good Practices for Deep Action Recognitioncited by 902
  4. 2013Exact solutions to the nonlinear dynamics of learning in deep linear neural networkscited by 763
  5. 2020Long Range Arena: A Benchmark for Efficient Transformerscited by 700
  6. 2017The Implicit Bias of Gradient Descent on Separable Datacited by 655
  7. 2019Implicit Regularization in Deep Matrix Factorizationcited by 405
  8. 2017Implicit Regularization in Matrix Factorizationcited by 322
  9. 2019Gradient Descent Maximizes the Margin of Homogeneous Neural Networkscited by 320
  10. 2020Kernel and Rich Regimes in Overparametrized Modelscited by 306
  11. 2018On the Optimization of Deep Networks: Implicit Acceleration by Overparameterizationcited by 284
  12. 2020Implicit Bias of Gradient Descent for Wide Two-layer Neural Networks Trained with the Logistic Losscited by 268
38 more
  1. 2018Implicit Bias of Gradient Descent on Linear Convolutional Networkscited by 251
  2. 2018Gradient descent aligns the layers of deep linear networkscited by 182
  3. 2020Directional convergence and alignment in deep learningcited by 178
  4. 2018Algorithmic Regularization in Learning Deep Homogeneous Models: Layers are Automatically Balancedcited by 177
  5. 2020Towards Resolving the Implicit Bias of Gradient Descent for Matrix Factorization: Greedy Low-Rank Learningcited by 150
  6. 2019Implicit Regularization of Discrete Gradient Dynamics in Linear Neural Networkscited by 124
  7. 2019Implicit regularization for deep neural networks driven by an Ornstein-Uhlenbeck like processcited by 123
  8. 2020Implicit Regularization in Deep Learning May Not Be Explainable by Normscited by 119
  9. 2020Graph Convolutional Networks for Hyperspectral Image Classificationcited by 113
  10. 2018Convergence of Gradient Descent on Separable Datacited by 110
  11. 2021Implicit Bias of SGD for Diagonal Linear Networks: a Provable Benefit of Stochasticitycited by 110
  12. 2021What Happens after SGD Reaches Zero Loss? --A Mathematical Frameworkcited by 107
  13. 2018Learning long-range spatial dependencies with horizontal gated-recurrent unitscited by 98
  14. 2015On the Expressive Power of Deep Learning: A Tensor Analysiscited by 97
  15. 2017Implicit Regularization in Deep Learningcited by 90
  16. 2018Overparameterized Nonlinear Learning: Gradient Descent Takes the Shortest Path?cited by 88
  17. 2020Shape Matters: Understanding the Implicit Bias of the Noise Covariancecited by 88
  18. 2018An analytic theory of generalization dynamics and transfer learning in deep linear networkscited by 85
  19. 2021Gradient Descent on Two-layer Nets: Margin Maximization and Simplicity Biascited by 83
  20. 2020A Unifying View on Implicit Bias in Training Linear Neural Networkscited by 75
  21. 2020Implicit Bias in Deep Linear Classification: Initialization Scale vs Training Accuracycited by 71
  22. 2021On the Implicit Bias of Initialization Shape: Beyond Infinitesimal Mirror Descentcited by 70
  23. 2019The Implicit Bias of Depth: How Incremental Learning Drives Generalizationcited by 64
  24. 2019Learning deep linear neural networks: Riemannian gradient flows and convergence to global minimizerscited by 61
  25. 2020The Surprising Simplicity of the Early-Time Learning Dynamics of Neural Networkscited by 58
  26. 2016Inductive Bias of Deep Convolutional Networks through Pooling Geometrycited by 53
  27. 2019Lexicographic and Depth-Sensitive Margins in Homogeneous and Non-Homogeneous Deep Modelscited by 53
  28. 2017Learning Relevant Features of Data with Multi-scale Tensor Networkscited by 51
  29. 2018Quantum Entanglement in Deep Learning Architecturescited by 50
  30. 2020Implicit Regularization in ReLU Networks with the Square Losscited by 45
  31. 2018Gradient descent with identity initialization efficiently learns positive definite linear transformations by deep residual networkscited by 44
  32. 2021Implicit Regularization in Tensor Factorizationcited by 44
  33. 2020The Depth-to-Width Interplay in Self-Attentioncited by 42
  34. 2020Gradient Descent for Deep Matrix Factorization: Dynamics and Implicit Bias towards Low Rankcited by 42
  35. 2021Continuous vs. Discrete Optimization of Deep Neural Networkscited by 34
  36. 2021On Margin Maximization in Linear and ReLU Networkscited by 34
  37. 2021The Inductive Bias of In-Context Learning: Rethinking Pretraining Example Designcited by 33
  38. 2016Convolutional Rectifier Networks as Generalized Tensor Decompositionscited by 32

50 most cited of 72 All 72 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 9; 2023: 10; 2024: 2; 2025: 5; 2026: 1.

27 papers.

Sort:

By area:

  1. 2022Understanding the Generalization Benefit of Normalization Layers: Sharpness Reductioncited by 86
  2. 2022On the Implicit Bias in Deep-Learning Algorithmscited by 86
  3. 2023Understanding Incremental Learning of Gradient Descent: A Fine-grained Analysis of Matrix Sensingcited by 42
  4. 2022Implicit Bias of Gradient Descent on Reparametrized Models: On Equivalence to Mirror Descentcited by 39
  5. 2022Incremental Learning in Diagonal Linear Networkscited by 28
  6. 2023Early Neuron Alignment in Two-layer ReLU Networks with Small Initializationcited by 28
  7. 2023Theoretical Analysis of Inductive Biases in Deep Convolutional Networkscited by 21
  8. 2022Permutation Search of Tensor Network Structures via Local Samplingcited by 13
  9. 2022On the Ability of Graph Neural Networks to Model Interactions Between Verticescited by 12
  10. 2023Robust Implicit Regularization via Weight Normalizationcited by 12
  11. 2022Learning Low Dimensional State Spaces with Overparameterized Recurrent Neural Netscited by 7
  12. 2025Why is Your Language Model a Poor Implicit Reward Model?cited by 6
15 more
  1. 2023A Quadratic Synchronization Rule for Distributed Deep Learningcited by 5
  2. 2022Behind the Scenes of Gradient Descent: A Trajectory Analysis via Basis Function Decompositioncited by 4
  3. 2023Implicit Regularization Leads to Benign Overfitting for Sparse Linear Regressioncited by 4
  4. 2023What Makes Data Suitable for a Locally Connected Neural Network? A Necessary and Sufficient Condition Based on Quantum Entanglementcited by 4
  5. 2023Implicit regularization in AI meets generalized hardness of approximation in optimization -- Sharp results for diagonal linear networkscited by 4
  6. 2024Implicit Bias of Policy Gradient in Linear Quadratic Control: Extrapolation to Unseen Initial Statescited by 4
  7. 2025Continuous Representation Methods, Theories, and Applications: An Overview and Perspectivescited by 4
  8. 2022Implicit Regularization with Polynomial Growth in Deep Tensor Factorizationcited by 3
  9. 2025Structured Transformations for Stable and Interpretable Neural Computationcited by 2
  10. 2025Neural Collapse under Gradient Flow on Shallow ReLU Networks for Orthogonally Separable Datacited by 2
  11. 2023Algorithmic Regularization in Tensor Optimization: Towards a Lifted Approach in Matrix Sensingcited by 1
  12. 2024Lecture Notes on Linear Neural Networks: A Tale of Optimization and Generalization in Deep Learningcited by 1
  13. 2025Understanding Learning Dynamics Through Structured Representationscited by 1
  14. 2023Sparse Mean Estimation in Adversarial Settings via Incremental Learningcited by 0
  15. 2026Stabilizing the Dynamic Low-Rank Trainingcited by 0