mlsanityData to 7 Oct 2026

The Low-Rank Simplicity Bias in Deep Networks

arXiv 2103.10427 ↗First posted 18 Mar 2021cs.LG cs.CVcited by 150

Not on the map. Why?

Abstract (arXiv)

Modern deep neural networks are highly over-parameterized compared to the data on which they are trained, yet they often generalize remarkably well. A flurry of recent work has asked: why do deep networks not overfit to their training data? In this work, we make a series of empirical observations that investigate and extend the hypothesis that deeper networks are inductively biased to find solutions with lower effective rank embeddings. We conjecture that this bias exists because the volume of functions that maps to low effective rank embedding increases with depth. We show empirically that our claim holds true on finite width linear and non-linear models on practical learning paradigms and show that on natural data, these are often the solutions that generalize well. We then show that the simplicity bias exists at both initialization and after training and is resilient to hyper-parameters and learning methods. We further demonstrate how linear over-parameterization of deep non-linear models can be used to induce low-rank bias, improving generalization performance on CIFAR and ImageNet without changing the modeling capacity.

Key facts. Cited by 150 papers in the map (data through 7 Oct 2026). Builds on 61 of its 95 references that are in the map. First posted on arXiv 18 Mar 2021.

Builds on

61 of 95 references.

References · 61

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  4. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  5. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,029
  6. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  7. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
  8. 2020Scaling Laws for Neural Language Modelscited by 6,411
  9. 2014Going Deeper with Convolutionscited by 4,231
  10. 2016Gaussian Error Linear Units (GELUs)cited by 3,431
  11. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  12. 2016Aggregated Residual Transformations for Deep Neural Networkscited by 2,540
38 more
  1. 2020Implicit Neural Representations with Periodic Activation Functionscited by 2,214
  2. 2016Understanding deep learning requires rethinking generalizationcited by 1,657
  3. 2016Neural Architecture Search with Reinforcement Learningcited by 1,489
  4. 2017A Closer Look at Memorization in Deep Networkscited by 936
  5. 2018Reconciling modern machine learning practice and the bias-variance trade-offcited by 919
  6. 2019Deep Double Descent: Where Bigger Models and More Data Hurtcited by 667
  7. 2018Gradient Descent Provably Optimizes Over-parameterized Neural Networkscited by 655
  8. 2017The Implicit Bias of Gradient Descent on Separable Datacited by 649
  9. 2018Gradient Descent Finds Global Minima of Deep Neural Networkscited by 572
  10. 2019Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networkscited by 562
  11. 2017Deep Learning Scaling is Predictable, Empiricallycited by 556
  12. 2019Benign Overfitting in Linear Regressioncited by 544
  13. 2019Implicit Regularization in Deep Matrix Factorizationcited by 402
  14. 2020The Pitfalls of Simplicity Bias in Neural Networkscited by 349
  15. 2014In Search of the Real Inductive Bias: On the Role of Implicit Regularization in Deep Learningcited by 339
  16. 2015Skip-Thought Vectorscited by 321
  17. 2017Implicit Regularization in Matrix Factorizationcited by 318
  18. 2018On the Optimization of Deep Networks: Implicit Acceleration by Overparameterizationcited by 280
  19. 2020Gradient Starvation: A Learning Proclivity in Neural Networkscited by 262
  20. 2018Implicit Bias of Gradient Descent on Linear Convolutional Networkscited by 251
  21. 2021Deep learning: a statistical viewpointcited by 239
  22. 2017High-dimensional dynamics of generalization error in neural networkscited by 238
  23. 2019SGD on Neural Networks Learns Functions of Increasing Complexitycited by 203
  24. 2014Qualitatively characterizing neural network optimization problemscited by 186
  25. 2018Gradient descent aligns the layers of deep linear networkscited by 182
  26. 2015The Power of Depth for Feedforward Neural Networkscited by 181
  27. 2018Deep learning generalizes because the parameter-function map is biased towards simple functionscited by 180
  28. 2017Spectral Norm Regularization for Improving the Generalizability of Deep Learningcited by 178
  29. 2019Blind Super-Resolution Kernel Estimation using an Internal-GANcited by 175
  30. 2020Directional convergence and alignment in deep learningcited by 172
  31. 2020Towards Resolving the Implicit Bias of Gradient Descent for Matrix Factorization: Greedy Low-Rank Learningcited by 148
  32. 2017Resurrecting the sigmoid in deep learning through dynamical isometry: theory and practicecited by 132
  33. 2019Implicit Regularization of Discrete Gradient Dynamics in Linear Neural Networkscited by 122
  34. 2020Implicit Regularization in Deep Learning May Not Be Explainable by Normscited by 118
  35. 2018Convergence of Gradient Descent on Separable Datacited by 108
  36. 2015Low-rank Solutions of Linear Matrix Equations via Procrustes Flowcited by 97
  37. 2019Fast Convergence of Natural Gradient Descent for Overparameterized Neural Networkscited by 76
  38. 2018The Emergence of Spectral Universality in Deep Networkscited by 74

50 most cited of 61 All 61 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 9; 2022: 12; 2023: 33; 2024: 32; 2025: 38; 2026: 26.

150 papers.

Sort:

By area:

  1. 2021Neural Fields in Visual Computing and Beyondcited by 539
  2. 2024The Platonic Representation Hypothesiscited by 362
  3. 2025When More is Less: Understanding Chain-of-Thought Length in LLMscited by 182
  4. 2021Can contrastive learning avoid shortcut solutions?cited by 114
  5. 2024KV Cache is 1 Bit Per Channel: Efficient Large Language Model Inference with Coupled Quantizationcited by 74
  6. 2023The No Free Lunch Theorem, Kolmogorov Complexity, and the Role of Inductive Biases in Machine Learningcited by 63
  7. 2021The Difficulty of Passive Learning in Deep Reinforcement Learningcited by 50
  8. 2024Neural Redshift: Random Networks are not Random Functionscited by 43
  9. 2023PLASTIC: Improving Input and Label Plasticity for Sample Efficient Reinforcement Learningcited by 42
  10. 2023Hypernetwork-based Meta-Learning for Low-Rank Physics-Informed Neural Networkscited by 42
  11. 2023Sharpness-Aware Minimization Leads to Low-Rank Featurescited by 38
  12. 2023Stochastic Collapse: How Gradient Noise Attracts SGD Dynamics Towards Simpler Subnetworkscited by 38
38 more
  1. 2023Unsupervised 3D Perception with 2D Vision-Language Distillation for Autonomous Drivingcited by 37
  2. 2023The Tunnel Effect: Building Data Representations in Deep Neural Networkscited by 34
  3. 2022Training invariances and the low-rank phenomenon: beyond linear networkscited by 33
  4. 2024Approaching Deep Learning through the Spectral Dynamics of Weightscited by 29
  5. 2023Deep neural networks have an inbuilt Occam's razorcited by 28
  6. 2025CoLA: Compute-Efficient Pre-Training of LLMs via Low-Rank Activationcited by 28
  7. 2024No Representation, No Trust: Connecting Representation, Collapse, and Trust Issues in PPOcited by 27
  8. 2023Simplicity Bias in 1-Hidden Layer Neural Networkscited by 26
  9. 2024Compressible Dynamics in Deep Overparameterized Low-Rank Learning & Adaptationcited by 26
  10. 2025An Analytical Theory of Spectral Bias in the Learning Dynamics of Diffusion Modelscited by 26
  11. 2025A Closer Look at Multimodal Representation Collapsecited by 25
  12. 2022An Empirical Study of Implicit Regularization in Deep Offline RLcited by 24
  13. 2022Mechanism of feature learning in deep fully connected networks and kernel machines that recursively learn featurescited by 24
  14. 2024Plasticity Loss in Deep Reinforcement Learning: A Surveycited by 24
  15. 2024Learned feature representations are biased by complexity, learning order, position, and morecited by 22
  16. 2023The Law of Parsimony in Gradient Descent for Learning Deep Linear Networkscited by 19
  17. 2022Overcoming the Spectral Bias of Neural Value Approximationcited by 18
  18. 2023Dynamics Generalisation in Reinforcement Learning via Adaptive Context-Aware Policiescited by 18
  19. 2023Variation Spaces for Multi-Output Neural Networks: Insights on Multi-Task Learning and Network Compressioncited by 17
  20. 2024Training Neural Networks from Scratch with Parallel Low-Rank Adapterscited by 17
  21. 2023(Almost) Provable Error Bounds Under Distribution Shift via Disagreement Discrepancycited by 16
  22. 2024Neural Collapse versus Low-rank Bias: Is Deep Neural Collapse Really Optimal?cited by 16
  23. 2025Block-Recurrent Dynamics in Vision Transformerscited by 16
  24. 2022On the Learning and Learnability of Quasimetricscited by 15
  25. 2024Understanding Visual Feature Reliance through the Lens of Complexitycited by 15
  26. 2021Layer Folding: Neural Network Depth Reduction using Activation Linearizationcited by 14
  27. 2022Evolution of Neural Tangent Kernels under Benign and Adversarial Trainingcited by 14
  28. 2024On The Specialization of Neural Modulescited by 14
  29. 2025Saddle-to-Saddle Dynamics Explains A Simplicity Bias Across Neural Network Architecturescited by 14
  30. 2023Outliers with Opposing Signals Have an Outsized Effect on Neural Network Optimizationcited by 13
  31. 2024Transformers Learn Low Sensitivity Functions: Investigations and Implicationscited by 13
  32. 2025Provable Post-Training Quantization: Theoretical Analysis of OPTQ and Qronoscited by 13
  33. 2023Efficient Storage of Fine-Tuned Models via Low-Rank Approximation of Weight Residualscited by 12
  34. 2023Robust low-rank training via approximate orthonormal constraintscited by 12
  35. 2024The Persistence of Neural Collapse Despite Low-Rank Biascited by 12
  36. 2022Machine Learning and Deep Learning -- A review for Ecologistscited by 11
  37. 2024Simplicity bias and optimization threshold in two-layer ReLU networkscited by 11
  38. 2022On the Origins of the Block Structure Phenomenon in Neural Network Representationscited by 10

50 most cited of 150 All 150 (JSON)