mlsanityData to 7 Oct 2026

Gradient Starvation: A Learning Proclivity in Neural Networks

arXiv 2011.09468 ↗First posted 18 Nov 2020cs.LG math.DS stat.MLcited by 262

Not on the map. Why?

Abstract (arXiv)

We identify and formalize a fundamental gradient descent phenomenon resulting in a learning proclivity in over-parameterized neural networks. Gradient Starvation arises when cross-entropy loss is minimized by capturing only a subset of features relevant for the task, despite the presence of other predictive features that fail to be discovered. This work provides a theoretical explanation for the emergence of such feature imbalance in neural networks. Using tools from Dynamical Systems theory, we identify simple properties of learning dynamics during gradient descent that lead to this imbalance, and prove that such a situation can be expected given certain statistical structure in training data. Based on our proposed formalism, we develop guarantees for a novel regularization method aimed at decoupling feature learning dynamics, improving accuracy and robustness in cases hindered by gradient starvation. We illustrate our findings with simple and real-world out-of-distribution (OOD) generalization experiments.

Key facts. Cited by 262 papers in the map (data through 7 Oct 2026). First posted on arXiv 18 Nov 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 28; 2022: 42; 2023: 69; 2024: 41; 2025: 35; 2026: 47.

262 papers.

Sort:

By area:

  1. 2021Just Train Twice: Improving Group Robustness without Training Group Informationcited by 522
  2. 2022Last Layer Re-Training is Sufficient for Robustness to Spurious Correlationscited by 379
  3. 2021Self-Supervised Learning with Data Augmentations Provably Isolates Content from Stylecited by 308
  4. 2021Invariance Principle Meets Information Bottleneck for Out-of-Distribution Generalizationcited by 220
  5. 2021Fishr: Invariant Gradient Variances for Out-of-Distribution Generalizationcited by 186
  6. 2021Simple data balancing achieves competitive worst-group-accuracycited by 177
  7. 2022Correct-N-Contrast: A Contrastive Approach for Improving Robustness to Spurious Correlationscited by 171
  8. 2024The pitfalls of next-token predictioncited by 156
  9. 2021The Low-Rank Simplicity Bias in Deep Networkscited by 150
  10. 2022On Feature Learning in the Presence of Spurious Correlationscited by 145
  11. 2024WARM: On the Benefits of Weight Averaged Reward Modelscited by 145
  12. 2023Sudden Drops in the Loss: Syntax Acquisition, Phase Transitions, and Simplicity Bias in MLMscited by 121
38 more
  1. 2021OoD-Bench: Quantifying and Understanding Two Dimensions of Out-of-Distribution Generalizationcited by 86
  2. 2022A Whac-A-Mole Dilemma: Shortcuts Come in Multiples Where Mitigating One Amplifies Otherscited by 85
  3. 2021Evading the Simplicity Bias: Training a Diverse Set of Models Discovers Solutions with Superior OOD Generalizationcited by 83
  4. 2024Disentangling the Causes of Plasticity Loss in Neural Networkscited by 82
  5. 2021Can Subnetwork Structure be the Key to Out-of-Distribution Generalization?cited by 75
  6. 2022Agree to Disagree: Diversity through Disagreement for Better Transferabilitycited by 69
  7. 2022Probable Domain Generalization via Quantile Risk Minimizationcited by 66
  8. 2022MaskTune: Mitigating Spurious Correlations by Forcing to Explorecited by 66
  9. 2023On the Foundations of Shortcut Learningcited by 64
  10. 2021Fairness via Representation Neutralizationcited by 63
  11. 2021Continual Learning via Local Module Compositioncited by 61
  12. 2021Better Safe Than Sorry: Preventing Delusive Adversaries with Adversarial Trainingcited by 58
  13. 2022Diversify and Disambiguate: Learning From Underspecified Datacited by 55
  14. 2023Towards Last-layer Retraining for Group Robustness with Fewer Annotationscited by 54
  15. 2023On Uni-Modal Feature Learning in Supervised Multi-Modal Learningcited by 53
  16. 2022Learning Debiased Classifier with Biased Committeecited by 51
  17. 2021SAND-mask: An Enhanced Gradient Masking Strategy for the Discovery of Invariances in Domain Generalizationcited by 50
  18. 2022Mechanistic Mode Connectivitycited by 50
  19. 2023Improved Test-Time Adaptation for Domain Generalizationcited by 45
  20. 2023Identifying Spurious Biases Early in Training through the Lens of Simplicity Biascited by 44
  21. 2024Neural Redshift: Random Networks are not Random Functionscited by 43
  22. 2022Understanding and Mitigating Overfitting in Prompt Tuning for Vision-Language Modelscited by 40
  23. 2022Pareto Invariant Risk Minimization: Towards Mitigating the Optimization Dilemma in Out-of-Distribution Generalizationcited by 38
  24. 2021Quantifying and Improving Transferability in Domain Generalizationcited by 35
  25. 2023Understanding and Improving Feature Learning for Out-of-Distribution Generalizationcited by 35
  26. 2022Rich Feature Construction for the Optimization-Generalization Dilemmacited by 34
  27. 2023Free Lunch for Domain Adversarial Training: Environment Label Smoothingcited by 33
  28. 2023Feedback-guided Data Synthesis for Imbalanced Classificationcited by 33
  29. 2025Tracing the Representation Geometry of Language Models from Pretraining to Post-trainingcited by 33
  30. 2023Predicting Grokking Long Before it Happens: A look into the loss landscape of models which grokcited by 31
  31. 2022Grounding Visual Representations with Texts for Domain Generalizationcited by 30
  32. 2023Don't blame Dataset Shift! Shortcut Learning due to Gradients and Cross Entropycited by 30
  33. 2024Calibrating Multi-modal Representations: A Pursuit of Group Robustness without Annotationscited by 30
  34. 2022WOODS: Benchmarks for Out-of-Distribution Generalization in Time Seriescited by 29
  35. 2023Overcoming Simplicity Bias in Deep Networks using a Feature Sievecited by 29
  36. 2023Domain Generalization in Computational Pathology: Survey and Guidelinescited by 29
  37. 2023What do neural networks learn in image classification? A frequency shortcut perspectivecited by 28
  38. 2025Roll the dice & look before you leap: Going beyond the creative limits of next-token predictioncited by 28

50 most cited of 262 All 262 (JSON)