mlsanityData to 13 Oct 2026

Coherent Gradients: An Approach to Understanding Generalization in Gradient Descent-based Optimization

arXiv 2002.10657 ↗First posted 25 Feb 2020cs.LG cs.CV stat.MLcited by 41

Not on the map. Why?

Abstract (arXiv)

An open question in the Deep Learning community is why neural networks trained with Gradient Descent generalize well on real datasets even though they are capable of fitting random data. We propose an approach to answering this question based on a hypothesis about the dynamics of gradient descent that we call Coherent Gradients: Gradients from similar examples are similar and so the overall gradient is stronger in certain directions where these reinforce each other. Thus changes to the network parameters during training are biased towards those that (locally) simultaneously benefit many examples when such similarity exists. We support this hypothesis with heuristic arguments and perturbative experiments and outline how this can explain several common empirical observations about Deep Learning. Furthermore, our analysis is not just descriptive, but prescriptive. It suggests a natural modification to gradient descent that can greatly reduce overfitting.

Key facts. Cited by 41 papers in the map (data through 13 Oct 2026). First posted on arXiv 25 Feb 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2021: 8; 2022: 7; 2023: 7; 2024: 3; 2025: 9; 2026: 7.

41 papers.

Sort:

  1. 2021Deep Learning Through the Lens of Example Difficultycited by 141
  2. 2021On the geometry of generalization and memorization in deep neural networkscited by 72
  3. 2023Can Neural Network Memorization Be Localized?cited by 68
  4. 2021Stochastic Training is Not Necessary for Generalizationcited by 63
  5. 2023Revisiting Weighted Aggregation in Federated Learning with Neural Networkscited by 58
  6. 2023T-MARS: Improving Visual Representations by Circumventing Text Feature Learningcited by 43
  7. 2022Characterizing Datapoints via Second-Split Forgettingcited by 37
  8. 2022Stabilizing Off-Policy Deep Reinforcement Learning from Pixelscited by 32
  9. 2021Implicit Gradient Alignment in Distributed and Federated Learningcited by 29
  10. 2022On the Generalization Mystery in Deep Learningcited by 25
  11. 2023FAIR-Ensemble: When Fairness Naturally Emerges From Deep Ensemblingcited by 11
  12. 2021Disparity Between Batches as a Signal for Early Stoppingcited by 9
29 more
  1. 2022Scaling Multimodal Pre-Training via Cross-Modality Gradient Harmonizationcited by 9
  2. 2025Subspace-Boosted Model Mergingcited by 9
  3. 2021Cockpit: A Practical Debugging Tool for the Training of Deep Neural Networkscited by 7
  4. 2021Understanding Memorization from the Perspective of Optimization via Efficient Influence Estimationcited by 7
  5. 2025Memorization Sinks: Isolating Memorization during LLM Trainingcited by 7
  6. 2023Differences Between Hard and Noisy-labeled Samples: An Empirical Studycited by 4
  7. 2024DPDR: Gradient Decomposition and Reconstruction for Differentially Private Deep Learningcited by 4
  8. 2023Exploring Simple, High Quality Out-of-Distribution Detection with L2 Normalizationcited by 3
  9. 2024Leveraging Gradients for Unsupervised Accuracy Estimation under Distribution Shiftcited by 3
  10. 2025Enhanced Sample Selection with Confidence Tracking: Identifying Correctly Labeled yet Hard-to-Learn Samples in Noisy Datacited by 3
  11. 2022Split-PU: Hardness-aware Training Strategy for Positive-Unlabeled Learningcited by 2
  12. 2022Leveraging Unlabeled Data to Track Memorizationcited by 2
  13. 2025Just One Layer Norm Guarantees Stable Extrapolationcited by 2
  14. 2025A Unified Stability Analysis of SAM vs SGD: Role of Data Coherence and Emergence of Simplicity Biascited by 2
  15. 2025Convergence of Adam in Deep ReLU Networks via Directional Complexity and Kakeya Boundscited by 1
  16. 2025Laplace Sample Information: Data Informativeness Through a Bayesian Lenscited by 1
  17. 2025Generalization vs. Memorization in Autoregressive Deep Learning: Or, Examining Temporal Decay of Gradient Coherencecited by 1
  18. 2025Gradient-Weight Alignment as a Train-Time Proxy for Generalization in Classification Taskscited by 1
  19. 2026Mechanisms of Width Scaling in Normalized Residual Networks: The Effective Alignment Dimensioncited by 1
  20. 2021Dynamics of Local Elasticity During Training of Neural Netscited by 0
  21. 2022Towards understanding deep learning with the natural clustering priorcited by 0
  22. 2023Low-Entropy Latent Variables Hurt Out-of-Distribution Performancecited by 0
  23. 2024Beyond Single-Model Views for Deep Learning: Optimization versus Generalizability of Stochastic Optimization Algorithmscited by 0
  24. 2026Loss Landscape Geometry of Partial Differential Equation Emulators: Or, Symmetry Learning via Gradient Alignmentcited by 0
  25. 2026Certification of Machine Learning Models via Directional Sharpnesscited by 0
  26. 2026Dangerous Liaisons of Convex Learning and Non-Affine Aggregationcited by 0
  27. 2026Measuring Structured Predictability in Neural Training Dynamics: A Cross-Regime Studycited by 0
  28. 2026Every Batch Is Its Own Validation Set: Leave-One-Out Gradient Matching for Online Data Selection in LLM Fine-Tuningcited by 0
  29. 2026Why Does Train-Validation Separation Emerge? Update-Pressure Density Dynamics in Pretrained Backbonescited by 0