mlsanityData to 7 Oct 2026

Fantastic Generalization Measures and Where to Find Them

arXiv 1912.02178 ↗First posted 4 Dec 2019cs.LG stat.MLcited by 507

Not on the map. Why?

Abstract (arXiv)

Generalization of deep networks has been of great interest in recent years, resulting in a number of theoretically and empirically motivated complexity measures. However, most papers proposing such measures study only a small set of models, leaving open the question of whether the conclusion drawn from those experiments would remain valid in other settings. We present the first large scale study of generalization in deep networks. We investigate more then 40 complexity measures taken from both theoretical bounds and empirical studies. We train over 10,000 convolutional networks by systematically varying commonly used hyperparameters. Hoping to uncover potentially causal relationships between each measure and generalization, we analyze carefully controlled experiments and show surprising failures of some measures as well as promising measures for further research.

Key facts. Cited by 507 papers in the map (data through 7 Oct 2026). First posted on arXiv 4 Dec 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 88; 2022: 104; 2023: 86; 2024: 100; 2025: 67; 2026: 62.

507 papers.

Sort:

By area:

  1. 2022Grokking: Generalization Beyond Overfitting on Small Algorithmic Datasetscited by 599
  2. 2021SWAD: Domain Generalization by Seeking Flat Minimacited by 431
  3. 2021ASAM: Adaptive Sharpness-Aware Minimization for Scale-Invariant Learning of Deep Neural Networkscited by 293
  4. 2022QDrop: Randomly Dropping Quantization for Extremely Low-bit Post-Training Quantizationcited by 175
  5. 2022Surrogate Gap Minimization Improves Sharpness-Aware Trainingcited by 171
  6. 2021Neural Collapse Under MSE Loss: Proximity to and Dynamics on the Central Pathcited by 156
  7. 2022Self-Stabilization: The Implicit Bias of Gradient Descent at the Edge of Stabilitycited by 151
  8. 2022Towards Understanding Sharpness-Aware Minimizationcited by 145
  9. 2021Inductive Biases and Variable Creation in Self-Attention Mechanismscited by 144
  10. 2022Minimizing the Accumulated Trajectory Error to Improve Dataset Distillationcited by 142
  11. 2021Deep Learning Through the Lens of Example Difficultycited by 139
  12. 2021Efficient Sharpness-aware Minimization for Improved Training of Neural Networkscited by 137
38 more
  1. 2022Understanding Gradient Descent on Edge of Stability in Deep Learningcited by 126
  2. 2021Overcoming Catastrophic Forgetting in Incremental Few-Shot Learning by Finding Flat Minimacited by 118
  3. 2021Assessing Generalization of SGD via Disagreementcited by 116
  4. 2021Predicting with Confidence on Unseen Distributionscited by 116
  5. 2021Local Learning Matters: Rethinking Data Heterogeneity in Federated Learningcited by 114
  6. 2021Scalable Marginal Likelihood Estimation for Model Selection in Deep Learningcited by 105
  7. 2021Reasonable Effectiveness of Random Weighting: A Litmus Test for Multi-Task Learningcited by 100
  8. 2022Improving Generalization in Federated Learning by Seeking Flat Minimacited by 98
  9. 2022Sharpness-Aware Training for Freecited by 95
  10. 2023A Modern Look at the Relationship between Sharpness and Generalizationcited by 91
  11. 2022More Than a Toy: Random Matrix Models Predict How Real-World Neural Representations Generalizecited by 87
  12. 2022Fisher SAM: Information Geometry and Sharpness Aware Minimisationcited by 84
  13. 2022Same Pre-training Loss, Better Downstream: Implicit Bias Matters for Language Modelscited by 84
  14. 2021Gradient Descent on Two-layer Nets: Margin Maximization and Simplicity Biascited by 82
  15. 2021Encoding-dependent generalization bounds for parametrized quantum circuitscited by 81
  16. 2022Cramming: Training a Language Model on a Single GPU in One Daycited by 81
  17. 2022When Do Flat Minima Optimizers Work?cited by 78
  18. 2024Why are Sensitive Functions Hard for Transformers?cited by 74
  19. 2023Why Do We Need Weight Decay in Modern Deep Learning?cited by 70
  20. 2022Overview frequency principle/spectral bias in deep learningcited by 68
  21. 2022Boosting the Transferability of Adversarial Attacks with Reverse Adversarial Perturbationcited by 67
  22. 2022How Does Sharpness-Aware Minimization Minimize Sharpness?cited by 64
  23. 2021What Does Rotation Prediction Tell Us about Classifier Accuracy under Varying Testing Environments?cited by 63
  24. 2021Stochastic Training is Not Necessary for Generalizationcited by 63
  25. 2021Relating Adversarially Robust Generalization to Flat Minimacited by 57
  26. 2022Sharp-MAML: Sharpness-Aware Model-Agnostic Meta Learningcited by 57
  27. 2022On the Maximum Hessian Eigenvalue and Generalizationcited by 57
  28. 2023Gradient Norm Aware Minimization Seeks First-Order Flatness and Improves Generalizationcited by 57
  29. 2022Bayesian Model Selection, the Marginal Likelihood, and Generalizationcited by 56
  30. 2021Exploring Memorization in Adversarial Trainingcited by 55
  31. 2025Deep Learning is Not So Mysterious or Differentcited by 55
  32. 2023The Local Learning Coefficient: A Singularity-Aware Complexity Measurecited by 50
  33. 2021Information-theoretic generalization bounds for black-box learning algorithmscited by 49
  34. 2023Boosting Adversarial Transferability by Achieving Flat Local Maximacited by 48
  35. 2023Understanding quantum machine learning also requires rethinking generalizationcited by 48
  36. 2022Aug-NeRF: Training Stronger Neural Radiance Fields with Triple-Level Physically-Grounded Augmentationscited by 46
  37. 2021Improved Regularization and Robustness for Fine-tuning in Neural Networkscited by 45
  38. 2023An Adaptive Policy to Employ Sharpness-Aware Minimizationcited by 43

50 most cited of 507 All 507 (JSON)