mlsanityData to 14 Oct 2026

Exploring Generalization in Deep Learning

arXiv 1706.08947 ↗First posted 27 Jun 2017cs.LGcited by 600

Not on the map. Why?

Abstract (arXiv)

With a goal of understanding what drives generalization in deep networks, we consider several recently suggested explanations, including norm-based control, sharpness and robustness. We study how these measures can ensure generalization, highlighting the importance of scale normalization, and making a connection between sharpness and PAC-Bayes theory. We then investigate how well the measures explain different observed phenomena.

Key facts. Cited by 600 papers in the map (data through 14 Oct 2026). First posted on arXiv 27 Jun 2017.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2020: 1; 2021: 126; 2022: 122; 2023: 119; 2024: 97; 2025: 78; 2026: 57.

600 papers.

Sort:

By area:

  1. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,631
  2. 2021ASAM: Adaptive Sharpness-Aware Minimization for Scale-Invariant Learning of Deep Neural Networkscited by 293
  3. 2021The Role of Permutation Invariance in Linear Mode Connectivity of Neural Networkscited by 291
  4. 2022U-NO: U-shaped Neural Operatorscited by 230
  5. 2023Language Modeling Is Compressioncited by 225
  6. 2022QDrop: Randomly Dropping Quantization for Extremely Low-bit Post-Training Quantizationcited by 180
  7. 2022SimGRACE: A Simple Framework for Graph Contrastive Learning without Data Augmentationcited by 171
  8. 2021Neural Collapse Under MSE Loss: Proximity to and Dynamics on the Central Pathcited by 158
  9. 2022Self-Stabilization: The Implicit Bias of Gradient Descent at the Edge of Stabilitycited by 155
  10. 2022Towards Understanding Sharpness-Aware Minimizationcited by 146
  11. 2022Leveraging Unlabeled Data to Predict Out-of-Distribution Performancecited by 145
  12. 2021Deep Learning Through the Lens of Example Difficultycited by 141
38 more
  1. 2021Efficient Sharpness-aware Minimization for Improved Training of Neural Networkscited by 137
  2. 2022Reconstructing Training Data from Trained Neural Networkscited by 133
  3. 2022Class-Incremental Continual Learning into the eXtended DER-versecited by 124
  4. 2022Penalizing Gradient Norm for Efficiently Improving Generalization in Deep Learningcited by 122
  5. 2022REPAIR: REnormalizing Permuted Activations for Interpolation Repaircited by 120
  6. 2021Assessing Generalization of SGD via Disagreementcited by 117
  7. 2021Exploring the Limits of Large Scale Pre-trainingcited by 104
  8. 2022Agreement-on-the-Line: Predicting the Performance of Neural Networks under Distribution Shiftcited by 94
  9. 2023A Modern Look at the Relationship between Sharpness and Generalizationcited by 92
  10. 2022Understanding the Generalization Benefit of Normalization Layers: Sharpness Reductioncited by 86
  11. 2022On the Implicit Bias in Deep-Learning Algorithmscited by 86
  12. 2021GBK-GNN: Gated Bi-Kernel Graph Neural Networks for Modeling Both Homophily and Heterophilycited by 84
  13. 2022Fisher SAM: Information Geometry and Sharpness Aware Minimisationcited by 84
  14. 2022Same Pre-training Loss, Better Downstream: Implicit Bias Matters for Language Modelscited by 84
  15. 2024MMPareto: Boosting Multimodal Learning with Innocent Unimodal Assistancecited by 73
  16. 2021Flattening Sharpness for Dynamic Gradient Projection Memory Benefits Continual Learningcited by 72
  17. 2021Intrinsic Dimension, Persistent Homology and Generalization in Neural Networkscited by 70
  18. 2022Make Sharpness-Aware Minimization Stronger: A Sparsified Perturbation Approachcited by 70
  19. 2022Synthetic Data -- what, why and how?cited by 67
  20. 2021Model Complexity of Deep Learning: A Surveycited by 65
  21. 2022How Does Sharpness-Aware Minimization Minimize Sharpness?cited by 65
  22. 2022β-DARTS: Beta-Decay Regularization for Differentiable Architecture Searchcited by 64
  23. 2021On Linear Stability of SGD and Input-Smoothness of Neural Networkscited by 63
  24. 2021What Does Rotation Prediction Tell Us about Classifier Accuracy under Varying Testing Environments?cited by 63
  25. 2021What Kinds of Functions do Deep Neural Networks Learn? Insights from Variational Spline Theorycited by 60
  26. 2021A Theory of PAC Learnability of Partial Concept Classescited by 60
  27. 2021Open-set Label Noise Can Improve Robustness Against Inherent Label Noisecited by 59
  28. 2021When Do Extended Physics-Informed Neural Networks (XPINNs) Improve Generalization?cited by 58
  29. 2021Relating Adversarially Robust Generalization to Flat Minimacited by 57
  30. 2021Multi-Prize Lottery Ticket Hypothesis: Finding Accurate Binary Neural Networks by Pruning A Randomly Weighted Networkcited by 55
  31. 2021Exploring Memorization in Adversarial Trainingcited by 55
  32. 2022Implicit Regularization Towards Rank Minimization in ReLU Networkscited by 53
  33. 2021Learning Theory Can (Sometimes) Explain Generalisation in Graph Neural Networkscited by 51
  34. 2023The Local Learning Coefficient: A Singularity-Aware Complexity Measurecited by 51
  35. 2022Towards Robust Referring Image Segmentationcited by 49
  36. 2022Accountability in an Algorithmic Society: Relationality, Responsibility, and Robustness in Machine Learningcited by 48
  37. 2023Boosting Adversarial Transferability by Achieving Flat Local Maximacited by 48
  38. 2024Learning from Teaching Regularization: Generalizable Correlations Should be Easy to Imitatecited by 47

50 most cited of 600 All 600 (JSON)