# Exploring Generalization in Deep Learning

arXiv: https://arxiv.org/abs/1706.08947 · HTML: https://mlsanity.com/paper/1706.08947/

## Key facts

- Cited by 600 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 27 Jun 2017.

## Built on by (most cited first, 30 of 600)

- [GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding](https://mlsanity.com/paper/2006.16668/) (2020)
- [ASAM: Adaptive Sharpness-Aware Minimization for Scale-Invariant Learning of Deep Neural Networks](https://mlsanity.com/paper/2102.11600/) (2021)
- [The Role of Permutation Invariance in Linear Mode Connectivity of Neural Networks](https://mlsanity.com/paper/2110.06296/) (2021)
- [U-NO: U-shaped Neural Operators](https://mlsanity.com/paper/2204.11127/) (2022)
- [Language Modeling Is Compression](https://mlsanity.com/paper/2309.10668/) (2023)
- [QDrop: Randomly Dropping Quantization for Extremely Low-bit Post-Training Quantization](https://mlsanity.com/paper/2203.05740/) (2022)
- [SimGRACE: A Simple Framework for Graph Contrastive Learning without Data Augmentation](https://mlsanity.com/paper/2202.03104/) (2022)
- [Neural Collapse Under MSE Loss: Proximity to and Dynamics on the Central Path](https://mlsanity.com/paper/2106.02073/) (2021)
- [Self-Stabilization: The Implicit Bias of Gradient Descent at the Edge of Stability](https://mlsanity.com/paper/2209.15594/) (2022)
- [Towards Understanding Sharpness-Aware Minimization](https://mlsanity.com/paper/2206.06232/) (2022)
- [Leveraging Unlabeled Data to Predict Out-of-Distribution Performance](https://mlsanity.com/paper/2201.04234/) (2022)
- [Deep Learning Through the Lens of Example Difficulty](https://mlsanity.com/paper/2106.09647/) (2021)
- [Efficient Sharpness-aware Minimization for Improved Training of Neural Networks](https://mlsanity.com/paper/2110.03141/) (2021)
- [Reconstructing Training Data from Trained Neural Networks](https://mlsanity.com/paper/2206.07758/) (2022)
- [Class-Incremental Continual Learning into the eXtended DER-verse](https://mlsanity.com/paper/2201.00766/) (2022)
- [Penalizing Gradient Norm for Efficiently Improving Generalization in Deep Learning](https://mlsanity.com/paper/2202.03599/) (2022)
- [REPAIR: REnormalizing Permuted Activations for Interpolation Repair](https://mlsanity.com/paper/2211.08403/) (2022)
- [Assessing Generalization of SGD via Disagreement](https://mlsanity.com/paper/2106.13799/) (2021)
- [Exploring the Limits of Large Scale Pre-training](https://mlsanity.com/paper/2110.02095/) (2021)
- [Agreement-on-the-Line: Predicting the Performance of Neural Networks under Distribution Shift](https://mlsanity.com/paper/2206.13089/) (2022)
- [A Modern Look at the Relationship between Sharpness and Generalization](https://mlsanity.com/paper/2302.07011/) (2023)
- [Understanding the Generalization Benefit of Normalization Layers: Sharpness Reduction](https://mlsanity.com/paper/2206.07085/) (2022)
- [On the Implicit Bias in Deep-Learning Algorithms](https://mlsanity.com/paper/2208.12591/) (2022)
- [GBK-GNN: Gated Bi-Kernel Graph Neural Networks for Modeling Both Homophily and Heterophily](https://mlsanity.com/paper/2110.15777/) (2021)
- [Fisher SAM: Information Geometry and Sharpness Aware Minimisation](https://mlsanity.com/paper/2206.04920/) (2022)
- [Same Pre-training Loss, Better Downstream: Implicit Bias Matters for Language Models](https://mlsanity.com/paper/2210.14199/) (2022)
- [MMPareto: Boosting Multimodal Learning with Innocent Unimodal Assistance](https://mlsanity.com/paper/2405.17730/) (2024)
- [Flattening Sharpness for Dynamic Gradient Projection Memory Benefits Continual Learning](https://mlsanity.com/paper/2110.04593/) (2021)
- [Intrinsic Dimension, Persistent Homology and Generalization in Neural Networks](https://mlsanity.com/paper/2111.13171/) (2021)
- [Make Sharpness-Aware Minimization Stronger: A Sparsified Perturbation Approach](https://mlsanity.com/paper/2210.05177/) (2022)

## Abstract (arXiv)

With a goal of understanding what drives generalization in deep networks, we consider several recently suggested explanations, including norm-based control, sharpness and robustness. We study how these measures can ensure generalization, highlighting the importance of scale normalization, and making a connection between sharpness and PAC-Bayes theory. We then investigate how well the measures explain different observed phenomena.