mlsanityData to 7 Oct 2026

Predicting the Generalization Gap in Deep Networks with Margin Distributions

arXiv 1810.00113 ↗First posted 28 Sep 2018stat.ML cs.LGcited by 111

Not on the map. Why?

Abstract (arXiv)

As shown in recent research, deep neural networks can perfectly fit randomly labeled data, but with very poor accuracy on held out data. This phenomenon indicates that loss functions such as cross-entropy are not a reliable indicator of generalization. This leads to the crucial question of how generalization gap should be predicted from the training data and network parameters. In this paper, we propose such a measure, and conduct extensive empirical studies on how well it can predict the generalization gap. Our measure is based on the concept of margin distribution, which are the distances of training points to the decision boundary. We find that it is necessary to use margin distributions at multiple layers of a deep network. On the CIFAR-10 and the CIFAR-100 datasets, our proposed measure correlates very strongly with the generalization gap. In addition, we find the following other factors to be of importance: normalizing margin values for scale independence, using characterizations of margin distribution rather than just the margin (closest distance to decision boundary), and working in log space instead of linear space (effectively using a product of margins rather than a sum). Our measure can be easily applied to feedforward deep networks with any architecture and may point towards new training loss functions that could enable better generalization.

Key facts. Cited by 111 papers in the map (data through 7 Oct 2026). First posted on arXiv 28 Sep 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 33; 2022: 24; 2023: 19; 2024: 14; 2025: 14; 2026: 7.

111 papers.

Sort:

By area:

  1. 2021Neural Collapse Under MSE Loss: Proximity to and Dynamics on the Central Pathcited by 156
  2. 2021Deep Learning Through the Lens of Example Difficultycited by 139
  3. 2021Assessing Generalization of SGD via Disagreementcited by 116
  4. 2021Predicting with Confidence on Unseen Distributionscited by 116
  5. 2021Exploring the Limits of Large Scale Pre-trainingcited by 104
  6. 2022Neural Collapse: A Review on Modelling Principles and Generalizationcited by 95
  7. 2023Permutation Equivariant Neural Functionalscited by 69
  8. 2021What Does Rotation Prediction Tell Us about Classifier Accuracy under Varying Testing Environments?cited by 63
  9. 2023Neural Functional Transformerscited by 50
  10. 2021Scalable Diverse Model Selection for Accessible Transfer Learningcited by 40
  11. 2024Towards Scalable and Versatile Weight Space Learningcited by 40
  12. 2022Why neural networks find simple solutions: the many regularizers of geometric complexitycited by 38
38 more
  1. 2022Model Zoos: A Dataset of Diverse Populations of Neural Network Modelscited by 38
  2. 2021Generalization bounds via distillationcited by 33
  3. 2021Pay attention to your loss: understanding misconceptions about 1-Lipschitz neural networkscited by 30
  4. 2021Noisy Feature Mixupcited by 27
  5. 2021Measuring Generalization with Optimal Transportcited by 24
  6. 2022On the Strong Correlation Between Model Invariance and Generalizationcited by 23
  7. 2023Confidence and Dispersity Speak: Characterising Prediction Matrix for Unsupervised Accuracy Estimationcited by 22
  8. 2022Multi-Object Navigation with dynamically learned neural implicit representationscited by 21
  9. 2025Early Stopping Against Label Noise Without Validation Datacited by 21
  10. 2022Evaluating natural language processing models with generalization metrics that do not need access to any training or testing datacited by 18
  11. 2023Neural (Tangent Kernel) Collapsecited by 18
  12. 2023Pseudo-Bag Mixup Augmentation for Multiple Instance Learning-Based Whole Slide Image Classificationcited by 18
  13. 2022Membership Inference Attacks and Generalization: A Causal Perspectivecited by 17
  14. 2024Energy-based Automated Model Evaluationcited by 16
  15. 2023On the Importance of Feature Separability in Predicting Out-Of-Distribution Errorcited by 15
  16. 2021Predicting Deep Neural Network Generalization with Perturbation Response Curvescited by 14
  17. 2022On the Relationship Between Explanation and Prediction: A Causal Viewcited by 14
  18. 2025Intermediate Layer Classifiers for OOD generalizationcited by 14
  19. 2021Robustness to Augmentations as a Generalization metriccited by 13
  20. 2021Why flatness does and does not correlate with generalization for deep neural networkscited by 13
  21. 2022Instance-Dependent Generalization Bounds via Optimal Transportcited by 13
  22. 2022Topology and geometry of data manifold in deep learningcited by 12
  23. 2024Dataset Difficulty and the Role of Inductive Biascited by 12
  24. 2021Deep Learning Generalization and the Convex Hull of Training Setscited by 10
  25. 2021Robustness to Pruning Predicts Generalization in Deep Neural Networkscited by 10
  26. 2021Label-Free Model Evaluation with Semi-Structured Dataset Representationscited by 10
  27. 2022Delving into the Openness of CLIPcited by 10
  28. 2023MarginMatch: Improving Semi-Supervised Learning with Pseudo-Marginscited by 10
  29. 2021Hyper-Representations: Self-Supervised Representation Learning on Neural Network Weights for Model Characteristic Predictioncited by 9
  30. 2022Predicting the generalization gap in neural networks using topological data analysiscited by 9
  31. 2021Ranking Models in Unlabeled New Environmentscited by 8
  32. 2021Well-classified Examples are Underestimated in Classification with Deep Neural Networkscited by 8
  33. 2022Deep Double Descent via Smooth Interpolationcited by 8
  34. 2024Accurate and Scalable Estimation of Epistemic Uncertainty for Graph Neural Networkscited by 8
  35. 2021Recent Advances in Large Margin Learningcited by 7
  36. 2023Selecting which Dense Retriever to use for Zero-Shot Searchcited by 7
  37. 2024Leveraging PAC-Bayes Theory and Gibbs Distributions for Generalization Bounds with Complexity Measurescited by 7
  38. 2021On Predicting Generalization using GANscited by 6

50 most cited of 111 All 111 (JSON)