mlsanityData to 14 Oct 2026

Inconsistency, Instability, and Generalization Gap of Deep Neural Network Training

arXiv 2306.00169 ↗First posted 31 May 2023cs.LGcited by 6

Not on the map. Why?

Abstract (arXiv)

As deep neural networks are highly expressive, it is important to find solutions with small generalization gap (the difference between the performance on the training data and unseen data). Focusing on the stochastic nature of training, we first present a theoretical analysis in which the bound of generalization gap depends on what we call inconsistency and instability of model outputs, which can be estimated on unlabeled data. Our empirical study based on this analysis shows that instability and inconsistency are strongly predictive of generalization gap in various settings. In particular, our finding indicates that inconsistency is a more reliable indicator of generalization gap than the sharpness of the loss landscape. Furthermore, we show that algorithmic reduction of inconsistency leads to superior performance. The results also provide a theoretical basis for existing methods such as co-distillation and ensemble.

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Builds on 33 of its 42 references that are in the map. First posted on arXiv 31 May 2023.

Builds on

33 of 42 references.

References · 33

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  4. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  5. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,442
  6. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,183
  7. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  8. 2016Wide Residual Networkscited by 3,070
  9. 2019CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Featurescited by 2,884
  10. 2019RandAugment: Practical automated data augmentation with a reduced search spacecited by 2,269
  11. 2020FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidencecited by 2,085
  12. 2017Improved Regularization of Convolutional Neural Networks with Cutoutcited by 1,628
21 more
  1. 2021MLP-Mixer: An all-MLP Architecture for Visioncited by 1,446
  2. 2020Sharpness-Aware Minimization for Efficiently Improving Generalizationcited by 1,417
  3. 2019MixMatch: A Holistic Approach to Semi-Supervised Learningcited by 1,391
  4. 2016On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimacited by 1,352
  5. 2018Averaging Weights Leads to Wider Optima and Better Generalizationcited by 1,209
  6. 2019Unsupervised Data Augmentation for Consistency Trainingcited by 1,060
  7. 2016Deep Networks with Stochastic Depthcited by 979
  8. 2017Deep Mutual Learningcited by 628
  9. 2019Fantastic Generalization Measures and Where to Find Themcited by 513
  10. 2017Information-theoretic analysis of generalization capability of learning algorithmscited by 303
  11. 2021When Vision Transformers Outperform ResNets without Pre-training or Strong Data Augmentationscited by 260
  12. 2018Large scale distributed neural network training through online distillationcited by 146
  13. 2021Assessing Generalization of SGD via Disagreementcited by 117
  14. 2015How much does your data exploration overfit? Controlling bias via information usagecited by 95
  15. 2017Shake-Shake regularizationcited by 87
  16. 2021Information-Theoretic Generalization Bounds for Stochastic Gradient Descentcited by 84
  17. 2020Distributional Generalization: A New Kind of Generalizationcited by 42
  18. 2019Budgeted Training: Rethinking Deep Neural Network Training Under Resource Constraintscited by 37
  19. 2018Iterate averaging as regularization for stochastic gradient descentcited by 27
  20. 2022A Note on "Assessing Generalization of SGD via Disagreement"cited by 16
  21. 2020Guided Learning of Nonconvex Models through Successive Functional Gradient Optimizationcited by 7

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 1; 2024: 0; 2025: 3; 2026: 2.

6 papers.

  1. 2023Soft ascent-descent as a stable and flexible alternative to floodingcited by 3
  2. 2025NeuralGrok: Accelerate Grokking by Neural Gradient Transformationcited by 2
  3. 2025Towards Better Generalization via Distributional Input Projection Networkcited by 1
  4. 2026Model Agreement via Anchoringcited by 1
  5. 2025Using Synthetic Data to estimate the True Error is theoretically and practically doablecited by 0
  6. 2026Inconsistency-Aware Minimization: Improving Generalization with Unlabeled Datacited by 0