mlsanityData to 14 Oct 2026

Rethinking generalization of classifiers in separable classes scenarios and over-parameterized regimes

arXiv 2410.16868 ↗First posted 22 Oct 2024cs.LG cs.CVcited by 0

Not on the map. Why?

Abstract (arXiv)

We investigate the learning dynamics of classifiers in scenarios where classes are separable or classifiers are over-parameterized. In both cases, Empirical Risk Minimization (ERM) results in zero training error. However, there are many global minima with a training error of zero, some of which generalize well and some of which do not. We show that in separable classes scenarios the proportion of "bad" global minima diminishes exponentially with the number of training data n. Our analysis provides bounds and learning curves dependent solely on the density distribution of the true error for the given classifier function set, irrespective of the set's size or complexity (e.g., number of parameters). This observation may shed light on the unexpectedly good generalization of over-parameterized Neural Networks. For the over-parameterized scenario, we propose a model for the density distribution of the true error, yielding learning curves that align with experiments on MNIST and CIFAR-10.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 34 of its 55 references that are in the map. First posted on arXiv 22 Oct 2024.

Builds on

34 of 55 references.

References · 34

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  4. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,611
  5. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,621
  6. 2016Understanding deep learning requires rethinking generalizationcited by 1,674
  7. 2018Reconciling modern machine learning practice and the bias-variance trade-offcited by 927
  8. 2019Wide Neural Networks of Any Depth Evolve as Linear Models Under Gradient Descentcited by 738
  9. 2017Spectrally-normalized margin bounds for neural networkscited by 686
  10. 2018On Lazy Training in Differentiable Programmingcited by 675
  11. 2019Large Batch Optimization for Deep Learning: Training BERT in 76 minutescited by 674
  12. 2017The Implicit Bias of Gradient Descent on Separable Datacited by 655
22 more
  1. 2017Exploring Generalization in Deep Learningcited by 600
  2. 2019On Exact Computation with an Infinitely Wide Neural Netcited by 573
  3. 2019Fine-Grained Analysis of Optimization and Generalization for Overparameterized Two-Layer Neural Networkscited by 566
  4. 2015Train faster, generalize better: Stability of stochastic gradient descentcited by 552
  5. 2019Fantastic Generalization Measures and Where to Find Themcited by 513
  6. 2018Learning and Generalization in Overparameterized Neural Networks, Going Beyond Two Layerscited by 340
  7. 2019Gradient Descent Maximizes the Margin of Homogeneous Neural Networkscited by 320
  8. 2018Stronger generalization bounds for deep nets via a compression approachcited by 307
  9. 2015Norm-Based Capacity Control in Neural Networkscited by 299
  10. 2017Size-Independent Sample Complexity of Neural Networkscited by 290
  11. 2018Learning Overparameterized Neural Networks via Stochastic Gradient Descent on Structured Datacited by 261
  12. 2017High-dimensional dynamics of generalization error in neural networkscited by 241
  13. 2019Uniform convergence may be unable to explain generalization in deep learningcited by 224
  14. 2017Fisher-Rao Metric, Geometry, and Complexity of Neural Networkscited by 100
  15. 2022On the Implicit Bias in Deep-Learning Algorithmscited by 86
  16. 2017Generalization Bounds of SGLD for Non-convex Learning: Two Theoretical Viewpointscited by 82
  17. 2017SGD Learns Over-parameterized Networks that Provably Generalize on Linearly Separable Datacited by 79
  18. 2019Data-dependent Sample Complexity of Deep Neural Networks via Lipschitz Augmentationcited by 71
  19. 2019Deterministic PAC-Bayesian generalization bounds for deep networks via generalizing noise-resiliencecited by 51
  20. 2022Stability and Generalization Analysis of Gradient Methods for Shallow Neural Networkscited by 21
  21. 2022Large Neural Networks Learning from Scratch with Very Few Data and without Explicit Regularizationcited by 3
  22. 2022Do highly over-parameterized neural networks generalize since bad solutions are rare?cited by 1

Built on by

Not cited yet.