mlsanityData to 7 Oct 2026

Error Bounds of Supervised Classification from Information-Theoretic Perspective

arXiv 2406.04567 ↗First posted 7 Jun 2024cs.LG cs.IRcited by 0

Not on the map. Why?

Abstract (arXiv)

In this paper, we explore bounds on the expected risk when using deep neural networks for supervised classification from an information theoretic perspective. Firstly, we introduce model risk and fitting error, which are derived from further decomposing the empirical risk. Model risk represents the expected value of the loss under the model's predicted probabilities and is exclusively dependent on the model. Fitting error measures the disparity between the empirical risk and model risk. Then, we derive the upper bound on fitting error, which links the back-propagated gradient and the model's parameter count with the fitting error. Furthermore, we demonstrate that the generalization errors are bounded by the classification uncertainty, which is characterized by both the smoothness of the distribution and the sample size. Based on the bounds on fitting error and generalization, by utilizing the triangle inequality, we establish an upper bound on the expected risk. This bound is applied to provide theoretical explanations for overparameterization, non-convex optimization and flat minima in deep learning. Finally, empirical verification confirms a significant positive correlation between the derived theoretical bounds and the practical expected risk, thereby affirming the practical relevance of the theoretical findings.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 41 of its 63 references that are in the map. First posted on arXiv 7 Jun 2024.

Builds on

41 of 63 references.

References · 41

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  3. 2019Searching for MobileNetV3cited by 1,984
  4. 2016Understanding deep learning requires rethinking generalizationcited by 1,657
  5. 2020Sharpness-Aware Minimization for Efficiently Improving Generalizationcited by 1,404
  6. 2016On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minimacited by 1,337
  7. 2018Reconciling modern machine learning practice and the bias-variance trade-offcited by 919
  8. 2018A Convergence Theory for Deep Learning via Over-Parameterizationcited by 784
  9. 2018On Lazy Training in Differentiable Programmingcited by 671
  10. 2018Gradient Descent Provably Optimizes Over-parameterized Neural Networkscited by 655
  11. 2017Opening the Black Box of Deep Neural Networks via Informationcited by 494
  12. 2018A Mean Field View of the Landscape of Two-Layers Neural Networkscited by 463
29 more
  1. 2017Sharp Minima Can Generalize For Deep Netscited by 462
  2. 2018On the Global Convergence of Gradient Descent for Over-parameterized Models using Optimal Transportcited by 422
  3. 2014In Search of the Real Inductive Bias: On the Role of Implicit Regularization in Deep Learningcited by 339
  4. 2017Information-theoretic analysis of generalization capability of learning algorithmscited by 299
  5. 2014The Loss Surfaces of Multilayer Networkscited by 278
  6. 2017Three Factors Influencing Minima in SGDcited by 270
  7. 2016Deep Learning without Poor Local Minimacited by 233
  8. 2021Invariance Principle Meets Information Bottleneck for Out-of-Distribution Generalizationcited by 220
  9. 2020The large learning rate phase of deep learning: the catapult mechanismcited by 206
  10. 2017Generalization in Deep Learningcited by 146
  11. 2018Stochastic Gradient Descent Optimizes Over-parameterized Deep ReLU Networkscited by 139
  12. 2019Information-Theoretic Generalization Bounds for SGLD via Data-Dependent Estimatescited by 118
  13. 2019Tightening Mutual Information Based Bounds on Generalization Errorcited by 117
  14. 2018Mean Field Analysis of Neural Networks: A Central Limit Theoremcited by 107
  15. 2020Sharpened Generalization Bounds based on Conditional Mutual Information and an Application to Noisy, Iterative Algorithmscited by 89
  16. 2021Information-Theoretic Generalization Bounds for Stochastic Gradient Descentcited by 84
  17. 2018Generalization Error Bounds for Noisy, Iterative Algorithmscited by 79
  18. 2019CARS: Continuous Evolution for Efficient Neural Architecture Searchcited by 79
  19. 2018Chaining Mutual Information and Tightening Generalization Boundscited by 65
  20. 2021Tighter expected generalization error bounds via Wasserstein distancecited by 43
  21. 2021On the Generalization of Models Trained with SGD: Information-Theoretic Bounds and Implicationscited by 31
  22. 2020Individually Conditional Individual Mutual Information Bound on Generalization Errorcited by 30
  23. 2018Small nonlinearities in activation functions create bad local minima in neural networkscited by 28
  24. 2022A New Family of Generalization Bounds Using Samplewise Evaluated CMIcited by 28
  25. 2020Information-theoretic analysis for transfer learningcited by 23
  26. 2021Information-Theoretic Characterization of the Generalization Error for Iterative Semi-Supervised Learningcited by 11
  27. 2022Evaluated CMI Bounds for Meta Learning: Tightness and Expressivenesscited by 11
  28. 2022Chained Generalisation Boundscited by 10
  29. 2022Annihilation of Spurious Minima in Two-Layer ReLU Networkscited by 9

Built on by

Not cited yet.