mlsanityData to 7 Oct 2026

Information-theoretic analysis of generalization capability of learning algorithms

arXiv 1705.07809 ↗First posted 22 May 2017cs.LG cs.IT math.ITcited by 299

Not on the map. Why?

Abstract (arXiv)

We derive upper bounds on the generalization error of a learning algorithm in terms of the mutual information between its input and output. The bounds provide an information-theoretic understanding of generalization in learning problems, and give theoretical guidelines for striking the right balance between data fit and generalization by controlling the input-output mutual information. We propose a number of methods for this purpose, among which are algorithms that regularize the ERM algorithm with relative entropy or with random noise. Our work extends and leads to nontrivial improvements on the recent results of Russo and Zou.

Key facts. Cited by 299 papers in the map (data through 7 Oct 2026). First posted on arXiv 22 May 2017.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 36; 2022: 63; 2023: 59; 2024: 40; 2025: 48; 2026: 53.

299 papers.

Sort:

By area:

  1. 2021User-friendly introduction to PAC-Bayes boundscited by 234
  2. 2023How Does Information Bottleneck Help Deep Learning?cited by 92
  3. 2021Information-Theoretic Generalization Bounds for Stochastic Gradient Descentcited by 84
  4. 2022What's Behind the Mask: Understanding Masked Graph Modeling for Graph Autoencoderscited by 82
  5. 2024Asymmetry in Low-Rank Adapters of Foundation Modelscited by 81
  6. 2023To Compress or Not to Compress- Self-Supervised Learning and Information Theory: A Reviewcited by 72
  7. 2021Information-theoretic generalization bounds for black-box learning algorithmscited by 49
  8. 2021Generalization Bounds For Meta-Learning: An Information-Theoretic Analysiscited by 45
  9. 2021Tighter expected generalization error bounds via Wasserstein distancecited by 43
  10. 2023Stability is Stable: Connections between Replicability, Privacy, and Adaptive Generalizationcited by 42
  11. 2021PRIME: A few primitives can boost robustness to common corruptionscited by 40
  12. 2025Think or Not? Exploring Thinking Efficiency in Large Reasoning Models via an Information-Theoretic Lenscited by 32
38 more
  1. 2021PAC-Bayes, MAC-Bayes and Conditional Mutual Information: Fast rate bounds that handle general VC classescited by 31
  2. 2021On the Generalization of Models Trained with SGD: Information-Theoretic Bounds and Implicationscited by 31
  3. 2023On the Generalization of Diffusion Modelcited by 30
  4. 2021Towards a Unified Information-Theoretic Framework for Generalizationcited by 29
  5. 2021PAC-Bayes Information Bottleneckcited by 28
  6. 2022A New Family of Generalization Bounds Using Samplewise Evaluated CMIcited by 28
  7. 2023Randomized Adversarial Training via Taylor Expansioncited by 28
  8. 2021Generalization Bounds using Lower Tail Exponents in Stochastic Optimizerscited by 27
  9. 2021Time-independent Generalization Bounds for SGLD in Non-convex Settingscited by 27
  10. 2022Generalization Bounds via Convex Analysiscited by 27
  11. 2022Single Trajectory Nonparametric Learning of Nonlinear Dynamicscited by 26
  12. 2022Stability and Generalization of Stochastic Optimization with Nonconvex and Nonsmooth Problemscited by 25
  13. 2023An Information-Theoretic Perspective on Variance-Invariance-Covariance Regularizationcited by 25
  14. 2024Knowledge Overshadowing Causes Amalgamated Hallucination in Large Language Modelscited by 24
  15. 2021Fractal Structure and Generalization Properties of Stochastic Optimization Algorithmscited by 23
  16. 2022Rate-Distortion Theoretic Generalization Bounds for Stochastic Learning Algorithmscited by 22
  17. 2024Mutual Information Estimation via Normalizing Flowscited by 22
  18. 2023Tighter Information-Theoretic Generalization Bounds from Supersamplescited by 21
  19. 2026There Will Be a Scientific Theory of Deep Learningcited by 21
  20. 2021RATT: Leveraging Unlabeled Data to Guarantee Generalizationcited by 20
  21. 2024Position: Understanding LLMs Requires More Than Statistical Generalizationcited by 20
  22. 2022Quantifying the Knowledge in a DNN to Explain Knowledge Distillation for Classificationcited by 19
  23. 2023Information Bottleneck Analysis of Deep Neural Networks via Lossy Compressioncited by 19
  24. 2023Fantastic Generalization Measures are Nowhere to be Foundcited by 19
  25. 2021Tight Mutual Information Estimation With Contrastive Fenchel-Legendre Optimizationcited by 18
  26. 2022An Information-theoretical Approach to Semi-supervised Learning under Covariate-shiftcited by 18
  27. 2022Benign Underfitting of Stochastic Gradient Descentcited by 18
  28. 2022Limitations of Information-Theoretic Generalization Bounds for Gradient Descent Methods in Stochastic Convex Optimizationcited by 18
  29. 2025Does Generation Require Memorization? Creative Diffusion Models using Ambient Diffusioncited by 17
  30. 2022Understanding Generalization via Leave-One-Out Conditional Mutual Informationcited by 16
  31. 2022On Rademacher Complexity-based Generalization Bounds for Deep Learningcited by 16
  32. 2023Information Theoretic Lower Bounds for Information Theoretic Upper Boundscited by 16
  33. 2021Generalization Bounds for Noisy Iterative Algorithms Using Properties of Additive Noise Channelscited by 15
  34. 2022What Do We Maximize in Self-Supervised Learning?cited by 15
  35. 2022Information-Theoretic Analysis of Unsupervised Domain Adaptationcited by 15
  36. 2023Algorithmic Stability of Heavy-Tailed SGD with General Loss Functionscited by 15
  37. 2023Generalization Bounds with Data-dependent Fractal Dimensionscited by 15
  38. 2024Towards a Theoretical Understanding of Synthetic Data in LLM Post-Training: A Reverse-Bottleneck Perspectivecited by 15

50 most cited of 299 All 299 (JSON)