mlsanityData to 14 Oct 2026

Information-theoretic generalization bounds for black-box learning algorithms

arXiv 2110.01584 ↗First posted 4 Oct 2021cs.LG stat.MLcited by 49

Not on the map. Why?

Abstract (arXiv)

We derive information-theoretic generalization bounds for supervised learning algorithms based on the information contained in predictions rather than in the output of the training algorithm. These bounds improve over the existing information-theoretic bounds, are applicable to a wider range of algorithms, and solve two key challenges: (a) they give meaningful results for deterministic algorithms and (b) they are significantly easier to estimate. We show experimentally that the proposed bounds closely follow the generalization gap in practical scenarios for deep learning.

Key facts. Cited by 49 papers in the map (data through 14 Oct 2026). Builds on 20 of its 42 references that are in the map. First posted on arXiv 4 Oct 2021.

Builds on

20 of 42 references.

References · 20

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  3. 2016Membership Inference Attacks against Machine Learning Modelscited by 2,281
  4. 2016Understanding deep learning requires rethinking generalizationcited by 1,674
  5. 2019Fantastic Generalization Measures and Where to Find Themcited by 513
  6. 2017Information-theoretic analysis of generalization capability of learning algorithmscited by 303
  7. 2020Reasoning About Generalization via Conditional Mutual Informationcited by 147
  8. 2019Information-Theoretic Generalization Bounds for SGLD via Data-Dependent Estimatescited by 119
  9. 2019Tightening Mutual Information Based Bounds on Generalization Errorcited by 117
  10. 2015How much does your data exploration overfit? Controlling bias via information usagecited by 95
  11. 2020Sharpened Generalization Bounds based on Conditional Mutual Information and an Application to Noisy, Iterative Algorithmscited by 89
  12. 2015Algorithmic Stability for Adaptive Data Analysiscited by 88
8 more
  1. 2021Information-Theoretic Generalization Bounds for Stochastic Gradient Descentcited by 84
  2. 2018Generalization Error Bounds for Noisy, Iterative Algorithmscited by 82
  3. 2019Generalization Error Bounds Via R'enyi-, f-Divergences and Maximal Leakagecited by 48
  4. 2017Calibrating Noise to Variance in Adaptive Data Analysiscited by 29
  5. 2020On Random Subset Generalization Error Bounds and the Stochastic Gradient Langevin Dynamics Algorithmcited by 28
  6. 2021Membership Inference Attacks on Deep Regression Models for Neuroimagingcited by 22
  7. 2021Estimating informativeness of samples with Smooth Unique Informationcited by 15
  8. 2016On-Average KL-Privacy and its equivalence to Generalization for Max-Entropy Mechanismscited by 10

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 10; 2023: 11; 2024: 10; 2025: 10; 2026: 8.

49 papers.

Sort:

By area:

  1. 2022A New Family of Generalization Bounds Using Samplewise Evaluated CMIcited by 28
  2. 2024Knowledge Overshadowing Causes Amalgamated Hallucination in Large Language Modelscited by 25
  3. 2022Rate-Distortion Theoretic Generalization Bounds for Stochastic Learning Algorithmscited by 22
  4. 2023Tighter Information-Theoretic Generalization Bounds from Supersamplescited by 22
  5. 2022Limitations of Information-Theoretic Generalization Bounds for Gradient Descent Methods in Stochastic Convex Optimizationcited by 19
  6. 2023Fantastic Generalization Measures are Nowhere to be Foundcited by 19
  7. 2022Understanding Generalization via Leave-One-Out Conditional Mutual Informationcited by 16
  8. 2023Generalization Bounds with Data-dependent Fractal Dimensionscited by 15
  9. 2022Rate-Distortion Theoretic Bounds on Generalization Error for Distributed Learningcited by 14
  10. 2024Information Complexity of Stochastic Convex Optimization: Applications to Generalization and Memorizationcited by 14
  11. 2023Time-Independent Information-Theoretic Generalization Bounds for SGLDcited by 12
  12. 2022On the Generalization for Transfer Learning: An Information-Theoretic Analysiscited by 11
37 more
  1. 2022Evaluated CMI Bounds for Meta Learning: Tightness and Expressivenesscited by 11
  2. 2024Minimum Description Length and Generalization Guarantees for Representation Learningcited by 11
  3. 2024Information-Theoretic Generalization Bounds for Deep Neural Networkscited by 10
  4. 2024Information-theoretic Generalization Analysis for Expected Calibration Errorcited by 10
  5. 2023Data-dependent Generalization Bounds via Variable-Size Compressibilitycited by 9
  6. 2022On Leave-One-Out Conditional Mutual Information For Generalizationcited by 8
  7. 2023A unified framework for information-theoretic generalization boundscited by 6
  8. 2023Sample-Conditioned Hypothesis Stability Sharpens Information-Theoretic Generalization Boundscited by 6
  9. 2023Recognizable Information Bottleneckcited by 5
  10. 2023Understanding the Generalization Ability of Deep Learning Algorithms: A Kernelized Renyi's Entropy Perspectivecited by 5
  11. 2023Comparing Comparators in Generalization Boundscited by 5
  12. 2024Slicing Mutual Information Generalization Bounds for Neural Networkscited by 4
  13. 2024Generalization Bounds via Conditional f-Informationcited by 4
  14. 2022Formal limitations of sample-wise information-theoretic generalization boundscited by 3
  15. 2024How Does Distribution Matching Help Domain Generalization: An Information-theoretic Analysiscited by 3
  16. 2025Fairness Overfitting in Machine Learning: An Information-Theoretic Perspectivecited by 3
  17. 2025Fisher information flow in artificial neural networkscited by 3
  18. 2023Information-Theoretic Generalization Bounds for Transductive Learning and its Applicationscited by 2
  19. 2024Class-wise Generalization Error: an Information-Theoretic Analysiscited by 2
  20. 2025Generalization Guarantees for Representation Learning via Data-Dependent Gaussian Mixture Priorscited by 2
  21. 2025Generalization in Federated Learning: A Conditional Mutual Information Frameworkcited by 2
  22. 2025Information-Theoretic Generalization Bounds of Replay-based Continual Learningcited by 2
  23. 2026Leveraging Flatness to Improve Information-Theoretic Generalization Bounds for SGDcited by 2
  24. 2022Black-Box Generalization: Stability of Zeroth-Order Learningcited by 1
  25. 2024Which Algorithms Have Tight Generalization Bounds?cited by 1
  26. 2025Multiview Representation Learning via Distributed Joint Latent Space Structuringcited by 1
  27. 2025MEL: Multi-level Ensemble Learning for Resource-Constrained Environmentscited by 1
  28. 2025Information-theoretic Generalization Analysis for VQ-VAEs: A Role of Latent Variablescited by 0
  29. 2025Information Theory in Open-world Machine Learning Foundations, Frameworks, and Future Directioncited by 0
  30. 2025An Information-Theoretic Analysis of OOD Generalization in Meta-Reinforcement Learningcited by 0
  31. 2026Pool-based Active Learning as Noisy Lossy Compression: Characterizing Label Complexity via Finite Blocklength Analysiscited by 0
  32. 2026Tighter Information-Theoretic Generalization Bounds via a Novel Class of Change of Measure Inequalitiescited by 0
  33. 2026Block-Sample MAC-Bayes Generalization Boundscited by 0
  34. 2026A Hierarchical Sampling Framework for bounding the Generalization Error of Federated Learningcited by 0
  35. 2026Information-Theoretic Generalization Bounds for Sequential Decision Makingcited by 0
  36. 2026Drift and Dependence: Layer-wise Information-Theoretic Bounds for Replay-Based Continual Learningcited by 0
  37. 2026Information-Theoretic Analysis of Next-Token Prediction under Markovian Datacited by 0