mlsanityData to 14 Oct 2026

Generalization Bounds via Conditional f-Information

arXiv 2410.22887 ↗First posted 30 Oct 2024stat.ML cs.IT cs.LGcited by 4

Not on the map. Why?

Abstract (arXiv)

In this work, we introduce novel information-theoretic generalization bounds using the conditional f-information framework, an extension of the traditional conditional mutual information (MI) framework. We provide a generic approach to derive generalization bounds via f-information in the supersample setting, applicable to both bounded and unbounded loss functions. Unlike previous MI-based bounds, our proof strategy does not rely on upper bounding the cumulant-generating function (CGF) in the variational formula of MI. Instead, we set the CGF or its upper bound to zero by carefully selecting the measurable function invoked in the variational formula. Although some of our techniques are partially inspired by recent advances in the coin-betting framework (e.g., Jang et al. (2023)), our results are independent of any previous findings from regret guarantees of online gambling algorithms. Additionally, our newly derived MI-based bound recovers many previous results and improves our understanding of their potential limitations. Finally, we empirically compare various f-information measures for generalization, demonstrating the improvement of our new bounds over the previous bounds.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Builds on 42 of its 63 references that are in the map. First posted on arXiv 30 Oct 2024.

Builds on

42 of 63 references.

References · 42

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2017Information-theoretic analysis of generalization capability of learning algorithmscited by 303
  3. 2020Reasoning About Generalization via Conditional Mutual Informationcited by 147
  4. 2019Information-Theoretic Generalization Bounds for SGLD via Data-Dependent Estimatescited by 119
  5. 2019Tightening Mutual Information Based Bounds on Generalization Errorcited by 117
  6. 2015How much does your data exploration overfit? Controlling bias via information usagecited by 95
  7. 2023How Does Information Bottleneck Help Deep Learning?cited by 93
  8. 2020Sharpened Generalization Bounds based on Conditional Mutual Information and an Application to Noisy, Iterative Algorithmscited by 89
  9. 2021Information-Theoretic Generalization Bounds for Stochastic Gradient Descentcited by 84
  10. 2018Generalization Error Bounds for Noisy, Iterative Algorithmscited by 82
  11. 2018Chaining Mutual Information and Tightening Generalization Boundscited by 65
  12. 2023Generalization Bounds: Perspectives from Information Theory and PAC-Bayescited by 61
30 more
  1. 2021Information-theoretic generalization bounds for black-box learning algorithmscited by 49
  2. 2021Tighter expected generalization error bounds via Wasserstein distancecited by 43
  3. 2021Tight Concentrations and Confidence Sequences from the Regret of Universal Portfoliocited by 43
  4. 2021PAC-Bayes, MAC-Bayes and Conditional Mutual Information: Fast rate bounds that handle general VC classescited by 31
  5. 2021On the Generalization of Models Trained with SGD: Information-Theoretic Bounds and Implicationscited by 31
  6. 2020Individually Conditional Individual Mutual Information Bound on Generalization Errorcited by 30
  7. 2019Parameter-Free Online Convex Optimization with Sub-Exponential Noisecited by 28
  8. 2020On Random Subset Generalization Error Bounds and the Stochastic Gradient Langevin Dynamics Algorithmcited by 28
  9. 2022A New Family of Generalization Bounds Using Samplewise Evaluated CMIcited by 28
  10. 2020Optimal Bounds between f-Divergences and Integral Probability Metricscited by 27
  11. 2022Generalization Bounds via Convex Analysiscited by 27
  12. 2023Tighter Information-Theoretic Generalization Bounds from Supersamplescited by 22
  13. 2022Limitations of Information-Theoretic Generalization Bounds for Gradient Descent Methods in Stochastic Convex Optimizationcited by 19
  14. 2023Information Theoretic Lower Bounds for Information Theoretic Upper Boundscited by 17
  15. 2022Understanding Generalization via Leave-One-Out Conditional Mutual Informationcited by 16
  16. 2020Optimizing Variational Representations of Divergences and Accelerating their Statistical Estimationcited by 15
  17. 2023Online-to-PAC Conversions: Generalization Bounds via Regret Analysiscited by 14
  18. 2023Tighter PAC-Bayes Bounds Through Coin-Bettingcited by 13
  19. 2023Time-Independent Information-Theoretic Generalization Bounds for SGLDcited by 12
  20. 2022Two Facets of SDE Under an Information-Theoretic Lens: Generalization of SGD via Training Trajectories and via Terminal Statescited by 11
  21. 2022Optimal Comparator Adaptive Online Learning with Switching Costcited by 9
  22. 2022On Leave-One-Out Conditional Mutual Information For Generalizationcited by 8
  23. 2022From Generalisation Error to Transportation-cost Inequalities and Backcited by 6
  24. 2022Fast Rate Generalization Error Bounds: Variations on a Themecited by 6
  25. 2023Sample-Conditioned Hypothesis Stability Sharpens Information-Theoretic Generalization Boundscited by 6
  26. 2024On f-Divergence Principled Domain Adaptation: An Improved Frameworkcited by 6
  27. 2023Exactly Tight Information-Theoretic Generalization Error Bound for the Quadratic Gaussian Problemcited by 4
  28. 2024Better-than-KL PAC-Bayes Boundscited by 4
  29. 2024An Information-Theoretic Framework for Out-of-Distribution Generalization with Applications to Stochastic Gradient Langevin Dynamicscited by 3
  30. 2024Tighter Generalisation Bounds via Interpolationcited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 2.

4 papers.

  1. 2025Generalization in Federated Learning: A Conditional Mutual Information Frameworkcited by 2
  2. 2025A fine-grained look at causal effects in causal spacescited by 0
  3. 2026Tighter Information-Theoretic Generalization Bounds via a Novel Class of Change of Measure Inequalitiescited by 0
  4. 2026An Accuracy-Information Tradeoff for Loss-Difference Conditional Mutual Informationcited by 0