mlsanityData to 7 Oct 2026

Tighter Information-Theoretic Generalization Bounds from Supersamples

arXiv 2302.02432 ↗First posted 5 Feb 2023stat.ML cs.IT cs.LGcited by 21

Not on the map. Why?

Abstract (arXiv)

In this work, we present a variety of novel information-theoretic generalization bounds for learning algorithms, from the supersample setting of Steinke & Zakynthinou (2020)-the setting of the "conditional mutual information" framework. Our development exploits projecting the loss pair (obtained from a training instance and a testing instance) down to a single number and correlating loss values with a Rademacher sequence (and its shifted variants). The presented bounds include square-root bounds, fast-rate bounds, including those based on variance and sharpness, and bounds for interpolating algorithms etc. We show theoretically or empirically that these bounds are tighter than all information-theoretic bounds known to date on the same supersample setting.

Key facts. Cited by 21 papers in the map (data through 7 Oct 2026). Builds on 32 of its 62 references that are in the map. First posted on arXiv 5 Feb 2023.

Builds on

32 of 62 references.

References · 32

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2016Understanding deep learning requires rethinking generalizationcited by 1,657
  3. 2017Information-theoretic analysis of generalization capability of learning algorithmscited by 299
  4. 2017Three Factors Influencing Minima in SGDcited by 270
  5. 2017Non-convex learning via Stochastic Gradient Langevin Dynamics: a nonasymptotic analysiscited by 242
  6. 2020Reasoning About Generalization via Conditional Mutual Informationcited by 145
  7. 2019Information-Theoretic Generalization Bounds for SGLD via Data-Dependent Estimatescited by 118
  8. 2019Tightening Mutual Information Based Bounds on Generalization Errorcited by 117
  9. 2015How much does your data exploration overfit? Controlling bias via information usagecited by 95
  10. 2020Sharpened Generalization Bounds based on Conditional Mutual Information and an Application to Noisy, Iterative Algorithmscited by 89
  11. 2021Information-Theoretic Generalization Bounds for Stochastic Gradient Descentcited by 84
  12. 2018Generalization Error Bounds for Noisy, Iterative Algorithmscited by 79
20 more
  1. 2018Chaining Mutual Information and Tightening Generalization Boundscited by 65
  2. 2021Information-theoretic generalization bounds for black-box learning algorithmscited by 49
  3. 2020Generalization Bounds via Information Density and Conditional Information Densitycited by 43
  4. 2021Tighter expected generalization error bounds via Wasserstein distancecited by 43
  5. 2021PAC-Bayes, MAC-Bayes and Conditional Mutual Information: Fast rate bounds that handle general VC classescited by 31
  6. 2021On the Generalization of Models Trained with SGD: Information-Theoretic Bounds and Implicationscited by 31
  7. 2020Individually Conditional Individual Mutual Information Bound on Generalization Errorcited by 30
  8. 2021Towards a Unified Information-Theoretic Framework for Generalizationcited by 29
  9. 2020On Random Subset Generalization Error Bounds and the Stochastic Gradient Langevin Dynamics Algorithmcited by 28
  10. 2022A New Family of Generalization Bounds Using Samplewise Evaluated CMIcited by 28
  11. 2020Information-theoretic analysis for transfer learningcited by 23
  12. 2019Fast-rate PAC-Bayes Generalization Bounds via Shifted Rademacher Processescited by 20
  13. 2022Limitations of Information-Theoretic Generalization Bounds for Gradient Descent Methods in Stochastic Convex Optimizationcited by 18
  14. 2022Understanding Generalization via Leave-One-Out Conditional Mutual Informationcited by 16
  15. 2022Information-Theoretic Analysis of Unsupervised Domain Adaptationcited by 15
  16. 2021Information-Theoretic Characterization of the Generalization Error for Iterative Semi-Supervised Learningcited by 11
  17. 2022Evaluated CMI Bounds for Meta Learning: Tightness and Expressivenesscited by 11
  18. 2022Two Facets of SDE Under an Information-Theoretic Lens: Generalization of SGD via Training Trajectories and via Terminal Statescited by 11
  19. 2022Chained Generalisation Boundscited by 10
  20. 2022On Leave-One-Out Conditional Mutual Information For Generalizationcited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 5; 2024: 7; 2025: 5; 2026: 4.

21 papers.

Sort:

By area:

  1. 2023Fantastic Generalization Measures are Nowhere to be Foundcited by 19
  2. 2023Time-Independent Information-Theoretic Generalization Bounds for SGLDcited by 12
  3. 2024Information-theoretic Generalization Analysis for Expected Calibration Errorcited by 9
  4. 2024On f-Divergence Principled Domain Adaptation: An Improved Frameworkcited by 6
  5. 2023Comparing Comparators in Generalization Boundscited by 5
  6. 2023Sample-Conditioned Hypothesis Stability Sharpens Information-Theoretic Generalization Boundscited by 5
  7. 2024Slicing Mutual Information Generalization Bounds for Neural Networkscited by 4
  8. 2024How Does Distribution Matching Help Domain Generalization: An Information-theoretic Analysiscited by 3
  9. 2024Generalization Bounds via Conditional f-Informationcited by 3
  10. 2025Fairness Overfitting in Machine Learning: An Information-Theoretic Perspectivecited by 3
  11. 2023Information-Theoretic Generalization Bounds for Transductive Learning and its Applicationscited by 2
  12. 2024Class-wise Generalization Error: an Information-Theoretic Analysiscited by 2
9 more
  1. 2025Generalization in Federated Learning: A Conditional Mutual Information Frameworkcited by 2
  2. 2025Generalization in VAE and Diffusion Models: A Unified Information-Theoretic Analysiscited by 2
  3. 2025Information-Theoretic Generalization Bounds of Replay-based Continual Learningcited by 2
  4. 2026Leveraging Flatness to Improve Information-Theoretic Generalization Bounds for SGDcited by 2
  5. 2024Which Algorithms Have Tight Generalization Bounds?cited by 1
  6. 2025Information-theoretic Generalization Analysis for VQ-VAEs: A Role of Latent Variablescited by 0
  7. 2026When Do Early-Exit Networks Generalize? A PAC-Bayesian Theory of Adaptive Depthcited by 0
  8. 2026Information-Theoretic Generalization Bounds for Sequential Decision Makingcited by 0
  9. 2026Towards the Connection between Activation Sparsity and Flat Minimacited by 0