mlsanityData to 14 Oct 2026

A unified framework for information-theoretic generalization bounds

arXiv 2305.11042 ↗First posted 18 May 2023cs.LG cs.IT math.ITcited by 6

Not on the map. Why?

Abstract (arXiv)

This paper presents a general methodology for deriving information-theoretic generalization bounds for learning algorithms. The main technical tool is a probabilistic decorrelation lemma based on a change of measure and a relaxation of Young's inequality in Lpsip Orlicz spaces. Using the decorrelation lemma in combination with other techniques, such as symmetrization, couplings, and chaining in the space of probability measures, we obtain new upper bounds on the generalization error, both in expectation and in high probability, and recover as special cases many of the existing generalization bounds, including the ones based on mutual information, conditional mutual information, stochastic chaining, and PAC-Bayes inequalities. In addition, the Fernique-Talagrand upper bound on the expected supremum of a subgaussian process emerges as a special case.

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Builds on 14 of its 34 references that are in the map. First posted on arXiv 18 May 2023.

Builds on

14 of 34 references.

References · 14

  1. 2017Information-theoretic analysis of generalization capability of learning algorithmscited by 303
  2. 2020Reasoning About Generalization via Conditional Mutual Informationcited by 147
  3. 2019Tightening Mutual Information Based Bounds on Generalization Errorcited by 117
  4. 2020Sharpened Generalization Bounds based on Conditional Mutual Information and an Application to Noisy, Iterative Algorithmscited by 89
  5. 2018Chaining Mutual Information and Tightening Generalization Boundscited by 65
  6. 2021Information-theoretic generalization bounds for black-box learning algorithmscited by 49
  7. 2019Generalization Error Bounds Via R'enyi-, f-Divergences and Maximal Leakagecited by 48
  8. 2020Generalization Bounds via Information Density and Conditional Information Densitycited by 43
  9. 2021Tighter expected generalization error bounds via Wasserstein distancecited by 43
  10. 2020Individually Conditional Individual Mutual Information Bound on Generalization Errorcited by 30
  11. 2021Towards a Unified Information-Theoretic Framework for Generalizationcited by 30
  12. 2021Generalization Bounds using Lower Tail Exponents in Stochastic Optimizerscited by 27
2 more
  1. 2022Generalization Bounds via Convex Analysiscited by 27
  2. 2022Chained Generalisation Boundscited by 10

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 1; 2024: 1; 2025: 1; 2026: 3.

6 papers.

  1. 2023Generalization Analysis of Machine Learning Algorithms via the Worst-Case Data-Generating Probability Measurecited by 7
  2. 2024Better-than-KL PAC-Bayes Boundscited by 4
  3. 2025An Information-Theoretic Analysis of OOD Generalization in Meta-Reinforcement Learningcited by 0
  4. 2026Tighter Information-Theoretic Generalization Bounds via a Novel Class of Change of Measure Inequalitiescited by 0
  5. 2026Tail-Aware Information-Theoretic Bounds for LLM Alignment under Heavy-Tailed Rewardscited by 0
  6. 2026Sample Complexity of Stochastic Optimization with Integer Variablescited by 0