mlsanityData to 7 Oct 2026

Tighter Expected Generalization Error Bounds via Convexity of Information Measures

arXiv 2202.12150 ↗First posted 24 Feb 2022cs.IT cs.LG math.ITcited by 11

Not on the map. Why?

Abstract (arXiv)

Generalization error bounds are essential to understanding machine learning algorithms. This paper presents novel expected generalization error upper bounds based on the average joint distribution between the output hypothesis and each input training sample. Multiple generalization error upper bounds based on different information measures are provided, including Wasserstein distance, total variation distance, KL divergence, and Jensen-Shannon divergence. Due to the convexity of the information measures, the proposed bounds in terms of Wasserstein distance and total variation distance are shown to be tighter than their counterparts based on individual samples in the literature. An example is provided to demonstrate the tightness of the proposed generalization error bounds.

Key facts. Cited by 11 papers in the map (data through 7 Oct 2026). Builds on 12 of its 32 references that are in the map. First posted on arXiv 24 Feb 2022.

Builds on

12 of 32 references.

References · 12

  1. 2017Information-theoretic analysis of generalization capability of learning algorithmscited by 299
  2. 2020Reasoning About Generalization via Conditional Mutual Informationcited by 145
  3. 2019Tightening Mutual Information Based Bounds on Generalization Errorcited by 117
  4. 2015How much does your data exploration overfit? Controlling bias via information usagecited by 95
  5. 2020Sharpened Generalization Bounds based on Conditional Mutual Information and an Application to Noisy, Iterative Algorithmscited by 89
  6. 2018Chaining Mutual Information and Tightening Generalization Boundscited by 65
  7. 2019Generalization Error Bounds Via R'enyi-, f-Divergences and Maximal Leakagecited by 48
  8. 2021Tighter expected generalization error bounds via Wasserstein distancecited by 43
  9. 2020Individually Conditional Individual Mutual Information Bound on Generalization Errorcited by 30
  10. 2020Jensen-Shannon Information Based Characterization of the Generalization Error of Learning Algorithmscited by 19
  11. 2019Information-Theoretic Understanding of Population Risk Improvement with Model Compressioncited by 11
  12. 2014PAC-Bayes Mini-tutorial: A Continuous Union Boundcited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 4; 2023: 3; 2024: 2; 2025: 0; 2026: 2.

11 papers.

  1. 2022Integral Probability Metrics PAC-Bayes Boundscited by 19
  2. 2022Empirical Risk Minimization with Relative Entropy Regularizationcited by 13
  3. 2023On the Validation of Gibbs Algorithms: Training Datasets, Test Datasets and their Aggregationcited by 10
  4. 2022f-divergences and their applications in lossy compression and bounding generalization errorcited by 7
  5. 2023Generalization Analysis of Machine Learning Algorithms via the Worst-Case Data-Generating Probability Measurecited by 7
  6. 2022Learning Algorithm Generalization Error Bounds via Auxiliary Distributionscited by 6
  7. 2024The Method of Gaps: Exact Expressions for the Generalization Error of Supervised Learning Algorithmscited by 5
  8. 2023Exactly Tight Information-Theoretic Generalization Error Bound for the Quadratic Gaussian Problemcited by 4
  9. 2024How Does Distribution Matching Help Domain Generalization: An Information-theoretic Analysiscited by 3
  10. 2026Empirical Risk Minimization with f-Divergence Regularizationcited by 0
  11. 2026Tighter Information-Theoretic Generalization Bounds via a Novel Class of Change of Measure Inequalitiescited by 0