mlsanityData to 7 Oct 2026

Information-Theoretic Bounds on the Moments of the Generalization Error of Learning Algorithms

arXiv 2102.02016 ↗First posted 3 Feb 2021cs.IT cs.LG math.ITcited by 12

Not on the map. Why?

Abstract (arXiv)

Generalization error bounds are critical to understanding the performance of machine learning models. In this work, building upon a new bound of the expected value of an arbitrary function of the population and empirical risk of a learning algorithm, we offer a more refined analysis of the generalization behaviour of a machine learning models based on a characterization of (bounds) to their generalization error moments. We discuss how the proposed bounds -- which also encompass new bounds to the expected generalization error -- relate to existing bounds in the literature. We also discuss how the proposed generalization error moment bounds can be used to construct new generalization error high-probability bounds.

Key facts. Cited by 12 papers in the map (data through 7 Oct 2026). Builds on 16 of its 34 references that are in the map. First posted on arXiv 3 Feb 2021.

Builds on

16 of 34 references.

References · 16

  1. 2017Information-theoretic analysis of generalization capability of learning algorithmscited by 299
  2. 2020Reasoning About Generalization via Conditional Mutual Informationcited by 145
  3. 2019Information-Theoretic Generalization Bounds for SGLD via Data-Dependent Estimatescited by 118
  4. 2019Tightening Mutual Information Based Bounds on Generalization Errorcited by 117
  5. 2015How much does your data exploration overfit? Controlling bias via information usagecited by 95
  6. 2019Sharper bounds for uniformly stable algorithmscited by 91
  7. 2018Chaining Mutual Information and Tightening Generalization Boundscited by 65
  8. 2020PAC-Bayes Analysis Beyond the Usual Boundscited by 63
  9. 2016A Strongly Quasiconvex PAC-Bayesian Boundcited by 54
  10. 2015Risk Bounds for the Majority Vote: From a PAC-Bayesian Analysis to a Learning Algorithmcited by 50
  11. 2019Generalization Error Bounds Via R'enyi-, f-Divergences and Maximal Leakagecited by 48
  12. 2016Simpler PAC-Bayesian Bounds for Hostile Datacited by 45
4 more
  1. 2020Generalization Bounds via Information Density and Conditional Information Densitycited by 43
  2. 2021Tighter expected generalization error bounds via Wasserstein distancecited by 43
  3. 2020Jensen-Shannon Information Based Characterization of the Generalization Error of Learning Algorithmscited by 19
  4. 2013Sharp Inequalities for f-divergencescited by 3

Built on by

Cited by year

Citing papers per year: 2021: 1; 2022: 6; 2023: 1; 2024: 3; 2025: 1.

12 papers.

  1. 2022An Information-theoretical Approach to Semi-supervised Learning under Covariate-shiftcited by 18
  2. 2021Characterizing and Understanding the Generalization Error of Transfer Learning with Gibbs Algorithmcited by 14
  3. 2022Chained Generalisation Boundscited by 10
  4. 2024Information-Theoretic Generalization Bounds for Deep Neural Networkscited by 10
  5. 2022f-divergences and their applications in lossy compression and bounding generalization errorcited by 7
  6. 2022Learning Algorithm Generalization Error Bounds via Auxiliary Distributionscited by 6
  7. 2022How Does Pseudo-Labeling Affect the Generalization Error of the Semi-Supervised Gibbs Algorithm?cited by 5
  8. 2024Generalization and Robustness of the Tilted Empirical Riskcited by 4
  9. 2022Formal limitations of sample-wise information-theoretic generalization boundscited by 3
  10. 2023Information-Theoretic Generalization Bounds for Transductive Learning and its Applicationscited by 2
  11. 2024Tighter Learning Guarantees on Digital Computers via Concentration of Measure on Finite Spacescited by 0
  12. 2025An Information-Theoretic Analysis of OOD Generalization in Meta-Reinforcement Learningcited by 0