mlsanityData to 7 Oct 2026

Reasoning About Generalization via Conditional Mutual Information

arXiv 2001.09122 ↗First posted 24 Jan 2020cs.LG cs.CR cs.DScited by 145

Not on the map. Why?

Abstract (arXiv)

We provide an information-theoretic framework for studying the generalization properties of machine learning algorithms. Our framework ties together existing approaches, including uniform convergence bounds and recent methods for adaptive data analysis. Specifically, we use Conditional Mutual Information (CMI) to quantify how well the input (i.e., the training data) can be recognized given the output (i.e., the trained model) of the learning algorithm. We show that bounds on CMI can be obtained from VC dimension, compression schemes, differential privacy, and other methods. We then show that bounded CMI implies various forms of generalization.

Key facts. Cited by 145 papers in the map (data through 7 Oct 2026). First posted on arXiv 24 Jan 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 25; 2022: 33; 2023: 26; 2024: 15; 2025: 25; 2026: 21.

145 papers.

Sort:

By area:

  1. 2021User-friendly introduction to PAC-Bayes boundscited by 234
  2. 2023Privacy Auditing with One (1) Training Runcited by 119
  3. 2023How Does Information Bottleneck Help Deep Learning?cited by 92
  4. 2021Invariant Information Bottleneck for Domain Generalizationcited by 86
  5. 2021Information-Theoretic Generalization Bounds for Stochastic Gradient Descentcited by 84
  6. 2023To Compress or Not to Compress- Self-Supervised Learning and Information Theory: A Reviewcited by 72
  7. 2025How much do language models memorize?cited by 53
  8. 2021Information-theoretic generalization bounds for black-box learning algorithmscited by 49
  9. 2021Generalization Bounds For Meta-Learning: An Information-Theoretic Analysiscited by 45
  10. 2021Tighter expected generalization error bounds via Wasserstein distancecited by 43
  11. 2023Stability is Stable: Connections between Replicability, Privacy, and Adaptive Generalizationcited by 42
  12. 2021PAC-Bayes, MAC-Bayes and Conditional Mutual Information: Fast rate bounds that handle general VC classescited by 31
38 more
  1. 2021On the Generalization of Models Trained with SGD: Information-Theoretic Bounds and Implicationscited by 31
  2. 2021Towards a Unified Information-Theoretic Framework for Generalizationcited by 29
  3. 2022A New Family of Generalization Bounds Using Samplewise Evaluated CMIcited by 28
  4. 2022Generalization Bounds via Convex Analysiscited by 27
  5. 2022Single Trajectory Nonparametric Learning of Nonlinear Dynamicscited by 26
  6. 2023An Information-Theoretic Perspective on Variance-Invariance-Covariance Regularizationcited by 25
  7. 2024Knowledge Overshadowing Causes Amalgamated Hallucination in Large Language Modelscited by 24
  8. 2022Rate-Distortion Theoretic Generalization Bounds for Stochastic Learning Algorithmscited by 22
  9. 2024Mutual Information Estimation via Normalizing Flowscited by 22
  10. 2023Tighter Information-Theoretic Generalization Bounds from Supersamplescited by 21
  11. 2023Information Bottleneck Analysis of Deep Neural Networks via Lossy Compressioncited by 19
  12. 2022An Information-theoretical Approach to Semi-supervised Learning under Covariate-shiftcited by 18
  13. 2022Limitations of Information-Theoretic Generalization Bounds for Gradient Descent Methods in Stochastic Convex Optimizationcited by 18
  14. 2023More PAC-Bayes bounds: From bounded losses, to losses with general tail behaviors, to anytime validitycited by 17
  15. 2025Does Generation Require Memorization? Creative Diffusion Models using Ambient Diffusioncited by 17
  16. 2022Understanding Generalization via Leave-One-Out Conditional Mutual Informationcited by 16
  17. 2023Information Theoretic Lower Bounds for Information Theoretic Upper Boundscited by 16
  18. 2021Generalization Bounds for Noisy Iterative Algorithms Using Properties of Additive Noise Channelscited by 15
  19. 2022What Do We Maximize in Self-Supervised Learning?cited by 15
  20. 2022Information-Theoretic Analysis of Unsupervised Domain Adaptationcited by 15
  21. 2023Generalization Bounds with Data-dependent Fractal Dimensionscited by 15
  22. 2021Information Complexity and Generalization Boundscited by 14
  23. 2021Characterizing and Understanding the Generalization Error of Transfer Learning with Gibbs Algorithmcited by 14
  24. 2022Rate-Distortion Theoretic Bounds on Generalization Error for Distributed Learningcited by 14
  25. 2022Generalization Error Bounds for Deep Neural Networks Trained by SGDcited by 14
  26. 2023Online-to-PAC Conversions: Generalization Bounds via Regret Analysiscited by 14
  27. 2025A Causal Adjustment Module for Debiasing Scene Graph Generationcited by 14
  28. 2021Information-Theoretic Bounds on the Moments of the Generalization Error of Learning Algorithmscited by 12
  29. 2024Information Complexity of Stochastic Convex Optimization: Applications to Generalization and Memorizationcited by 12
  30. 2021Information-Theoretic Characterization of the Generalization Error for Iterative Semi-Supervised Learningcited by 11
  31. 2022Tighter Expected Generalization Error Bounds via Convexity of Information Measurescited by 11
  32. 2022On the Generalization for Transfer Learning: An Information-Theoretic Analysiscited by 11
  33. 2022Evaluated CMI Bounds for Meta Learning: Tightness and Expressivenesscited by 11
  34. 2022Two Facets of SDE Under an Information-Theoretic Lens: Generalization of SGD via Training Trajectories and via Terminal Statescited by 11
  35. 2024Minimum Description Length and Generalization Guarantees for Representation Learningcited by 11
  36. 2021An Information-Theoretic Analysis of the Impact of Task Similarity on Meta-Learningcited by 10
  37. 2021Rate-Distortion Analysis of Minimum Excess Risk in Bayesian Learningcited by 10
  38. 2021Information-Theoretic Analysis of Epistemic Uncertainty in Bayesian Meta-learningcited by 10

50 most cited of 145 All 145 (JSON)