mlsanityData to 14 Oct 2026

Information-Theoretic Characterization of the Generalization Error for Iterative Semi-Supervised Learning

arXiv 2110.00926 ↗First posted 3 Oct 2021cs.LG cs.IT math.ITcited by 11

Not on the map. Why?

Abstract (arXiv)

Using information-theoretic principles, we consider the generalization error (gen-error) of iterative semi-supervised learning (SSL) algorithms that iteratively generate pseudo-labels for a large amount of unlabelled data to progressively refine the model parameters. In contrast to most previous works that em bound the gen-error, we provide an em exact expression for the gen-error and particularize it to the binary Gaussian mixture model. Our theoretical results suggest that when the class conditional variances are not too large, the gen-error decreases with the number of iterations, but quickly saturates. On the flip side, if the class conditional variances (and so amount of overlap between the classes) are large, the gen-error increases with the number of iterations. To mitigate this undesirable effect, we show that regularization can reduce the gen-error. The theoretical results are corroborated by extensive experiments on the MNIST and CIFAR datasets in which we notice that for easy-to-distinguish classes, the gen-error improves after several pseudo-labelling iterations, but saturates afterwards, and for more difficult-to-distinguish classes, regularization improves the generalization performance.

Key facts. Cited by 11 papers in the map (data through 14 Oct 2026). Builds on 18 of its 49 references that are in the map. First posted on arXiv 3 Oct 2021.

Builds on

18 of 49 references.

References · 18

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2019Learning Imbalanced Datasets with Label-Distribution-Aware Margin Losscited by 989
  3. 2019Unlabeled Data Improves Adversarial Robustnesscited by 416
  4. 2017Information-theoretic analysis of generalization capability of learning algorithmscited by 303
  5. 2020Reasoning About Generalization via Conditional Mutual Informationcited by 147
  6. 2017Generalization in Deep Learningcited by 146
  7. 2019Information-Theoretic Generalization Bounds for SGLD via Data-Dependent Estimatescited by 119
  8. 2019Tightening Mutual Information Based Bounds on Generalization Errorcited by 117
  9. 2020Classification vs regression in overparameterized regimes: Does the loss function matter?cited by 97
  10. 2020Sharpened Generalization Bounds based on Conditional Mutual Information and an Application to Noisy, Iterative Algorithmscited by 89
  11. 2018Generalization Error Bounds for Noisy, Iterative Algorithmscited by 82
  12. 2020The role of regularization in classification of high-dimensional noisy Gaussian mixturecited by 63
6 more
  1. 2019Generalization Error Bounds Via R'enyi-, f-Divergences and Maximal Leakagecited by 48
  2. 2020Binary Classification of Gaussian Mixtures: Abundance of Support Vectors, Benign Overfitting and Regularizationcited by 27
  3. 2020Information-theoretic analysis for transfer learningcited by 23
  4. 2018Unsupervised Learning using Pretrained CNN and Associative Memory Bankcited by 12
  5. 2020Information-Theoretic Bounds on Transfer Generalization Gap Based on Jensen-Shannon Divergencecited by 11
  6. 2020Semi-Supervised Learning: the Case When Unlabeled Data is Equally Usefulcited by 9

Built on by

Cited by year

Citing papers per year: 2022: 2; 2023: 3; 2024: 5; 2025: 1.

11 papers.

  1. 2022Semi-supervised Vision Transformers at Scalecited by 49
  2. 2023Toward Understanding Generative Data Augmentationcited by 25
  3. 2023Tighter Information-Theoretic Generalization Bounds from Supersamplescited by 22
  4. 2022How Does Pseudo-Labeling Affect the Generalization Error of the Semi-Supervised Gibbs Algorithm?cited by 5
  5. 2024On Mesa-Optimization in Autoregressively Trained Transformers: Emergence and Capabilitycited by 5
  6. 2024Towards Understanding Why FixMatch Generalizes Better Than Supervised Learningcited by 5
  7. 2024Robust Semi-supervised Learning via f-Divergence and α-R'enyi Divergencecited by 3
  8. 2023Information-Theoretic Generalization Bounds for Transductive Learning and its Applicationscited by 2
  9. 2025A Theory for Conditional Generative Modeling on Multiple Data Sourcescited by 1
  10. 2024Error Bounds of Supervised Classification from Information-Theoretic Perspectivecited by 0
  11. 2024Optimal Exact Recovery in Semi-Supervised Learning: A Study of Spectral Methods and Graph Convolutional Networkscited by 0