mlsanityData to 7 Oct 2026

An Information-theoretical Approach to Semi-supervised Learning under Covariate-shift

arXiv 2202.12123 ↗First posted 24 Feb 2022cs.IT math.IT stat.MLcited by 18

Not on the map. Why?

Abstract (arXiv)

A common assumption in semi-supervised learning is that the labeled, unlabeled, and test data are drawn from the same distribution. However, this assumption is not satisfied in many applications. In many scenarios, the data is collected sequentially (e.g., healthcare) and the distribution of the data may change over time often exhibiting so-called covariate shifts. In this paper, we propose an approach for semi-supervised learning algorithms that is capable of addressing this issue. Our framework also recovers some popular methods, including entropy minimization and pseudo-labeling. We provide new information-theoretical based generalization error upper bounds inspired by our novel framework. Our bounds are applicable to both general semi-supervised learning and the covariate-shift scenario. Finally, we show numerically that our method outperforms previous approaches proposed for semi-supervised learning under the covariate shift.

Key facts. Cited by 18 papers in the map (data through 7 Oct 2026). Builds on 26 of its 52 references that are in the map. First posted on arXiv 24 Feb 2022.

Builds on

26 of 52 references.

References · 26

  1. 2017On Calibration of Modern Neural Networkscited by 4,860
  2. 2015Domain-Adversarial Training of Neural Networkscited by 3,023
  3. 2021In Defense of Pseudo-Labeling: An Uncertainty-Aware Pseudo-label Selection Framework for Semi-Supervised Learningcited by 301
  4. 2017Information-theoretic analysis of generalization capability of learning algorithmscited by 299
  5. 2019Label Propagation for Deep Semi-supervised Learningcited by 287
  6. 2021A Survey on Deep Semi-supervised Learningcited by 250
  7. 2018A DIRT-T Approach to Unsupervised Domain Adaptationcited by 215
  8. 2020Theoretical Analysis of Self-Training with Deep Networks on Unlabeled Datacited by 179
  9. 2020Reasoning About Generalization via Conditional Mutual Informationcited by 145
  10. 2020An Overview of Deep Semi-Supervised Learningcited by 121
  11. 2019Tightening Mutual Information Based Bounds on Generalization Errorcited by 117
  12. 2015How much does your data exploration overfit? Controlling bias via information usagecited by 95
14 more
  1. 2020Sharpened Generalization Bounds based on Conditional Mutual Information and an Application to Noisy, Iterative Algorithmscited by 89
  2. 2020Self-training Avoids Using Spurious Features Under Domain Shiftcited by 69
  3. 2017On Loss Functions for Deep Neural Networks in Classificationcited by 68
  4. 2018Chaining Mutual Information and Tightening Generalization Boundscited by 65
  5. 2019Generalization Error Bounds Via R'enyi-, f-Divergences and Maximal Leakagecited by 48
  6. 2019A Tunable Loss Function for Robust Classification: Calibration, Landscape, and Generalizationcited by 23
  7. 2020Unlabelled Data Improves Bayesian Uncertainty Calibration under Covariate Shiftcited by 19
  8. 2020Jensen-Shannon Information Based Characterization of the Generalization Error of Learning Algorithmscited by 19
  9. 2019When can unlabeled data improve the learning rate?cited by 14
  10. 2021Information-Theoretic Bounds on the Moments of the Generalization Error of Learning Algorithmscited by 12
  11. 2019Information-Theoretic Understanding of Population Risk Improvement with Model Compressioncited by 11
  12. 2020Semi-Supervised Learning: the Case When Unlabeled Data is Equally Usefulcited by 9
  13. 2018Semi-Generative Modelling: Covariate-Shift Adaptation with Cause and Effect Featurescited by 8
  14. 2019Improvability Through Semi-Supervised Learning: A Survey of Theoretical Resultscited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 5; 2023: 5; 2024: 6; 2025: 1; 2026: 1.

18 papers.

Sort:

  1. 2022Information-Theoretic Analysis of Unsupervised Domain Adaptationcited by 15
  2. 2022On the Generalization for Transfer Learning: An Information-Theoretic Analysiscited by 11
  3. 2022Don't fear the unlabelled: safe semi-supervised learning via simple debiasingcited by 10
  4. 2023Approximately Bayes-Optimal Pseudo Label Selectioncited by 10
  5. 2022Learning Algorithm Generalization Error Bounds via Auxiliary Distributionscited by 6
  6. 2024Semi-Supervised Learning guided by the Generalized Bayes Rule under Soft Revisioncited by 6
  7. 2022How Does Pseudo-Labeling Affect the Generalization Error of the Semi-Supervised Gibbs Algorithm?cited by 5
  8. 2023Nearest Neighbor Sampling for Covariate Shift Adaptationcited by 5
  9. 2023Domain adaptation using optimal transport for invariant learning using histopathology datasetscited by 4
  10. 2024Robust Semi-supervised Learning via f-Divergence and α-R'enyi Divergencecited by 3
  11. 2024Generalized Semi-Supervised Learning via Self-Supervised Feature Adaptationcited by 3
  12. 2023Information-Theoretic Generalization Bounds for Transductive Learning and its Applicationscited by 2
6 more
  1. 2025The Capabilities and Limitations of Weak-to-Strong Generalization: Generalization and Calibrationcited by 2
  2. 2024Harnessing the Power of Vicinity-Informed Analysis for Classification under Covariate Shiftcited by 1
  3. 2023In all LikelihoodS: How to Reliably Select Pseudo-Labeled Data for Self-Training in Semi-Supervised Learningcited by 0
  4. 2024Theory-inspired Label Shift Adaptation via Aligned Distribution Mixturecited by 0
  5. 2024Optimal Exact Recovery in Semi-Supervised Learning: A Study of Spectral Methods and Graph Convolutional Networkscited by 0
  6. 2026Beyond Parameter Aggregation: Semantic Consensus for Federated Fine-Tuning of LLMscited by 0