# An Information-theoretical Approach to Semi-supervised Learning under Covariate-shift

arXiv: https://arxiv.org/abs/2202.12123 · HTML: https://mlsanity.com/paper/2202.12123/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 52 references that are in the map.
- First posted on arXiv 24 Feb 2022.

## Builds on (most cited first, 26 of 26)

- [On Calibration of Modern Neural Networks](https://mlsanity.com/paper/1706.04599/) (2017)
- [Domain-Adversarial Training of Neural Networks](https://mlsanity.com/paper/1505.07818/) (2015)
- [Information-theoretic analysis of generalization capability of learning algorithms](https://mlsanity.com/paper/1705.07809/) (2017)
- [In Defense of Pseudo-Labeling: An Uncertainty-Aware Pseudo-label Selection Framework for Semi-Supervised Learning](https://mlsanity.com/paper/2101.06329/) (2021)
- [Label Propagation for Deep Semi-supervised Learning](https://mlsanity.com/paper/1904.04717/) (2019)
- [A Survey on Deep Semi-supervised Learning](https://mlsanity.com/paper/2103.00550/) (2021)
- [A DIRT-T Approach to Unsupervised Domain Adaptation](https://mlsanity.com/paper/1802.08735/) (2018)
- [Theoretical Analysis of Self-Training with Deep Networks on Unlabeled Data](https://mlsanity.com/paper/2010.03622/) (2020)
- [Reasoning About Generalization via Conditional Mutual Information](https://mlsanity.com/paper/2001.09122/) (2020)
- [An Overview of Deep Semi-Supervised Learning](https://mlsanity.com/paper/2006.05278/) (2020)
- [Tightening Mutual Information Based Bounds on Generalization Error](https://mlsanity.com/paper/1901.04609/) (2019)
- [How much does your data exploration overfit? Controlling bias via information usage](https://mlsanity.com/paper/1511.05219/) (2015)
- [Sharpened Generalization Bounds based on Conditional Mutual Information and an Application to Noisy, Iterative Algorithms](https://mlsanity.com/paper/2004.12983/) (2020)
- [Self-training Avoids Using Spurious Features Under Domain Shift](https://mlsanity.com/paper/2006.10032/) (2020)
- [On Loss Functions for Deep Neural Networks in Classification](https://mlsanity.com/paper/1702.05659/) (2017)
- [Chaining Mutual Information and Tightening Generalization Bounds](https://mlsanity.com/paper/1806.03803/) (2018)
- [Generalization Error Bounds Via R'enyi-, f-Divergences and Maximal Leakage](https://mlsanity.com/paper/1912.01439/) (2019)
- [A Tunable Loss Function for Robust Classification: Calibration, Landscape, and Generalization](https://mlsanity.com/paper/1906.02314/) (2019)
- [Unlabelled Data Improves Bayesian Uncertainty Calibration under Covariate Shift](https://mlsanity.com/paper/2006.14988/) (2020)
- [Jensen-Shannon Information Based Characterization of the Generalization Error of Learning Algorithms](https://mlsanity.com/paper/2010.12664/) (2020)
- [When can unlabeled data improve the learning rate?](https://mlsanity.com/paper/1905.11866/) (2019)
- [Information-Theoretic Bounds on the Moments of the Generalization Error of Learning Algorithms](https://mlsanity.com/paper/2102.02016/) (2021)
- [Information-Theoretic Understanding of Population Risk Improvement with Model Compression](https://mlsanity.com/paper/1901.09421/) (2019)
- [Semi-Supervised Learning: the Case When Unlabeled Data is Equally Useful](https://mlsanity.com/paper/2005.11018/) (2020)
- [Semi-Generative Modelling: Covariate-Shift Adaptation with Cause and Effect Features](https://mlsanity.com/paper/1807.07879/) (2018)
- [Improvability Through Semi-Supervised Learning: A Survey of Theoretical Results](https://mlsanity.com/paper/1908.09574/) (2019)

## Built on by (most cited first, 18 of 18)

- [Information-Theoretic Analysis of Unsupervised Domain Adaptation](https://mlsanity.com/paper/2210.00706/) (2022)
- [On the Generalization for Transfer Learning: An Information-Theoretic Analysis](https://mlsanity.com/paper/2207.05377/) (2022)
- [Don't fear the unlabelled: safe semi-supervised learning via simple debiasing](https://mlsanity.com/paper/2203.07512/) (2022)
- [Approximately Bayes-Optimal Pseudo Label Selection](https://mlsanity.com/paper/2302.08883/) (2023)
- [Learning Algorithm Generalization Error Bounds via Auxiliary Distributions](https://mlsanity.com/paper/2210.00483/) (2022)
- [Semi-Supervised Learning guided by the Generalized Bayes Rule under Soft Revision](https://mlsanity.com/paper/2405.15294/) (2024)
- [How Does Pseudo-Labeling Affect the Generalization Error of the Semi-Supervised Gibbs Algorithm?](https://mlsanity.com/paper/2210.08188/) (2022)
- [Nearest Neighbor Sampling for Covariate Shift Adaptation](https://mlsanity.com/paper/2312.09969/) (2023)
- [Domain adaptation using optimal transport for invariant learning using histopathology datasets](https://mlsanity.com/paper/2303.02241/) (2023)
- [Robust Semi-supervised Learning via f-Divergence and α-R'enyi Divergence](https://mlsanity.com/paper/2405.00454/) (2024)
- [Generalized Semi-Supervised Learning via Self-Supervised Feature Adaptation](https://mlsanity.com/paper/2405.20596/) (2024)
- [Information-Theoretic Generalization Bounds for Transductive Learning and its Applications](https://mlsanity.com/paper/2311.04561/) (2023)
- [The Capabilities and Limitations of Weak-to-Strong Generalization: Generalization and Calibration](https://mlsanity.com/paper/2502.01458/) (2025)
- [Harnessing the Power of Vicinity-Informed Analysis for Classification under Covariate Shift](https://mlsanity.com/paper/2405.16906/) (2024)
- [In all LikelihoodS: How to Reliably Select Pseudo-Labeled Data for Self-Training in Semi-Supervised Learning](https://mlsanity.com/paper/2303.01117/) (2023)
- [Theory-inspired Label Shift Adaptation via Aligned Distribution Mixture](https://mlsanity.com/paper/2411.02047/) (2024)
- [Optimal Exact Recovery in Semi-Supervised Learning: A Study of Spectral Methods and Graph Convolutional Networks](https://mlsanity.com/paper/2412.13754/) (2024)
- [Beyond Parameter Aggregation: Semantic Consensus for Federated Fine-Tuning of LLMs](https://mlsanity.com/paper/2605.11857/) (2026)

## Abstract (arXiv)

A common assumption in semi-supervised learning is that the labeled, unlabeled, and test data are drawn from the same distribution. However, this assumption is not satisfied in many applications. In many scenarios, the data is collected sequentially (e.g., healthcare) and the distribution of the data may change over time often exhibiting so-called covariate shifts. In this paper, we propose an approach for semi-supervised learning algorithms that is capable of addressing this issue. Our framework also recovers some popular methods, including entropy minimization and pseudo-labeling. We provide new information-theoretical based generalization error upper bounds inspired by our novel framework. Our bounds are applicable to both general semi-supervised learning and the covariate-shift scenario. Finally, we show numerically that our method outperforms previous approaches proposed for semi-supervised learning under the covariate shift.