# Leveraging Ensemble Diversity for Robust Self-Training in the Presence of Sample Selection Bias

arXiv: https://arxiv.org/abs/2310.14814 · HTML: https://mlsanity.com/paper/2310.14814/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 74 references that are in the map.
- First posted on arXiv 23 Oct 2023.

## Builds on (most cited first, 18 of 18)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Convex optimization](https://mlsanity.com/paper/2106.01946/) (2021)
- [FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidence](https://mlsanity.com/paper/2001.07685/) (2020)
- [Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hypersphere](https://mlsanity.com/paper/2005.10242/) (2020)
- [FlexMatch: Boosting Semi-Supervised Learning with Curriculum Pseudo Labeling](https://mlsanity.com/paper/2110.08263/) (2021)
- [Mitigating Neural Network Overconfidence with Logit Normalization](https://mlsanity.com/paper/2205.09310/) (2022)
- [Assessing Generalization of SGD via Disagreement](https://mlsanity.com/paper/2106.13799/) (2021)
- [Self-Training: A Survey](https://mlsanity.com/paper/2202.12040/) (2022)
- [Debiased Self-Training for Semi-Supervised Learning](https://mlsanity.com/paper/2202.07136/) (2022)
- [Detecting Errors and Estimating Accuracy on Unlabeled Data with Self-training Ensembles](https://mlsanity.com/paper/2106.15728/) (2021)
- [Diversity and Generalization in Neural Network Ensembles](https://mlsanity.com/paper/2110.13786/) (2021)
- [Generalized Negative Correlation Learning for Deep Ensembling](https://mlsanity.com/paper/2011.02952/) (2020)
- [Partly Supervised Multitask Learning](https://mlsanity.com/paper/2005.02523/) (2020)
- [Random Matrix Analysis to Balance between Supervised and Unsupervised Learning under the Low Density Separation Assumption](https://mlsanity.com/paper/2310.13434/) (2023)
- [Multi-class Probabilistic Bounds for Self-learning](https://mlsanity.com/paper/2109.14422/) (2021)

## Built on by (most cited first, 7 of 7)

- [Transcendence: Generative Models Can Outperform The Experts That Train Them](https://mlsanity.com/paper/2406.11741/) (2024)
- [MANO: Exploiting Matrix Norm for Unsupervised Accuracy Estimation Under Distribution Shifts](https://mlsanity.com/paper/2405.18979/) (2024)
- [A Mechanistic Study of Transformers Training Dynamics](https://mlsanity.com/paper/2410.24050/) (2024)
- [Single-Teacher View Augmentation: Boosting Knowledge Distillation via Angular Diversity](https://mlsanity.com/paper/2510.22480/) (2025)
- [Calibrated Adaptive Teacher for Domain Adaptive Intelligent Fault Diagnosis](https://mlsanity.com/paper/2312.02826/) (2023)
- [Easing Optimization Paths: a Circuit Perspective](https://mlsanity.com/paper/2501.02362/) (2025)
- [Angular Regularization for Positive-Unlabeled Learning on the Hypersphere](https://mlsanity.com/paper/2512.06785/) (2025)

## Abstract (arXiv)

Self-training is a well-known approach for semi-supervised learning. It consists of iteratively assigning pseudo-labels to unlabeled data for which the model is confident and treating them as labeled examples. For neural networks, softmax prediction probabilities are often used as a confidence measure, although they are known to be overconfident, even for wrong predictions. This phenomenon is particularly intensified in the presence of sample selection bias, i.e., when data labeling is subject to some constraint. To address this issue, we propose a novel confidence measure, called T-similarity, built upon the prediction diversity of an ensemble of linear classifiers. We provide the theoretical analysis of our approach by studying stationary points and describing the relationship between the diversity of the individual members and their performance. We empirically demonstrate the benefit of our confidence measure for three different pseudo-labeling policies on classification datasets of various data modalities. The code is available at https://github.com/ambroiseodt/tsim.