Data to 14 Oct 2026

Theoretical Analysis of Self-Training with Deep Networks on Unlabeled Data

arXiv 2010.03622 ↗First posted 7 Oct 2020cs.LG stat.MLcited by 180

Not on the map. Why?

Abstract (arXiv)

Self-training algorithms, which train a model to fit pseudolabels predicted by another previously-learned model, have been very successful for learning with unlabeled data using neural networks. However, the current theoretical understanding of self-training only applies to linear models. This work provides a unified theoretical analysis of self-training with deep networks for semi-supervised learning, unsupervised domain adaptation, and unsupervised learning. At the core of our analysis is a simple but realistic "expansion" assumption, which states that a low probability subset of the data must expand to a neighborhood with large probability relative to the subset. We also assume that neighborhoods of examples in different classes have minimal overlap. We prove that under these assumptions, the minimizers of population objectives based on self-training and input-consistency regularization will achieve high accuracy with respect to ground-truth labels. By using off-the-shelf generalization bounds, we immediately convert this result to sample complexity guarantees for neural nets that are polynomial in the margin and Lipschitzness. Our results help explain the empirical successes of recently proposed self-training algorithms which use input consistency regularization.

Key facts. Cited by 180 papers in the map (data through 14 Oct 2026). First posted on arXiv 7 Oct 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 41; 2022: 51; 2023: 35; 2024: 23; 2025: 16; 2026: 14.

180 papers.

Sort:

By area:

2023Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervisioncited by 467
2021Provable Guarantees for Self-Supervised Deep Learning with Spectral Contrastive Losscited by 313
2023A Survey on Self-supervised Learning: Algorithms, Applications, and Future Trendscited by 207
2021Want To Reduce Labeling Cost? GPT-3 Can Helpcited by 196
2021Self-Supervised Learning of Graph Neural Networks: A Unified Reviewcited by 184
2021Cycle Self-Training for Domain Adaptationcited by 120
2021Toward Understanding the Feature Learning Process of Self-supervised Contrastive Learningcited by 119
2021Towards the Generalization of Contrastive Self-Supervised Learningcited by 99
2022Deep Unsupervised Domain Adaptation: A Review of Recent Advances and Perspectivescited by 98
2022Self-Training: A Surveycited by 86
2021Why Do Pretrained Language Models Help in Downstream Tasks? An Analysis of Head and Prompt Tuningcited by 82
2022Connect, Not Collapse: Explaining Contrastive Learning for Unsupervised Domain Adaptationcited by 67

38 more

2022Debiased Self-Training for Semi-Supervised Learningcited by 62
2021SITA: Single Image Test-time Adaptationcited by 59
2022Divide and Contrast: Source-free Domain Adaptation via Adaptive Contrastive Learningcited by 53
2021A Theory of Label Propagation for Subpopulation Shiftcited by 48
2024Theoretical Analysis of Weak-to-Strong Generalizationcited by 48
2021Understanding Negative Samples in Instance Discriminative Self-supervised Representation Learningcited by 47
2021SemiFL: Semi-Supervised Federated Learning for Unlabeled Clients with Alternate Trainingcited by 45
2022Orchestra: Unsupervised Federated Learning via Globally Consistent Clusteringcited by 45
2022Beyond Separability: Analyzing the Linear Transferability of Contrastive Representations to Related Subpopulationscited by 44
2021Generate, Annotate, and Learn: NLP with Synthetic Textcited by 42
2022Structural Pruning via Latency-Saliency Knapsackcited by 41
2024Quantifying the Gain in Weak-to-Strong Generalizationcited by 41
2021Self-Tuning for Data-Efficient Deep Learningcited by 39
2023InfoNCE Loss Provably Learns Cluster-Preserving Representationscited by 37
2021If your data distribution shifts, use self-learningcited by 36
2022Understanding and Improving the Role of Projection Head in Self-Supervised Learningcited by 36
2023On the Stepwise Nature of Self-Supervised Learningcited by 36
2023Semi-Supervised Semantic Segmentation via Gentle Teaching Assistantcited by 34
2022LOT: Layer-wise Orthogonal Training on Improving ℓ2 Certified Robustnesscited by 33
2022Refign: Align and Refine for Adaptation of Semantic Segmentation to Adverse Conditionscited by 31
2022Sample Efficiency of Data Augmentation Consistency Regularizationcited by 28
2022Your Contrastive Learning Is Secretly Doing Stochastic Neighbor Embeddingcited by 28
2021Self-supervised Geometric Perceptioncited by 26
2021Multimodal Knowledge Expansioncited by 26
2022Gradual Domain Adaptation without Indexed Intermediate Domainscited by 26
2021On the Surrogate Gap between Contrastive and Supervised Lossescited by 25
2022Co-training Improves Prompt-based Learning for Large Language Modelscited by 24
2021Revisiting Self-Training for Few-Shot Learning of Language Modelcited by 23
2024Uncertainty in Graph Neural Networks: A Surveycited by 23
2022Self-Supervised Representation Learning via Latent Graph Predictioncited by 22
2022MutexMatch: Semi-Supervised Learning with Mutex-Based Consistency Regularizationcited by 22
2022Multiple Instance Learning via Iterative Self-Paced Supervised Contrastive Learningcited by 22
2024High-dimensional Analysis of Knowledge Distillation: Weak-to-Strong Generalization and Scaling Lawscited by 22
2022Transferring Fairness under Distribution Shifts via Fair Consistency Regularizationcited by 21
2022First Steps Toward Understanding the Extrapolation of Nonlinear Models to Unseen Domainscited by 21
2024Weak-to-Strong Generalization Through the Data-Centric Lenscited by 21
2022ACT: Semi-supervised Domain-adaptive Medical Image Segmentation with Asymmetric Co-trainingcited by 20
2023Robust Data Pruning under Label Noise via Maximizing Re-labeling Accuracycited by 20

50 most cited of 180 All 180 (JSON)