mlsanityData to 7 Oct 2026

On the Importance of Calibration in Semi-supervised Learning

arXiv 2210.04783 ↗First posted 10 Oct 2022cs.LG cs.CV physics.app-phcited by 6

Not on the map. Why?

Abstract (arXiv)

State-of-the-art (SOTA) semi-supervised learning (SSL) methods have been highly successful in leveraging a mix of labeled and unlabeled data by combining techniques of consistency regularization and pseudo-labeling. During pseudo-labeling, the model's predictions on unlabeled data are used for training and thus, model calibration is important in mitigating confirmation bias. Yet, many SOTA methods are optimized for model performance, with little focus directed to improve model calibration. In this work, we empirically demonstrate that model calibration is strongly correlated with model performance and propose to improve calibration via approximate Bayesian techniques. We introduce a family of new SSL models that optimizes for calibration and demonstrate their effectiveness across standard vision benchmarks of CIFAR-10, CIFAR-100 and ImageNet, giving up to 15.9% improvement in test accuracy. Furthermore, we also demonstrate their effectiveness in additional realistic and challenging problems, such as class-imbalanced datasets and in photonics science.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Builds on 35 of its 56 references that are in the map. First posted on arXiv 10 Oct 2022.

Builds on

35 of 56 references.

References · 35

  1. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  2. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,155
  3. 2017On Calibration of Modern Neural Networkscited by 4,860
  4. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,712
  5. 2015Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learningcited by 4,416
  6. 2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,888
  7. 2019RandAugment: Practical automated data augmentation with a reduced search spacecited by 2,261
  8. 2020FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidencecited by 2,080
  9. 2016Concrete Problems in AI Safetycited by 1,582
  10. 2019MixMatch: A Holistic Approach to Semi-Supervised Learningcited by 1,390
  11. 2019Self-training with Noisy Student improves ImageNet classificationcited by 1,304
  12. 2018Averaging Weights Leads to Wider Optima and Better Generalizationcited by 1,194
23 more
  1. 2019Learning Imbalanced Datasets with Label-Distribution-Aware Margin Losscited by 987
  2. 2019When Does Label Smoothing Help?cited by 985
  3. 2018Loss Surfaces, Mode Connectivity, and Fast Ensembling of DNNscited by 585
  4. 2015Weight Uncertainty in Neural Networkscited by 554
  5. 2021FlexMatch: Boosting Semi-Supervised Learning with Curriculum Pseudo Labelingcited by 537
  6. 2019Pseudo-Labeling and Confirmation Bias in Deep Semi-Supervised Learningcited by 507
  7. 2015Variational Dropout and the Local Reparameterization Trickcited by 489
  8. 2020Bayesian Deep Learning and a Probabilistic Perspective of Generalizationcited by 480
  9. 2017Mean teachers are better role models: Weight-averaged consistency targets improve semi-supervised deep learning resultscited by 470
  10. 2019Measuring Calibration in Deep Learningcited by 443
  11. 2016Regularization With Stochastic Transformations and Perturbations for Deep Semi-Supervised Learningcited by 441
  12. 2021Revisiting the Calibration of Modern Neural Networkscited by 387
  13. 2020BatchEnsemble: An Alternative Approach to Efficient Ensemble and Lifelong Learningcited by 368
  14. 2019ReMixMatch: Semi-Supervised Learning with Distribution Alignment and Augmentation Anchoringcited by 349
  15. 2019On Mixup Training: Improved Calibration and Predictive Uncertainty for Deep Neural Networkscited by 304
  16. 2021In Defense of Pseudo-Labeling: An Uncertainty-Aware Pseudo-label Selection Framework for Semi-Supervised Learningcited by 301
  17. 2014Learning with Pseudo-Ensemblescited by 241
  18. 2021Dash: Semi-Supervised Learning with Dynamic Thresholdingcited by 147
  19. 2021Semi-Supervised Learning of Visual Features by Non-Parametrically Predicting View Assignments with Support Samplescited by 114
  20. 2016PAC-Bayesian Theory Meets Bayesian Inferencecited by 87
  21. 2020Not All Unlabeled Data are Equal: Learning to Weight Data in Semi-supervised Learningcited by 53
  22. 2020Curriculum Labeling: Revisiting Pseudo-Labeling for Semi-Supervised Learningcited by 14
  23. 2021Surrogate- and invariance-boosted contrastive learning for data-scarce applications in sciencecited by 6

Built on by

Cited by year

Citing papers per year: 2023: 2; 2024: 3; 2025: 1.

6 papers.

  1. 2023RoPAWS: Robust Semi-supervised Representation Learning from Uncurated Datacited by 10
  2. 2024Continuous Contrastive Learning for Long-Tailed Semi-Supervised Recognitioncited by 10
  3. 2023Flexible Distribution Alignment: Towards Long-tailed Semi-supervised Learning with Proper Calibrationcited by 7
  4. 2024EOL: Transductive Few-Shot Open-Set Recognition by Enhancing Outlier Logitscited by 2
  5. 2024Imbalanced Semi-Supervised Learning via Label Refinement and Threshold Adjustmentcited by 0
  6. 2025SpikeMatch: Semi-Supervised Learning with Temporal Dynamics of Spiking Neural Networkscited by 0