mlsanityData to 14 Oct 2026

Erasing the Bias: Fine-Tuning Foundation Models for Semi-Supervised Learning

arXiv 2405.11756 ↗First posted 20 May 2024cs.LGcited by 19

Not on the map. Why?

Abstract (arXiv)

Semi-supervised learning (SSL) has witnessed remarkable progress, resulting in the emergence of numerous method variations. However, practitioners often encounter challenges when attempting to deploy these methods due to their subpar performance. In this paper, we present a novel SSL approach named FineSSL that significantly addresses this limitation by adapting pre-trained foundation models. We identify the aggregated biases and cognitive deviation problems inherent in foundation models, and propose a simple yet effective solution by imposing balanced margin softmax and decoupled label smoothing. Through extensive experiments, we demonstrate that FineSSL sets a new state of the art for SSL on multiple benchmark datasets, reduces the training cost by over six times, and can seamlessly integrate various fine-tuning and modern SSL algorithms. The source code is available at https://github.com/Gank0078/FineSSL.

Key facts. Cited by 19 papers in the map (data through 14 Oct 2026). Builds on 52 of its 60 references that are in the map. First posted on arXiv 20 May 2024.

Builds on

52 of 60 references.

References · 52

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  5. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  6. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,513
  7. 2017On Calibration of Modern Neural Networkscited by 4,960
  8. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,236
  9. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,864
  10. 2016Wide Residual Networkscited by 3,070
  11. 2021Learning to Prompt for Vision-Language Modelscited by 2,281
  12. 2020FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidencecited by 2,085
38 more
  1. 2022Visual Prompt Tuningcited by 1,681
  2. 2019MixMatch: A Holistic Approach to Semi-Supervised Learningcited by 1,391
  3. 2020Big Self-Supervised Models are Strong Semi-Supervised Learnerscited by 1,380
  4. 2019Self-training with Noisy Student improves ImageNet classificationcited by 1,311
  5. 2021BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-modelscited by 1,245
  6. 2016Temporal Ensembling for Semi-Supervised Learningcited by 1,057
  7. 2019Learning Imbalanced Datasets with Label-Distribution-Aware Margin Losscited by 989
  8. 2019When Does Label Smoothing Help?cited by 988
  9. 2022Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learningcited by 896
  10. 2022Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distributioncited by 739
  11. 2022AdaptFormer: Adapting Vision Transformers for Scalable Visual Recognitioncited by 719
  12. 2020Long-tail learning via logit adjustmentcited by 618
  13. 2021FlexMatch: Boosting Semi-Supervised Learning with Curriculum Pseudo Labelingcited by 540
  14. 2019Pseudo-Labeling and Confirmation Bias in Deep Semi-Supervised Learningcited by 510
  15. 2016Regularization With Stochastic Transformations and Perturbations for Deep Semi-Supervised Learningcited by 441
  16. 2019On Mixup Training: Improved Calibration and Predictive Uncertainty for Deep Neural Networkscited by 305
  17. 2021In Defense of Pseudo-Labeling: An Uncertainty-Aware Pseudo-label Selection Framework for Semi-Supervised Learningcited by 301
  18. 2022Scaling & Shifting Your Features: A New Baseline for Efficient Model Tuningcited by 294
  19. 2014Learning with Pseudo-Ensemblescited by 241
  20. 2022FreeMatch: Self-adaptive Thresholding for Semi-supervised Learningcited by 232
  21. 2021Improving Calibration for Long-Tailed Recognitioncited by 212
  22. 2021Open-World Semi-Supervised Learningcited by 165
  23. 2021Dash: Semi-Supervised Learning with Dynamic Thresholdingcited by 148
  24. 2023SoftMatch: Addressing the Quantity-Quality Trade-off in Semi-supervised Learningcited by 133
  25. 2020Distribution Aligning Refinery of Pseudo-label for Imbalanced Semi-supervised Learningcited by 105
  26. 2022USB: A Unified Semi-supervised Learning Benchmark for Classificationcited by 95
  27. 2022Debiased Learning from Naturally Imbalanced Pseudo-Labelscited by 85
  28. 2022Revisiting Parameter-Efficient Tuning: Are We Really There Yet?cited by 76
  29. 2022Class-Aware Contrastive Semi-Supervised Learningcited by 62
  30. 2023Long-Tail Learning with Foundation Model: Heavy Fine-Tuning Hurtscited by 58
  31. 2021Self-Tuning for Data-Efficient Deep Learningcited by 39
  32. 2021A Realistic Evaluation of Semi-Supervised Learning for Fine-Grained Classificationcited by 36
  33. 2023Generalized Logit Adjustment: Calibrating Fine-tuned Models by Removing Label Bias in Foundation Modelscited by 32
  34. 2024Facing the Elephant in the Room: Visual Prompt Tuning or Full Finetuning?cited by 31
  35. 2023Enhancing CLIP with CLIP: Exploring Pseudolabeling for Limited-Label Prompt Tuningcited by 30
  36. 2021Semi-Supervised Vision Transformerscited by 24
  37. 2022Contrastive Regularization for Semi-Supervised Learningcited by 24
  38. 2020Unlabelled Data Improves Bayesian Uncertainty Calibration under Covariate Shiftcited by 21

50 most cited of 52 All 52 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 4; 2025: 10; 2026: 5.

19 papers.

Sort:

By area:

  1. 2024UniMatch V2: Pushing the Limit of Semi-Supervised Semantic Segmentationcited by 54
  2. 2024Lessons and Insights from a Unifying Study of Parameter-Efficient Fine-Tuning (PEFT) in Visual Recognitioncited by 16
  3. 2025LADA: Scalable Label-Specific CLIP Adapter for Continual Learningcited by 11
  4. 2024Continuous Contrastive Learning for Long-Tailed Semi-Supervised Recognitioncited by 10
  5. 2025Revisiting semi-supervised learning in the era of foundation modelscited by 6
  6. 2025Foundation Models for Scientific Discovery: From Paradigm Enhancement to Paradigm Transitioncited by 6
  7. 2025FairTune: A Bias-Aware Fine-Tuning Framework Towards Fair Heart Rate Prediction from PPGcited by 2
  8. 2026Triple Expert Learning from Noisy Labels for Semi-Supervised Vision Foundation Model Adaptationcited by 2
  9. 2024Decouple then Converge: Handling Unknown Unlabeled Distributions in Long-Tailed Semi-Supervised Learningcited by 1
  10. 2025Enhancing Semi-supervised Learning with Zero-shot Pseudolabelscited by 1
  11. 2025Towards Micro-Action Recognition with Limited Annotations: An Asynchronous Pseudo Labeling and Training Approachcited by 1
  12. 2026SCOPE: Structured Prototype-Guided Adaptation for EEG Foundation Models with Limited Labelscited by 1
7 more
  1. 2026Sparsity Hurts: Simple Linear Adapter Can Boost Generalized Category Discoverycited by 1
  2. 2025ULFine: Unbiased Lightweight Fine-tuning for Foundation-Model-Assisted Long-Tailed Semi-Supervised Learningcited by 0
  3. 2025CAST: Contrastive Adaptation and Distillation for Semi-Supervised Instance Segmentationcited by 0
  4. 2025Bi-CoG: Bi-Consistency-Guided Self-Training for Vision-Language Modelscited by 0
  5. 2025Solving Semi-Supervised Few-Shot Learning from an Auto-Annotation Perspectivecited by 0
  6. 2026Training a Student Expert via Semi-Supervised Foundation Model Distillationcited by 0
  7. 2026SHED: Style-Homogenized Embedding Alignment for Domain Generalizationcited by 0