mlsanityData to 7 Oct 2026

Rethinking Semi-supervised Learning with Language Models

arXiv 2305.13002 ↗First posted 22 May 2023cs.CL cs.AI cs.LGcited by 17

Not on the map. Why?

Abstract (arXiv)

Semi-supervised learning (SSL) is a popular setting aiming to effectively utilize unlabelled data to improve model performance in downstream natural language processing (NLP) tasks. Currently, there are two popular approaches to make use of unlabelled data: Self-training (ST) and Task-adaptive pre-training (TAPT). ST uses a teacher model to assign pseudo-labels to the unlabelled data, while TAPT continues pre-training on the unlabelled data before fine-tuning. To the best of our knowledge, the effectiveness of TAPT in SSL tasks has not been systematically studied, and no previous work has directly compared TAPT and ST in terms of their ability to utilize the pool of unlabelled data. In this paper, we provide an extensive empirical study comparing five state-of-the-art ST approaches and TAPT across various NLP tasks and data sizes, including in- and out-of-domain settings. Surprisingly, we find that TAPT is a strong and more robust SSL learner, even when using just a few hundred unlabelled samples or in the presence of domain shifts, compared to more sophisticated ST approaches, and tends to bring greater improvements in SSL than in fully-supervised settings. Our further analysis demonstrates the risks of using ST approaches when the size of labelled or unlabelled data is small or when domain shifts exist. We offer a fresh perspective for future SSL research, suggesting the use of unsupervised pre-training objectives over dependency on pseudo labels.

Key facts. Cited by 17 papers in the map (data through 7 Oct 2026). Builds on 51 of its 80 references that are in the map. First posted on arXiv 22 May 2023.

Builds on

51 of 80 references.

References · 51

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  3. 2016Semi-Supervised Classification with Graph Convolutional Networkscited by 11,282
  4. 2020Scaling Laws for Neural Language Modelscited by 6,411
  5. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  6. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  7. 2015Character-level Convolutional Networks for Text Classificationcited by 2,293
  8. 2020FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidencecited by 2,080
  9. 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,813
  10. 2020Don't Stop Pretraining: Adapt Language Models to Domains and Taskscited by 1,548
  11. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,486
  12. 2019SciBERT: A Pretrained Language Model for Scientific Textcited by 1,404
38 more
  1. 2019MixMatch: A Holistic Approach to Semi-Supervised Learningcited by 1,390
  2. 2019Self-training with Noisy Student improves ImageNet classificationcited by 1,304
  3. 2020Making Pre-trained Language Models Better Few-shot Learnerscited by 1,255
  4. 2019Unsupervised Data Augmentation for Consistency Trainingcited by 1,057
  5. 2016Temporal Ensembling for Semi-Supervised Learningcited by 1,054
  6. 2019Publicly Available Clinical BERT Embeddingscited by 995
  7. 2020Exploiting Cloze Questions for Few Shot Text Classification and Natural Language Inferencecited by 934
  8. 2017Virtual Adversarial Training: A Regularization Method for Supervised and Semi-Supervised Learningcited by 918
  9. 2017Maximum Classifier Discrepancy for Unsupervised Domain Adaptationcited by 682
  10. 2021FlexMatch: Boosting Semi-Supervised Learning with Curriculum Pseudo Labelingcited by 537
  11. 2019Pseudo-Labeling and Confirmation Bias in Deep Semi-Supervised Learningcited by 507
  12. 2020Rethinking Pre-training and Self-trainingcited by 352
  13. 2019ReMixMatch: Semi-Supervised Learning with Distribution Alignment and Augmentation Anchoringcited by 349
  14. 2020MixText: Linguistically-Informed Interpolation of Hidden Space for Semi-Supervised Text Classificationcited by 186
  15. 2019Revisiting Self-Training for Neural Sequence Generationcited by 154
  16. 2021Dash: Semi-Supervised Learning with Dynamic Thresholdingcited by 147
  17. 2018A robust self-learning method for fully unsupervised cross-lingual mappings of word embeddingscited by 145
  18. 2020Neural Unsupervised Domain Adaptation in NLP---A Surveycited by 142
  19. 2019Zero-Shot Entity Linking by Reading Entity Descriptionscited by 124
  20. 2021AdaMatch: A Unified Approach to Semi-Supervised Learning and Domain Adaptationcited by 104
  21. 2020Self-training and Pre-training are Complementary for Speech Recognitioncited by 98
  22. 2022USB: A Unified Semi-supervised Learning Benchmark for Classificationcited by 95
  23. 2020DialoGLUE: A Natural Language Understanding Benchmark for Task-Oriented Dialoguecited by 89
  24. 2018Semi-Supervised Sequence Modeling with Cross-View Trainingcited by 75
  25. 2022StepGame: A New Benchmark for Robust Multi-Hop Spatial Reasoning in Textscited by 65
  26. 2022Debiased Self-Training for Semi-Supervised Learningcited by 62
  27. 2019Variational Pretraining for Semi-supervised Text Classificationcited by 41
  28. 2018Strong Baselines for Neural Semi-supervised Learning under Domain Shiftcited by 39
  29. 2021Self-Tuning for Data-Efficient Deep Learningcited by 39
  30. 2020Local Additivity Based Data Augmentation for Semi-supervised NERcited by 31
  31. 2021On the Importance of Effectively Adapting Pretrained Language Models for Active Learningcited by 31
  32. 2022Learning to Execute Actions or Ask Clarification Questionscited by 30
  33. 2022Zero-Shot Text Classification with Self-Trainingcited by 26
  34. 2021Extending CLIP for Category-to-image Retrieval in E-commercecited by 24
  35. 2023Don't Stop Pretraining? Make Prompt-based Fine-tuning Powerful Learnercited by 23
  36. 2019A Good Sample is Hard to Find: Noise Injection Sampling and Self-Training for Neural Language Generation Modelscited by 17
  37. 2022Meta-Learning the Difference: Preparing Large Language Models for Efficient Adaptationcited by 14
  38. 2021Task-adaptive Pre-training and Self-training are Complementary for Natural Language Understandingcited by 12

50 most cited of 51 All 51 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 5; 2024: 6; 2025: 3; 2026: 3.

17 papers.

Sort:

By area:

  1. 2024Instruction Tuning With Loss Over Instructionscited by 40
  2. 2024Securing Large Language Models: Threats, Vulnerabilities and Responsible Practicescited by 36
  3. 2023DePT: Decomposed Prompt Tuning for Parameter-Efficient Fine-tuningcited by 34
  4. 2024Semi-supervised Fine-tuning for Large Language Modelscited by 8
  5. 2025ZeroVO: Visual Odometry with Minimal Assumptionscited by 4
  6. 2023Self Contrastive Learning for Session-based Recommendationcited by 3
  7. 2023Rethink the Effectiveness of Text Data Augmentation: An Empirical Analysiscited by 2
  8. 2024CPRM: A LLM-based Continual Pre-training Framework for Relevance Modeling in Commercial Searchcited by 1
  9. 2025Enhancing Semi-supervised Learning with Zero-shot Pseudolabelscited by 1
  10. 2023Solving Dialogue Grounding Embodied Task in a Simulated Environment using Further Masked Language Modelingcited by 0
  11. 2023Lexical Entrainment for Conversational Systemscited by 0
  12. 2024Improving Agent Interactions in Virtual Environments with Language Modelscited by 0
5 more
  1. 2024A Comparative Study of Pre-training and Self-trainingcited by 0
  2. 2025Towards Anomaly-Aware Pre-Training and Fine-Tuning for Graph Anomaly Detectioncited by 0
  3. 2026BYORn: Bootstrap Your Own Responses to Defend Large Vision-Language Models Against Backdoor Attackscited by 0
  4. 2026DECSELFMASK: Leveraging Unlabeled Text via Self-Relevance-Guided Masking for Decoder-Only Classificationcited by 0
  5. 2026Knowledgeless Language Models: Suppressing Parametric Recall for Evidence-Grounded Language Modelingcited by 0