mlsanityData to 14 Oct 2026

Revisiting Self-Training for Few-Shot Learning of Language Model

arXiv 2110.01256 ↗First posted 4 Oct 2021cs.CLcited by 23

On the map: Synthetic data and self-improvement for LLMs → Self-training for few-shot NLP

Abstract (arXiv)

As unlabeled data carry rich task-relevant information, they are proven useful for few-shot learning of language model. The question is how to effectively make use of such data. In this work, we revisit the self-training technique for language model fine-tuning and present a state-of-the-art prompt-based few-shot learner, SFLM. Given two views of a text sample via weak and strong augmentation techniques, SFLM generates a pseudo label on the weakly augmented version. Then, the model predicts the same pseudo label when fine-tuned with the strongly augmented version. This simple approach is shown to outperform other state-of-the-art supervised and semi-supervised counterparts on six sentence classification and six sentence-pair classification benchmarking tasks. In addition, SFLM only relies on a few in-domain unlabeled data. We conduct a comprehensive analysis to demonstrate the robustness of our proposed approach under various settings, including augmentation techniques, model scale, and few-shot knowledge transfer across tasks.

Key facts. Cited by 23 papers in the map (data through 14 Oct 2026). Placed in Self-training for few-shot NLP, one of 13 directions in Synthetic data and self-improvement for LLMs. Builds on 34 of its 54 references that are in the map. First posted on arXiv 4 Oct 2021.

Builds on

34 of 54 references.

References · 34

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  3. 2020Language Models are Few-Shot Learnerscited by 32,419
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  5. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  6. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  7. 2017Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networkscited by 4,897
  8. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,148
  9. 2017Prototypical Networks for Few-shot Learningcited by 3,015
  10. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,811
  11. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,790
  12. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,682
22 more
  1. 2016Matching Networks for One Shot Learningcited by 2,255
  2. 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,173
  3. 2020FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidencecited by 2,085
  4. 2015A large annotated corpus for learning natural language inferencecited by 1,847
  5. 2019Self-training with Noisy Student improves ImageNet classificationcited by 1,311
  6. 2020Making Pre-trained Language Models Better Few-shot Learnerscited by 1,256
  7. 2019Unsupervised Data Augmentation for Consistency Trainingcited by 1,060
  8. 2020Exploiting Cloze Questions for Few Shot Text Classification and Natural Language Inferencecited by 938
  9. 2019EDA: Easy Data Augmentation Techniques for Boosting Performance on Text Classification Taskscited by 807
  10. 2018AutoAugment: Learning Augmentation Policies from Datacited by 730
  11. 2020It's Not Just Size That Matters: Small Language Models Are Also Few-Shot Learnerscited by 669
  12. 2020Rethinking Pre-training and Self-trainingcited by 352
  13. 2018SentEval: An Evaluation Toolkit for Universal Sentence Representationscited by 294
  14. 2020Supervised Contrastive Learning for Pre-trained Language Model Fine-tuningcited by 276
  15. 2018FewRel: A Large-Scale Supervised Few-Shot Relation Classification Dataset with State-of-the-Art Evaluationcited by 254
  16. 2020CLEAR: Contrastive Learning for Sentence Representationcited by 186
  17. 2020Theoretical Analysis of Self-Training with Deep Networks on Unlabeled Datacited by 180
  18. 2019Revisiting Self-Training for Neural Sequence Generationcited by 155
  19. 2020Self-training Improves Pre-training for Natural Language Understandingcited by 96
  20. 2018Diverse Few-Shot Text Classification with Multiple Metricscited by 61
  21. 2019Few-shot Text Classification with Distributional Signaturescited by 58
  22. 2020Self-Supervised Meta-Learning for Few-Shot Natural Language Classification Taskscited by 51

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 5; 2023: 13; 2024: 2; 2025: 2; 2026: 1.

23 papers.

Sort:

By area:

  1. 2022BadPrompt: Backdoor Attacks on Continuous Promptscited by 63
  2. 2023Sibling-Attack: Rethinking Transferable Adversarial Attacks against Face Recognitioncited by 34
  3. 2023How to Design Translation Prompts for ChatGPT: An Empirical Studycited by 31
  4. 2022FineD-Eval: Fine-grained Automatic Dialogue-Level Evaluationcited by 13
  5. 2023RT-LM: Uncertainty-Aware Resource Management for Real-Time Inference of Language Modelscited by 11
  6. 2023How Many Validation Labels Do You Need? Exploring the Design Space of Label-Efficient Model Rankingcited by 8
  7. 2023RED: A Systematic Real-Time Scheduling Approach for Robotic Environmental Dynamicscited by 6
  8. 2025Active Few-Shot Learning for Text Classificationcited by 6
  9. 2022DuNST: Dual Noisy Self Training for Semi-Supervised Controllable Text Generationcited by 5
  10. 2023R^3: On-device Real-Time Deep Reinforcement Learning for Autonomous Roboticscited by 5
  11. 2023LLM aided semi-supervision for Extractive Dialog Summarizationcited by 4
  12. 2023From Wrong To Right: A Recursive Approach Towards Vision-Language Explanationcited by 4
11 more
  1. 2024IDoFew: Intermediate Training Using Dual-Clustering in Language Models for Few Labels Text Classificationcited by 4
  2. 2022Pathologies of Pre-trained Language Models in Few-shot Fine-tuningcited by 3
  3. 2023AdaSent: Efficient Domain-Adapted Sentence Embeddings for Few-Shot Classificationcited by 3
  4. 2023Data-efficient Active Learning for Structured Prediction with Partial Annotation and Self-Trainingcited by 2
  5. 2022Semi-Supervised Lifelong Language Learningcited by 1
  6. 2023Enhancing Black-Box Few-Shot Text Classification with Prompt-Based Data Augmentationcited by 1
  7. 2023KEST: Kernel Distance Based Efficient Self-Training for Improving Controllable Text Generationcited by 0
  8. 2023Boosting Prompt-Based Self-Training With Mapping-Free Automatic Verbalizer for Multi-Class Classificationcited by 0
  9. 2024CrossTune: Black-Box Few-Shot Classification with Label Enhancementcited by 0
  10. 2025Can LLMs Solve My Grandma's Riddle? Evaluating Multilingual Large Language Models on Reasoning Traditional Bangla Tricky Riddlescited by 0
  11. 2026TetraRL: A Self-Adaptive Runtime for On-Device Deep Reinforcement Learning Systemscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2021LiST: Lite Prompted Self-training Makes Parameter-Efficient Few-shot Learners
  2. 2022Zero-Shot Text Classification with Self-Training
  3. 2021Self-training Improves Pre-training for Few-shot Learning in Task-oriented Dialog Systems
  4. 2024Few-Shot Recognition via Stage-Wise Retrieval-Augmented Finetuning
  5. 2023Generation-driven Contrastive Self-training for Zero-shot Text Classification with Instruction-following LLM
  6. 2021STraTA: Self-Training with Task Augmentation for Better Few-shot Learning
  7. 2024Making Large Vision Language Models to be Good Few-shot Learners
  8. 2024Many-Shot In-Context Learning