# Revisiting Self-Training for Few-Shot Learning of Language Model

arXiv: https://arxiv.org/abs/2110.01256 · HTML: https://mlsanity.com/paper/2110.01256/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- Placed in Self-training for few-shot NLP, one of 13 directions in Synthetic data and self-improvement for LLMs.
- Builds on 34 of its 54 references that are in the map.
- First posted on arXiv 4 Oct 2021.

## Placed in

- Synthetic data and self-improvement for LLMs / [Self-training for few-shot NLP](https://mlsanity.com/synth/self-training-for-few-shot-nlp/)

## Builds on (most cited first, 30 of 34)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018)
- [Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks](https://mlsanity.com/paper/1703.03400/) (2017)
- [SQuAD: 100,000+ Questions for Machine Comprehension of Text](https://mlsanity.com/paper/1606.05250/) (2016)
- [Prototypical Networks for Few-shot Learning](https://mlsanity.com/paper/1703.05175/) (2017)
- [XLNet: Generalized Autoregressive Pretraining for Language Understanding](https://mlsanity.com/paper/1906.08237/) (2019)
- [ALBERT: A Lite BERT for Self-supervised Learning of Language Representations](https://mlsanity.com/paper/1909.11942/) (2019)
- [A Broad-Coverage Challenge Corpus for Sentence Understanding through Inference](https://mlsanity.com/paper/1704.05426/) (2017)
- [Matching Networks for One Shot Learning](https://mlsanity.com/paper/1606.04080/) (2016)
- [SimCSE: Simple Contrastive Learning of Sentence Embeddings](https://mlsanity.com/paper/2104.08821/) (2021)
- [FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidence](https://mlsanity.com/paper/2001.07685/) (2020)
- [A large annotated corpus for learning natural language inference](https://mlsanity.com/paper/1508.05326/) (2015)
- [Self-training with Noisy Student improves ImageNet classification](https://mlsanity.com/paper/1911.04252/) (2019)
- [Making Pre-trained Language Models Better Few-shot Learners](https://mlsanity.com/paper/2012.15723/) (2020)
- [Unsupervised Data Augmentation for Consistency Training](https://mlsanity.com/paper/1904.12848/) (2019)
- [Exploiting Cloze Questions for Few Shot Text Classification and Natural Language Inference](https://mlsanity.com/paper/2001.07676/) (2020)
- [EDA: Easy Data Augmentation Techniques for Boosting Performance on Text Classification Tasks](https://mlsanity.com/paper/1901.11196/) (2019)
- [AutoAugment: Learning Augmentation Policies from Data](https://mlsanity.com/paper/1805.09501/) (2018)
- [It's Not Just Size That Matters: Small Language Models Are Also Few-Shot Learners](https://mlsanity.com/paper/2009.07118/) (2020)
- [Rethinking Pre-training and Self-training](https://mlsanity.com/paper/2006.06882/) (2020)
- [SentEval: An Evaluation Toolkit for Universal Sentence Representations](https://mlsanity.com/paper/1803.05449/) (2018)
- [Supervised Contrastive Learning for Pre-trained Language Model Fine-tuning](https://mlsanity.com/paper/2011.01403/) (2020)
- [FewRel: A Large-Scale Supervised Few-Shot Relation Classification Dataset with State-of-the-Art Evaluation](https://mlsanity.com/paper/1810.10147/) (2018)
- [CLEAR: Contrastive Learning for Sentence Representation](https://mlsanity.com/paper/2012.15466/) (2020)
- [Theoretical Analysis of Self-Training with Deep Networks on Unlabeled Data](https://mlsanity.com/paper/2010.03622/) (2020)
- [Revisiting Self-Training for Neural Sequence Generation](https://mlsanity.com/paper/1909.13788/) (2019)

## Built on by (most cited first, 23 of 23)

- [BadPrompt: Backdoor Attacks on Continuous Prompts](https://mlsanity.com/paper/2211.14719/) (2022)
- [Sibling-Attack: Rethinking Transferable Adversarial Attacks against Face Recognition](https://mlsanity.com/paper/2303.12512/) (2023)
- [How to Design Translation Prompts for ChatGPT: An Empirical Study](https://mlsanity.com/paper/2304.02182/) (2023)
- [FineD-Eval: Fine-grained Automatic Dialogue-Level Evaluation](https://mlsanity.com/paper/2210.13832/) (2022)
- [RT-LM: Uncertainty-Aware Resource Management for Real-Time Inference of Language Models](https://mlsanity.com/paper/2309.06619/) (2023)
- [How Many Validation Labels Do You Need? Exploring the Design Space of Label-Efficient Model Ranking](https://mlsanity.com/paper/2312.01619/) (2023)
- [RED: A Systematic Real-Time Scheduling Approach for Robotic Environmental Dynamics](https://mlsanity.com/paper/2308.15368/) (2023)
- [Active Few-Shot Learning for Text Classification](https://mlsanity.com/paper/2502.18782/) (2025)
- [DuNST: Dual Noisy Self Training for Semi-Supervised Controllable Text Generation](https://mlsanity.com/paper/2212.08724/) (2022)
- [R^3: On-device Real-Time Deep Reinforcement Learning for Autonomous Robotics](https://mlsanity.com/paper/2308.15039/) (2023)
- [LLM aided semi-supervision for Extractive Dialog Summarization](https://mlsanity.com/paper/2311.11462/) (2023)
- [From Wrong To Right: A Recursive Approach Towards Vision-Language Explanation](https://mlsanity.com/paper/2311.12391/) (2023)
- [IDoFew: Intermediate Training Using Dual-Clustering in Language Models for Few Labels Text Classification](https://mlsanity.com/paper/2401.04025/) (2024)
- [Pathologies of Pre-trained Language Models in Few-shot Fine-tuning](https://mlsanity.com/paper/2204.08039/) (2022)
- [AdaSent: Efficient Domain-Adapted Sentence Embeddings for Few-Shot Classification](https://mlsanity.com/paper/2311.00408/) (2023)
- [Data-efficient Active Learning for Structured Prediction with Partial Annotation and Self-Training](https://mlsanity.com/paper/2305.12634/) (2023)
- [Semi-Supervised Lifelong Language Learning](https://mlsanity.com/paper/2211.13050/) (2022)
- [Enhancing Black-Box Few-Shot Text Classification with Prompt-Based Data Augmentation](https://mlsanity.com/paper/2305.13785/) (2023)
- [KEST: Kernel Distance Based Efficient Self-Training for Improving Controllable Text Generation](https://mlsanity.com/paper/2306.10414/) (2023)
- [Boosting Prompt-Based Self-Training With Mapping-Free Automatic Verbalizer for Multi-Class Classification](https://mlsanity.com/paper/2312.04982/) (2023)
- [CrossTune: Black-Box Few-Shot Classification with Label Enhancement](https://mlsanity.com/paper/2403.12468/) (2024)
- [Can LLMs Solve My Grandma's Riddle? Evaluating Multilingual Large Language Models on Reasoning Traditional Bangla Tricky Riddles](https://mlsanity.com/paper/2512.20324/) (2025)
- [TetraRL: A Self-Adaptive Runtime for On-Device Deep Reinforcement Learning Systems](https://mlsanity.com/paper/2606.13891/) (2026)

## Abstract (arXiv)

As unlabeled data carry rich task-relevant information, they are proven useful for few-shot learning of language model. The question is how to effectively make use of such data. In this work, we revisit the self-training technique for language model fine-tuning and present a state-of-the-art prompt-based few-shot learner, SFLM. Given two views of a text sample via weak and strong augmentation techniques, SFLM generates a pseudo label on the weakly augmented version. Then, the model predicts the same pseudo label when fine-tuned with the strongly augmented version. This simple approach is shown to outperform other state-of-the-art supervised and semi-supervised counterparts on six sentence classification and six sentence-pair classification benchmarking tasks. In addition, SFLM only relies on a few in-domain unlabeled data. We conduct a comprehensive analysis to demonstrate the robustness of our proposed approach under various settings, including augmentation techniques, model scale, and few-shot knowledge transfer across tasks.