Data to 14 Oct 2026

Self-training Improves Pre-training for Natural Language Understanding

arXiv 2010.02194 ↗First posted 5 Oct 2020cs.CLcited by 96

Not on the map. Why?

Abstract (arXiv)

Unsupervised pre-training has led to much recent progress in natural language understanding. In this paper, we study self-training as another way to leverage unlabeled data through semi-supervised learning. To obtain additional data for a specific task, we introduce SentAugment, a data augmentation method which computes task-specific query embeddings from labeled data to retrieve sentences from a bank of billions of unlabeled sentences crawled from the web. Unlike previous semi-supervised methods, our approach does not require in-domain unlabeled data and is therefore more generally applicable. Experiments show that self-training is complementary to strong RoBERTa baselines on a variety of tasks. Our augmentation approach leads to scalable and effective self-training with improvements of up to 2.6% on standard text classification benchmarks. Finally, we also show strong gains on knowledge-distillation and few-shot learning.

Key facts. Cited by 96 papers in the map (data through 14 Oct 2026). First posted on arXiv 5 Oct 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 39; 2022: 32; 2023: 11; 2024: 8; 2025: 4; 2026: 2.

96 papers.

Sort:

By area:

2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,724
2022Discovering Language Model Behaviors with Model-Written Evaluationscited by 836
2021Self-Supervised Representation Learning: Introduction, Advances and Challengescited by 148
2021Cycle Self-Training for Domain Adaptationcited by 120
2021Multi-Task Self-Training for Learning General Representationscited by 85
2021Protecting Intellectual Property of Language Generation APIs with Lexical Watermarkcited by 84
2021Data Augmentation Approaches in Natural Language Processing: A Surveycited by 82
2021Improving Question Answering Model Robustness with Synthetic Adversarial Data Generationcited by 79
2022Improving In-Context Few-Shot Learning via Self-Supervised Trainingcited by 57
2022ZeroPrompt: Scaling Prompt-Based Pretraining to 1,000 Tasks Improves Zero-Shot Generalizationcited by 54
2021Knowing More About Questions Can Help: Improving Calibration in Question Answeringcited by 47
2021Generate, Annotate, and Learn: NLP with Synthetic Textcited by 42

38 more

2021Improved Text Classification via Contrastive Adversarial Trainingcited by 38
2021Large-Scale Self- and Semi-Supervised Learning for Speech Translationcited by 36
2021Multimodal Co-learning: Challenges, Applications with Datasets, Recent Advances and Future Directionscited by 35
2024Active Evaluation Acquisition for Efficient LLM Benchmarkingcited by 35
2021STraTA: Self-Training with Task Augmentation for Better Few-shot Learningcited by 33
2024SimRAG: Self-Improving Retrieval-Augmented Generation for Adapting Large Language Models to Specialized Domainscited by 28
2022Zero-Shot Text Classification with Self-Trainingcited by 26
2023Denevil: Towards Deciphering and Navigating the Ethical Values of Large Language Models via Instruction Learningcited by 26
2021Revisiting Self-Training for Few-Shot Learning of Language Modelcited by 23
2024Understanding Layer Significance in LLM Alignmentcited by 19
2021XtremeDistilTransformers: Task Transfer for Task-agnostic Distillationcited by 18
2021Named Entity Recognition with Small Strongly Labeled and Large Weakly Labeled Datacited by 18
2021Self-training Improves Pre-training for Few-shot Learning in Task-oriented Dialog Systemscited by 18
2022You can't pick your neighbors, or can you? When and how to rely on retrieval in the kNN-LMcited by 18
2021Crossing the Conversational Chasm: A Primer on Natural Language Processing for Multilingual Task-Oriented Dialogue Systemscited by 17
2021AUGNLG: Few-shot Natural Language Generation using Self-trained Data Augmentationcited by 17
2022Improving Contrastive Learning of Sentence Embeddings with Case-Augmented Positives and Retrieved Negativescited by 16
2022Cold-Start Data Selection for Few-shot Language Model Fine-tuning: A Prompt-Based Uncertainty Propagation Approachcited by 16
2021Credal Self-Supervised Learningcited by 15
2022Learning Instructions with Unlabeled Data for Zero-Shot Cross-Task Generalizationcited by 15
2023Neighborhood-Regularized Self-Training for Learning with Few Labelscited by 14
2021Task-adaptive Pre-training and Self-training are Complementary for Natural Language Understandingcited by 12
2022Leveraging QA Datasets to Improve Generative Data Augmentationcited by 12
2022AANG: Automating Auxiliary Learningcited by 12
2021VarCLR: Variable Semantic Representation Pre-training via Contrastive Learningcited by 11
2022Improving Short Text Classification With Augmented Data Using GPT-3cited by 11
2021Not Far Away, Not So Close: Sample Efficient Nearest Neighbour Data Augmentation via MiniMaxcited by 10
2021Diverse Distributions of Self-Supervised Tasks for Meta-Learning in NLPcited by 10
2021Fine-tuning Encoders for Improved Monolingual and Zero-shot Polylingual Neural Topic Modelingcited by 8
2021Nearest Neighbour Few-Shot Learning for Cross-lingual Classificationcited by 8
2022Conformal Credal Self-Supervised Learningcited by 8
2024Pre-training Differentially Private Models with Limited Public Datacited by 8
2021Unsupervised Self-Training for Sentiment Analysis of Code-Switched Datacited by 7
2021Pretext Tasks selection for multitask self-supervised speech representation learningcited by 7
2021Self-Training with Differentiable Teachercited by 7
2022The Dialog Must Go On: Improving Visual Dialog via Generative Self-Trainingcited by 7
2022Unsupervised Visual Representation Learning via Mutual Information Regularized Assignmentcited by 7
2021Self-Teaching Machines to Read and Comprehend with Large-Scale Multi-Subject Question-Answering Datacited by 6

50 most cited of 96 All 96 (JSON)