mlsanityData to 7 Oct 2026

STraTA: Self-Training with Task Augmentation for Better Few-shot Learning

arXiv 2109.06270 ↗First posted 13 Sep 2021cs.CLcited by 33

On the map: Synthetic data and self-improvement for LLMs → Self-training for few-shot NLP

Abstract (arXiv)

Despite their recent successes in tackling many NLP tasks, large-scale pre-trained language models do not perform as well in few-shot settings where only a handful of training examples are available. To address this shortcoming, we propose STraTA, which stands for Self-Training with Task Augmentation, an approach that builds on two key ideas for effective leverage of unlabeled data. First, STraTA uses task augmentation, a novel technique that synthesizes a large amount of data for auxiliary-task fine-tuning from target-task unlabeled texts. Second, STraTA performs self-training by further fine-tuning the strong base model created by task augmentation on a broad distribution of pseudo-labeled data. Our experiments demonstrate that STraTA can substantially improve sample efficiency across 12 few-shot benchmarks. Remarkably, on the SST-2 sentiment dataset, STraTA, with only 8 training examples per class, achieves comparable results to standard fine-tuning with 67K training examples. Our analyses reveal that task augmentation and self-training are both complementary and independently effective.

Key facts. Cited by 33 papers in the map (data through 7 Oct 2026). Placed in Self-training for few-shot NLP, one of 13 directions in Synthetic data and self-improvement for LLMs. Builds on 50 of its 67 references that are in the map. First posted on arXiv 13 Sep 2021.

Builds on

50 of 67 references.

References · 50

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  5. 2017On Calibration of Modern Neural Networkscited by 4,860
  6. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  7. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  8. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,801
  9. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
  10. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
  11. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,201
  12. 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,157
38 more
  1. 2020FixMatch: Simplifying Semi-Supervised Learning with Consistency and Confidencecited by 2,080
  2. 2015A large annotated corpus for learning natural language inferencecited by 1,837
  3. 2020Don't Stop Pretraining: Adapt Language Models to Domains and Taskscited by 1,548
  4. 2019Self-training with Noisy Student improves ImageNet classificationcited by 1,304
  5. 2020Making Pre-trained Language Models Better Few-shot Learnerscited by 1,255
  6. 2019Unsupervised Data Augmentation for Consistency Trainingcited by 1,057
  7. 2017SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluationcited by 998
  8. 2019When Does Label Smoothing Help?cited by 985
  9. 2018Adafactor: Adaptive Learning Rates with Sublinear Memory Costcited by 896
  10. 2019EDA: Easy Data Augmentation Techniques for Boosting Performance on Text Classification Taskscited by 800
  11. 2015Improving Neural Machine Translation Models with Monolingual Datacited by 784
  12. 2020It's Not Just Size That Matters: Small Language Models Are Also Few-Shot Learnerscited by 668
  13. 2017Regularizing Neural Networks by Penalizing Confident Output Distributionscited by 461
  14. 2020How Can We Know When Language Models Know? On the Calibration of Language Models for Question Answeringcited by 439
  15. 2017Supervised Learning of Universal Sentence Representations from Natural Language Inference Datacited by 428
  16. 2018SentEval: An Evaluation Toolkit for Universal Sentence Representationscited by 294
  17. 2020Supervised Contrastive Learning for Pre-trained Language Model Fine-tuningcited by 275
  18. 2020On the Stability of Fine-tuning BERT: Misconceptions, Explanations, and Strong Baselinescited by 254
  19. 2019SMART: Robust and Efficient Fine-Tuning for Pre-trained Natural Language Models through Principled Regularized Optimizationcited by 253
  20. 2020Revisiting Few-sample BERT Fine-tuningcited by 237
  21. 2018Sentence Encoders on STILTs: Supplementary Training on Intermediate Labeled-data Taskscited by 214
  22. 2021Muppet: Massive Multi-task Representations with Pre-Finetuningcited by 192
  23. 2021Generating Datasets with Pretrained Language Modelscited by 192
  24. 2019Not Enough Data? Deep Learning to the Rescue!cited by 174
  25. 2020Better Fine-Tuning by Reducing Representational Collapsecited by 156
  26. 2019Revisiting Self-Training for Neural Sequence Generationcited by 154
  27. 2020Intermediate-Task Transfer Learning with Pretrained Models for Natural Language Understanding: When and Why Does It Work?cited by 127
  28. 2020Exploring and Predicting Transferability across NLP Taskscited by 126
  29. 2021Entailment as Few-Shot Learnercited by 122
  30. 2021Improving and Simplifying Pattern Exploiting Trainingcited by 97
  31. 2020Self-training Improves Pre-training for Natural Language Understandingcited by 96
  32. 2020Training Question Answering Models From Synthetic Datacited by 95
  33. 2019Learning and Evaluating General Linguistic Intelligencecited by 78
  34. 2021Neural Data Augmentation via Example Extrapolationcited by 54
  35. 2020Self-Supervised Meta-Learning for Few-Shot Natural Language Classification Taskscited by 51
  36. 2021Generate, Annotate, and Learn: NLP with Synthetic Textcited by 42
  37. 2019Towards Realistic Practices In Low-Resource Natural Language Processing: The Development Setcited by 16
  38. 2020Neural Semi-supervised Learning for Text Classification Under Large-Scale Pretrainingcited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 3; 2022: 10; 2023: 9; 2024: 7; 2025: 3; 2026: 1.

33 papers.

Sort:

By area:

  1. 2021SPoT: Better Frozen Model Adaptation through Soft Prompt Transfercited by 235
  2. 2022ZeroGen: Efficient Zero-shot Learning via Dataset Generationcited by 209
  3. 2022Overcoming Catastrophic Forgetting in Zero-Shot Cross-Lingual Generationcited by 62
  4. 2022Improving In-Context Few-Shot Learning via Self-Supervised Trainingcited by 56
  5. 2022Stretching Sentence-pair NLI Models to Reason over Long Documents and Clusterscited by 44
  6. 2024A Survey on Data Augmentation in Large Model Eracited by 31
  7. 2024SimRAG: Self-Improving Retrieval-Augmented Generation for Adapting Large Language Models to Specialized Domainscited by 28
  8. 2021Unsupervised Neural Machine Translation with Generative Language Models Onlycited by 24
  9. 2024A synthetic data approach for domain generalization of NLI modelscited by 15
  10. 2024Language Models can Exploit Cross-Task In-context Learning for Data-Scarce Novel Taskscited by 13
  11. 2022Leveraging QA Datasets to Improve Generative Data Augmentationcited by 12
  12. 2023On the Importance of Spatial Relations for Few-shot Action Recognitioncited by 11
21 more
  1. 2021LiST: Lite Prompted Self-training Makes Parameter-Efficient Few-shot Learnerscited by 9
  2. 2022Automatic Rule Induction for Interpretable Semi-Supervised Learningcited by 6
  3. 2025Efficient Model Development through Fine-tuning Transfercited by 6
  4. 2022DuNST: Dual Noisy Self Training for Semi-Supervised Controllable Text Generationcited by 5
  5. 2023MetaModulation: Learning Variational Feature Hierarchies for Few-Shot Learning with Fewer Taskscited by 4
  6. 2023A Unified Framework for Generative Data Augmentation: A Comprehensive Surveycited by 4
  7. 2024LLM-GAN: Construct Generative Adversarial Network Through Large Language Models For Explainable Fake News Detectioncited by 4
  8. 2023BLIAM: Literature-based Data Synthesis for Synergistic Drug Combination Predictioncited by 3
  9. 2025Bridging Generative and Discriminative Learning: Few-Shot Relation Extraction via Two-Stage Knowledge-Guided Pre-trainingcited by 3
  10. 2023Data-efficient Active Learning for Structured Prediction with Partial Annotation and Self-Trainingcited by 2
  11. 2023Few-shot Unified Question Answering: Tuning Models or Prompts?cited by 2
  12. 2023Neural Networks Against (and For) Self-Training: Classification with Small Labeled and Large Unlabeled Setscited by 2
  13. 2024Auto-GDA: Automatic Domain Adaptation for Efficient Grounding Verification in Retrieval-Augmented Generationcited by 2
  14. 2025A Model Can Help Itself: Reward-Free Self-Training for LLM Reasoningcited by 2
  15. 2022Few-shot Mining of Naturally Occurring Inputs and Outputscited by 1
  16. 2022Self-Training with Purpose Preserving Augmentation Improves Few-shot Generative Dialogue State Trackingcited by 0
  17. 2022An Augmentation Strategy for Visually Rich Documentscited by 0
  18. 2023Few-shot Multimodal Multitask Multilingual Learningcited by 0
  19. 2023KEST: Kernel Distance Based Efficient Self-Training for Improving Controllable Text Generationcited by 0
  20. 2024Self-Training for Sample-Efficient Active Learning for Text Classification with Pre-Trained Language Modelscited by 0
  21. 2026Post-Generation Curation of Synthetic Images via Homogeneous-Heterogeneous Splittingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2021Revisiting Self-Training for Few-Shot Learning of Language Model
  2. 2021Self-training Improves Pre-training for Few-shot Learning in Task-oriented Dialog Systems
  3. 2021Self-training with Few-shot Rationalization: Teacher Explanations Aid Student in Few-shot NLU
  4. 2023Self-training Strategies for Sentiment Analysis: An Empirical Study
  5. 2022kNN-Prompt: Nearest Neighbor Zero-Shot Inference
  6. 2023Few Shot Rationale Generation using Self-Training with Dual Teachers
  7. 2022UL2: Unifying Language Learning Paradigms
  8. 2026Two-dimensional early exit optimisation of LLM inference