mlsanityData to 7 Oct 2026

Generating Training Data with Language Models: Towards Zero-Shot Language Understanding

arXiv 2202.04538 ↗First posted 9 Feb 2022cs.CL cs.LGcited by 199

Not on the map. Why?

Cited by 29 in Synthetic data and self-improvement for LLMs, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Pretrained language models (PLMs) have demonstrated remarkable performance in various natural language processing tasks: Unidirectional PLMs (e.g., GPT) are well known for their superior text generation capabilities; bidirectional PLMs (e.g., BERT) have been the prominent choice for natural language understanding (NLU) tasks. While both types of models have achieved promising few-shot learning performance, their potential for zero-shot learning has been underexplored. In this paper, we present a simple approach that uses both types of PLMs for fully zero-shot learning of NLU tasks without requiring any task-specific data: A unidirectional PLM generates class-conditioned texts guided by prompts, which are used as the training data for fine-tuning a bidirectional PLM. With quality training data selected based on the generation probability and regularization techniques (label smoothing and temporal ensembling) applied to the fine-tuning stage for better generalization and stability, our approach demonstrates strong performance across seven classification tasks of the GLUE benchmark (e.g., 72.3/73.8 on MNLI-m/mm and 92.8 on SST-2), significantly outperforming zero-shot prompting methods and achieving even comparable results to strong few-shot approaches using 32 training samples per class.

Key facts. Cited by 199 papers in the map (data through 7 Oct 2026). Builds on 72 of its 104 references that are in the map. First posted on arXiv 9 Feb 2022.

Builds on

72 of 104 references.

References · 72

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  6. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  7. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  8. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  9. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,451
  10. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  11. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  12. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
38 more
  1. 2019The Curious Case of Neural Text Degenerationcited by 2,881
  2. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
  3. 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,168
  4. 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,157
  5. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  6. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,718
  7. 2016Understanding deep learning requires rethinking generalizationcited by 1,657
  8. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,462
  9. 2021Calibrate Before Use: Improving Few-Shot Performance of Language Modelscited by 1,371
  10. 2020Making Pre-trained Language Models Better Few-shot Learnerscited by 1,255
  11. 2020RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Modelscited by 1,235
  12. 2019A Comprehensive Survey on Transfer Learningcited by 1,154
  13. 2019Unsupervised Data Augmentation for Consistency Trainingcited by 1,057
  14. 2016Temporal Ensembling for Semi-Supervised Learningcited by 1,054
  15. 2019When Does Label Smoothing Help?cited by 985
  16. 2018Neural Network Acceptability Judgmentscited by 952
  17. 2020Exploiting Cloze Questions for Few Shot Text Classification and Natural Language Inferencecited by 934
  18. 2019Plug and Play Language Models: A Simple Approach to Controlled Text Generationcited by 793
  19. 2019CTRL: A Conditional Transformer Language Model for Controllable Generationcited by 754
  20. 2021BARTScore: Evaluating Generated Text as Text Generationcited by 719
  21. 2021GPT Understands, Toocited by 715
  22. 2021Cross-Task Generalization via Natural Language Crowdsourcing Instructionscited by 696
  23. 2020It's Not Just Size That Matters: Small Language Models Are Also Few-Shot Learnerscited by 668
  24. 2019Neural Text Generation with Unlikelihood Trainingcited by 489
  25. 2021DExperts: Decoding-Time Controlled Text Generation with Experts and Anti-Expertscited by 434
  26. 2020GeDi: Generative Discriminator Guided Sequence Generationcited by 377
  27. 2021FUDGE: Controlled Text Generation With Future Discriminatorscited by 374
  28. 2021True Few-Shot Learning with Language Modelscited by 343
  29. 2019Benchmarking Zero-shot Text Classification: Datasets, Evaluation and Entailment Approachcited by 299
  30. 2021Understanding Factuality in Abstractive Summarization with FRANK: A Benchmark for Factuality Metricscited by 285
  31. 2016Adversarial Training Methods for Semi-Supervised Text Classificationcited by 280
  32. 2017Toward Controlled Generation of Textcited by 271
  33. 2020Does label smoothing mitigate label noise?cited by 204
  34. 2021How Many Data Points is a Prompt Worth?cited by 197
  35. 2021Generating Datasets with Pretrained Language Modelscited by 192
  36. 2020MixText: Linguistically-Informed Interpolation of Hidden Space for Semi-Supervised Text Classificationcited by 186
  37. 2021GPT3Mix: Leveraging Large-scale Language Models for Text Augmentationcited by 185
  38. 2019Not Enough Data? Deep Learning to the Rescue!cited by 174

50 most cited of 72 All 72 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 19; 2023: 60; 2024: 78; 2025: 29; 2026: 13.

199 papers.

Sort:

By area:

  1. 2023CAMEL: Communicative Agents for "Mind" Exploration of Large Language Model Societycited by 1,348
  2. 2022Discovering Language Model Behaviors with Model-Written Evaluationscited by 805
  3. 2022Large Language Models Can Self-Improvecited by 656
  4. 2024A Survey on Large Language Models for Code Generationcited by 646
  5. 2022Large Language Models Are Reasoning Teacherscited by 318
  6. 2024A Survey on Knowledge Distillation of Large Language Modelscited by 276
  7. 2024On LLMs-Driven Synthetic Data Generation, Curation, and Evaluation: A Surveycited by 240
  8. 2023Large Language Model as Attributed Training Data Generator: A Tale of Diversity and Biascited by 230
  9. 2024When Can LLMs Actually Correct Their Own Mistakes? A Critical Survey of Self-Correction of LLMscited by 230
  10. 2022ZeroGen: Efficient Zero-shot Learning via Dataset Generationcited by 209
  11. 2022Is GPT-3 a Good Data Annotator?cited by 208
  12. 2023G-LLaVA: Solving Geometric Problem with Multi-Modal Large Language Modelcited by 208
38 more
  1. 2023Large Language Models on Graphs: A Comprehensive Surveycited by 190
  2. 2023StableRep: Synthetic Images from Text-to-Image Models Make Strong Visual Representation Learnerscited by 177
  3. 2023Symbolic Chain-of-Thought Distillation: Small Models Can Also "Think" Step-by-Stepcited by 144
  4. 2024AutoWebGLM: A Large Language Model-based Web Navigating Agentcited by 130
  5. 2023DetGPT: Detect What You Need via Reasoningcited by 112
  6. 2023Scaling Laws of Synthetic Images for Model Training ... for Nowcited by 107
  7. 2024Best Practices and Lessons Learned on Synthetic Datacited by 104
  8. 2023Synthetic Data Generation with Large Language Models for Text Classification: Potential and Limitationscited by 96
  9. 2023Exploiting Asymmetry for Synthetic Training Data Generation: SynthIE and the Case of Information Extractioncited by 89
  10. 2023LLMaAA: Making Large Language Models as Active Annotatorscited by 78
  11. 2022Language Models Can Teach Themselves to Program Bettercited by 75
  12. 2024Strengthening Multimodal Large Language Model with Bootstrapped Preference Optimizationcited by 75
  13. 2023PLACES: Prompting Language Models for Social Conversation Synthesiscited by 72
  14. 2023Learning Vision from Models Rivals Learning Vision from Datacited by 71
  15. 2022ProGen: Progressive Zero-shot Dataset Generation via In-context Feedbackcited by 67
  16. 2022InPars: Data Augmentation for Information Retrieval using Large Language Modelscited by 60
  17. 2022RelationPrompt: Leveraging Prompts to Generate Synthetic Data for Zero-Shot Relation Triplet Extractioncited by 60
  18. 2022Prototypical Calibration for Few-shot Learning of Language Modelscited by 58
  19. 2024TheoremLlama: Transforming General-Purpose LLMs into Lean4 Expertscited by 58
  20. 2024Do LLMs Have Distinct and Consistent Personality? TRAIT: Personality Testset designed for LLMs with Psychometricscited by 56
  21. 2023VITATECS: A Diagnostic Dataset for Temporal Concept Understanding of Video-Language Modelscited by 55
  22. 2023Large Language Models in Education: Vision and Opportunitiescited by 54
  23. 2024Mitigating the Privacy Issues in Retrieval-Augmented Generation (RAG) via Pure Synthetic Datacited by 52
  24. 2023FlexKBQA: A Flexible LLM-Powered Framework for Few-Shot Knowledge Base Question Answeringcited by 51
  25. 2022Data Augmentation for Intent Classification with Off-the-shelf Large Language Modelscited by 49
  26. 2022Self-Guided Noise-Free Data Generation for Efficient Zero-Shot Learningcited by 49
  27. 2024Forcing Diffuse Distributions out of Language Modelscited by 48
  28. 2023Measuring and Improving Chain-of-Thought Reasoning in Vision-Language Modelscited by 45
  29. 2025EvolveSearch: An Iterative Self-Evolving Search Agentcited by 42
  30. 2024What is the Role of Small Models in the LLM Era: A Surveycited by 41
  31. 2022Tuning Language Models as Training Data Generators for Augmentation-Enhanced Few-Shot Learningcited by 36
  32. 2023Controllable Mixed-Initiative Dialogue Generation through Promptingcited by 32
  33. 2024A Survey on Data Augmentation in Large Model Eracited by 31
  34. 2024Generative AI for Synthetic Data Generation: Methods, Challenges and the Futurecited by 31
  35. 2023Let's Synthesize Step by Step: Iterative Dataset Synthesis with Large Language Models by Extrapolating Errors from Small Modelscited by 30
  36. 2025CoT-Self-Instruct: Building high-quality synthetic prompts for reasoning and non-reasoning taskscited by 30
  37. 2023Improving Contrastive Learning of Sentence Embeddings from AI Feedbackcited by 29
  38. 2023Making Large Language Models Better Data Creatorscited by 28

50 most cited of 199 All 199 (JSON)