mlsanityData to 7 Oct 2026

ZeroGen: Efficient Zero-shot Learning via Dataset Generation

arXiv 2202.07922 ↗First posted 16 Feb 2022cs.CL cs.AIcited by 209

Not on the map. Why?

Cited by 29 in Synthetic data and self-improvement for LLMs, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

There is a growing interest in dataset generation recently due to the superior generative capacity of large pre-trained language models (PLMs). In this paper, we study a flexible and efficient zero-short learning method, textscZeroGen. Given a zero-shot task, we first generate a dataset from scratch using PLMs in an unsupervised manner. Then, we train a tiny task model (e.g., LSTM) under the supervision of the synthesized dataset. This approach allows highly efficient inference as the final task model only has orders of magnitude fewer parameters comparing to PLMs (e.g., GPT2-XL). Apart from being annotation-free and efficient, we argue that textscZeroGen can also provide useful insights from the perspective of data-free model-agnostic knowledge distillation, and unreferenced text generation evaluation. Experiments and analysis on different NLP tasks, namely, text classification, question answering, and natural language inference, show the effectiveness of textscZeroGen.

Key facts. Cited by 209 papers in the map (data through 7 Oct 2026). Builds on 61 of its 100 references that are in the map. First posted on arXiv 16 Feb 2022.

Builds on

61 of 100 references.

References · 61

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  4. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  5. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  6. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  7. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  8. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  9. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  10. 2019The Curious Case of Neural Text Degenerationcited by 2,881
  11. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
  12. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,222
38 more
  1. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,201
  2. 2019Language Models as Knowledge Bases?cited by 1,922
  3. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,462
  4. 2018Hierarchical Neural Story Generationcited by 1,182
  5. 2019TinyBERT: Distilling BERT for Natural Language Understandingcited by 1,153
  6. 2019How Can We Know What Language Models Know?cited by 953
  7. 2019CTRL: A Conditional Transformer Language Model for Controllable Generationcited by 754
  8. 2021Prompt Programming for Large Language Models: Beyond the Few-Shot Paradigmcited by 599
  9. 2020Learning from Noisy Labels with Deep Neural Networks: A Surveycited by 580
  10. 2019Neural Text Generation with Unlikelihood Trainingcited by 489
  11. 2018Texygen: A Benchmarking Platform for Text Generation Modelscited by 484
  12. 2020MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devicescited by 476
  13. 2020UnifiedQA: Crossing Format Boundaries With a Single QA Systemcited by 475
  14. 2021MAUVE: Measuring the Gap Between Neural Text and Human Text using Divergence Frontierscited by 466
  15. 2017Non-Autoregressive Neural Machine Translationcited by 457
  16. 2019What do you learn from context? Probing for sentence structure in contextualized word representationscited by 436
  17. 2019Q-BERT: Hessian Based Ultra Low Precision Quantization of BERTcited by 355
  18. 2021Self-Diagnosis and Self-Debiasing: A Proposal for Reducing Corpus-Based Bias in NLPcited by 336
  19. 2016Diverse Beam Search: Decoding Diverse Solutions from Neural Sequence Modelscited by 299
  20. 2019Q8BERT: Quantized 8Bit BERTcited by 296
  21. 2016Bidirectional Attention Flow for Machine Comprehensioncited by 278
  22. 2021Surface Form Competition: Why the Highest Probability Answer Isn't Always Rightcited by 256
  23. 2021I-BERT: Integer-only BERT Quantizationcited by 250
  24. 2019Structured Pruning of Large Language Modelscited by 201
  25. 2022WANLI: Worker and AI Collaboration for Natural Language Inference Dataset Creationcited by 200
  26. 2022Generating Training Data with Language Models: Towards Zero-Shot Language Understandingcited by 199
  27. 2021Generating Datasets with Pretrained Language Modelscited by 192
  28. 2019Assessing BERT's Syntactic Abilitiescited by 191
  29. 2020Compressing BERT: Studying the Effects of Weight Pruning on Transfer Learningcited by 184
  30. 2021Reframing Instructional Prompts to GPTk's Languagecited by 176
  31. 2019Not Enough Data? Deep Learning to the Rescue!cited by 174
  32. 2020Data Augmentation using Pre-trained Transformer Modelscited by 168
  33. 2020Beat the AI: Investigating Adversarial Human Annotation for Reading Comprehensioncited by 131
  34. 2021Adapting Language Models for Zero-shot Learning by Meta-tuning on Dataset and Prompt Collectionscited by 128
  35. 2020Training Question Answering Models From Synthetic Datacited by 95
  36. 2019A Tensorized Transformer for Language Modelingcited by 85
  37. 2021Towards Zero-Label Language Learningcited by 76
  38. 2020UNION: An Unreferenced Metric for Evaluating Open-ended Story Generationcited by 60

50 most cited of 61 All 61 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 14; 2023: 48; 2024: 89; 2025: 39; 2026: 19.

209 papers.

Sort:

By area:

  1. 2022Unnatural Instructions: Tuning Language Models with (Almost) No Human Laborcited by 395
  2. 2024A Survey on Knowledge Distillation of Large Language Modelscited by 276
  3. 2024On LLMs-Driven Synthetic Data Generation, Curation, and Evaluation: A Surveycited by 240
  4. 2023Large Language Model as Attributed Training Data Generator: A Tale of Diversity and Biascited by 230
  5. 2023G-LLaVA: Solving Geometric Problem with Multi-Modal Large Language Modelcited by 208
  6. 2022Self-Adaptive In-Context Learning: An Information Compression Perspective for In-Context Example Selection and Orderingcited by 158
  7. 2024AgentTrek: Agent Trajectory Synthesis via Guiding Replay with Web Tutorialscited by 123
  8. 2023DetGPT: Detect What You Need via Reasoningcited by 112
  9. 2024MLLM-Protector: Ensuring MLLM's Safety without Hurting Performancecited by 111
  10. 2023Synthetic Data Generation with Large Language Models for Text Classification: Potential and Limitationscited by 96
  11. 2023Exploiting Asymmetry for Synthetic Training Data Generation: SynthIE and the Case of Information Extractioncited by 89
  12. 2023Controlled Text Generation with Natural Language Instructionscited by 87
38 more
  1. 2023LLMaAA: Making Large Language Models as Active Annotatorscited by 78
  2. 2022Self-Generated In-Context Learning: Leveraging Auto-regressive Language Models as a Demonstration Generatorcited by 68
  3. 2025Explorer: Scaling Exploration-driven Web Trajectory Synthesis for Multimodal Web Agentscited by 68
  4. 2022ProGen: Progressive Zero-shot Dataset Generation via In-context Feedbackcited by 67
  5. 2023Synthetic Prompting: Generating Chain-of-Thought Demonstrations for Large Language Modelscited by 61
  6. 2022RelationPrompt: Leveraging Prompts to Generate Synthetic Data for Zero-Shot Relation Triplet Extractioncited by 60
  7. 2022Prototypical Calibration for Few-shot Learning of Language Modelscited by 58
  8. 2022Contrastive Search Is What You Need For Neural Text Generationcited by 58
  9. 2024TheoremLlama: Transforming General-Purpose LLMs into Lean4 Expertscited by 58
  10. 2024On the Diversity of Synthetic Data and its Impact on Training Large Language Modelscited by 56
  11. 2024Mitigating the Privacy Issues in Retrieval-Augmented Generation (RAG) via Pure Synthetic Datacited by 52
  12. 2022Language Models in the Loop: Incorporating Prompting into Weak Supervisioncited by 51
  13. 2023FlexKBQA: A Flexible LLM-Powered Framework for Few-Shot Knowledge Base Question Answeringcited by 51
  14. 2022Self-Guided Noise-Free Data Generation for Efficient Zero-Shot Learningcited by 49
  15. 2024Forcing Diffuse Distributions out of Language Modelscited by 48
  16. 2022In-Context Learning for Few-Shot Dialogue State Trackingcited by 47
  17. 2024DataDreamer: A Tool for Synthetic Data Generation and Reproducible LLM Workflowscited by 43
  18. 2024What is the Role of Small Models in the LLM Era: A Surveycited by 41
  19. 2022Self-Prompting Large Language Models for Zero-Shot Open-Domain QAcited by 39
  20. 2022Tuning Language Models as Training Data Generators for Augmentation-Enhanced Few-Shot Learningcited by 36
  21. 2023Give Me the Facts! A Survey on Factual Knowledge Probing in Pre-trained Language Modelscited by 32
  22. 2024A Comprehensive Evaluation of Cognitive Biases in LLMscited by 32
  23. 2023Impossible Distillation: from Low-Quality Model to High-Quality Dataset & Model for Summarization and Paraphrasingcited by 31
  24. 2024A Survey on Data Augmentation in Large Model Eracited by 31
  25. 2024Generative AI for Synthetic Data Generation: Methods, Challenges and the Futurecited by 31
  26. 2024Surveying the Effects of Quality, Diversity, and Complexity in Synthetic Data From Large Language Modelscited by 31
  27. 2023Let's Synthesize Step by Step: Iterative Dataset Synthesis with Large Language Models by Extrapolating Errors from Small Modelscited by 30
  28. 2024CoCA: Regaining Safety-awareness of Multimodal Large Language Models with Constitutional Calibrationcited by 30
  29. 2023Improving Contrastive Learning of Sentence Embeddings from AI Feedbackcited by 29
  30. 2023Making Large Language Models Better Data Creatorscited by 28
  31. 2023Let GPT be a Math Tutor: Teaching Math Word Problem Solvers with Customized Exercise Generationcited by 27
  32. 2023Language Versatilists vs. Specialists: An Empirical Revisiting on Multilingual Transfer Abilitycited by 27
  33. 2023InstructProtein: Aligning Human and Protein Language via Knowledge Instructioncited by 27
  34. 2024Is Programming by Example solved by LLMs?cited by 27
  35. 2023ReGen: Zero-Shot Text Classification via Training Data Generation with Progressive Dense Retrievalcited by 26
  36. 2024TELEClass: Taxonomy Enrichment and LLM-Enhanced Hierarchical Text Classification with Minimal Supervisioncited by 25
  37. 2023Contrastive Learning of Sentence Embeddings from Scratchcited by 24
  38. 2024Personalized Visual Instruction Tuningcited by 24

50 most cited of 209 All 209 (JSON)