mlsanityData to 13 Oct 2026

Towards Zero-Label Language Learning

arXiv 2109.09193 ↗First posted 19 Sep 2021cs.CL cs.LGcited by 76

Not on the map. Why?

Abstract (arXiv)

This paper explores zero-label learning in Natural Language Processing (NLP), whereby no human-annotated data is used anywhere during training and models are trained purely on synthetic data. At the core of our framework is a novel approach for better leveraging the powerful pretrained language models. Specifically, inspired by the recent success of few-shot inference on GPT-3, we present a training data creation procedure named Unsupervised Data Generation (UDG), which leverages few-shot prompts to synthesize high-quality training data without real human annotations. Our method enables zero-label learning as we train task-specific models solely on the synthetic data, yet we achieve better or comparable results from strong baseline models trained on human-labeled data. Furthermore, when mixed with labeled data, our approach serves as a highly effective data augmentation procedure, achieving new state-of-the-art results on the SuperGLUE benchmark.

Key facts. Cited by 76 papers in the map (data through 13 Oct 2026). Builds on 21 of its 24 references that are in the map. First posted on arXiv 19 Sep 2021.

Builds on

21 of 24 references.

References · 21

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,304
  2. 2020Language Models are Few-Shot Learnerscited by 32,320
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,549
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,322
  5. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,810
  6. 2015Character-level Convolutional Networks for Text Classificationcited by 2,319
  7. 2020Extracting Training Data from Large Language Modelscited by 2,181
  8. 2019Language Models as Knowledge Bases?cited by 1,931
  9. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,725
  10. 2019Unsupervised Data Augmentation for Consistency Trainingcited by 1,058
  11. 2019EDA: Easy Data Augmentation Techniques for Boosting Performance on Text Classification Taskscited by 802
  12. 2020How Much Knowledge Can You Pack Into the Parameters of a Language Model?cited by 717
9 more
  1. 2020It's Not Just Size That Matters: Small Language Models Are Also Few-Shot Learnerscited by 668
  2. 2020Towards a Human-like Open-Domain Chatbotcited by 469
  3. 2020Data Augmentation using Pre-trained Transformer Modelscited by 168
  4. 2018QANet: Combining Local Convolution with Global Self-Attention for Reading Comprehensioncited by 164
  5. 2021Improving and Simplifying Pattern Exploiting Trainingcited by 97
  6. 2020Language Models are Open Knowledge Graphscited by 95
  7. 2021Neural Data Augmentation via Example Extrapolationcited by 54
  8. 2020DARE: Data Augmented Relation Extraction with GPT-2cited by 43
  9. 2020A little goes a long way: Improving toxic language classification despite data scarcitycited by 12

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2021: 8; 2022: 27; 2023: 24; 2024: 11; 2025: 4; 2026: 2.

76 papers.

Sort:

By area:

  1. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,693
  2. 2021Ethical and social risks of harm from Language Modelscited by 964
  3. 2023RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedbackcited by 617
  4. 2022Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Modelcited by 559
  5. 2022A Holistic Approach to Undesired Content Detection in the Real Worldcited by 354
  6. 2024A Survey on Knowledge Distillation of Large Language Modelscited by 279
  7. 2022ZeroGen: Efficient Zero-shot Learning via Dataset Generationcited by 210
  8. 2022Generating Training Data with Language Models: Towards Zero-Shot Language Understandingcited by 199
  9. 2022Learning by Distilling Contextcited by 132
  10. 2023Synthetic Data Generation with Large Language Models for Text Classification: Potential and Limitationscited by 98
  11. 2023Exploiting Asymmetry for Synthetic Training Data Generation: SynthIE and the Case of Information Extractioncited by 89
  12. 2022AugESC: Dialogue Augmentation with Large Language Models for Emotional Support Conversationcited by 79
38 more
  1. 2023LLMaAA: Making Large Language Models as Active Annotatorscited by 78
  2. 2022ProGen: Progressive Zero-shot Dataset Generation via In-context Feedbackcited by 67
  3. 2023Leveraging Large Language Models in Conversational Recommender Systemscited by 63
  4. 2022Prototypical Calibration for Few-shot Learning of Language Modelscited by 58
  5. 2022PromDA: Prompt-based Data Augmentation for Low-Resource NLU Taskscited by 51
  6. 2022Self-Guided Noise-Free Data Generation for Efficient Zero-Shot Learningcited by 50
  7. 2023Faithful Persona-based Conversational Dataset Generation with Large Language Modelscited by 47
  8. 2022Model Cascading: Towards Jointly Improving Efficiency and Accuracy of NLP Systemscited by 44
  9. 2022Controllable Dialogue Simulation with In-Context Learningcited by 42
  10. 2022DoCoGen: Domain Counterfactual Generation for Low Resource Domain Adaptationcited by 36
  11. 2022Tuning Language Models as Training Data Generators for Augmentation-Enhanced Few-Shot Learningcited by 36
  12. 2024GoldCoin: Grounding Large Language Models in Privacy Laws via Contextual Integrity Theorycited by 36
  13. 2023Ask an Expert: Leveraging Language Models to Improve Strategic Reasoning in Goal-Oriented Dialogue Modelscited by 30
  14. 2022LINGUIST: Language Model Instruction Tuning to Generate Annotated Utterances for Intent Classification and Slot Taggingcited by 29
  15. 2022Geographic and Geopolitical Biases of Language Modelscited by 29
  16. 2023Making Large Language Models Better Data Creatorscited by 28
  17. 2023COKE: A Cognitive Knowledge Graph for Machine Theory of Mindcited by 26
  18. 2024TELEClass: Taxonomy Enrichment and LLM-Enhanced Hierarchical Text Classification with Minimal Supervisioncited by 25
  19. 2021Unsupervised Neural Machine Translation with Generative Language Models Onlycited by 24
  20. 2023Mixture of Soft Prompts for Controllable Data Generationcited by 23
  21. 2022CLASP: Few-Shot Cross-Lingual Data Augmentation for Semantic Parsingcited by 21
  22. 2023The Magic of IF: Investigating Causal Reasoning Abilities in Large Language Models of Codecited by 21
  23. 2022LAD: Language Models as Data for Zero-Shot Dialogcited by 18
  24. 2023Semantic-Guided Generative Image Augmentation Method with Diffusion Models for Image Classificationcited by 16
  25. 2022Transforming Sequence Tagging Into A Seq2Seq Taskcited by 15
  26. 2023TeacherLM: Teaching to Fish Rather Than Giving the Fish, Language Modeling Likewisecited by 15
  27. 2024Multimodal Large Language Model is a Human-Aligned Annotator for Text-to-Image Generationcited by 15
  28. 2022Predicting Fine-Tuning Performance with Probingcited by 13
  29. 2022BenchCLAMP: A Benchmark for Evaluating Language Models on Syntactic and Semantic Parsingcited by 12
  30. 2024Knowledge Distillation Using Frontier Open-source LLMs: Generalizability and the Role of Synthetic Datacited by 12
  31. 2021Smaaprat: DialoGPT for Natural Language Generation of Swedish Dialogue by Transfer Learningcited by 11
  32. 2024JOBSKAPE: A Framework for Generating Synthetic Job Postings to Enhance Skill Matchingcited by 11
  33. 2022Can Open-Domain QA Reader Utilize External Knowledge Efficiently like Humans?cited by 10
  34. 2024mCSQA: Multilingual Commonsense Reasoning Dataset with Unified Creation Strategy by Language Models and Humanscited by 10
  35. 2024ConvSDG: Session Data Generation for Conversational Searchcited by 8
  36. 2021Deep Transfer Learning & Beyond: Transformer Language Models in Information Systems Researchcited by 7
  37. 2021Teaching Models new APIs: Domain-Agnostic Simulators for Task Oriented Dialoguecited by 6
  38. 2021Unsupervised Natural Language Inference Using PHL Triplet Generationcited by 6

50 most cited of 76 All 76 (JSON)