mlsanityData to 7 Oct 2026

Generating Datasets with Pretrained Language Models

arXiv 2104.07540 ↗First posted 15 Apr 2021cs.CL cs.LGcited by 192

Not on the map. Why?

Abstract (arXiv)

To obtain high-quality sentence embeddings from pretrained language models (PLMs), they must either be augmented with additional pretraining objectives or finetuned on a large set of labeled text pairs. While the latter approach typically outperforms the former, it requires great human effort to generate suitable datasets of sufficient size. In this paper, we show how PLMs can be leveraged to obtain high-quality sentence embeddings without the need for labeled data, finetuning or modifications to the pretraining objective: We utilize the generative abilities of large and high-performing PLMs to generate entire datasets of labeled text pairs from scratch, which we then use for finetuning much smaller and more efficient models. Our fully unsupervised approach outperforms strong baselines on several semantic textual similarity datasets.

Key facts. Cited by 192 papers in the map (data through 7 Oct 2026). Builds on 39 of its 58 references that are in the map. First posted on arXiv 15 Apr 2021.

Builds on

39 of 58 references.

References · 39

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  5. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  6. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,451
  7. 2013Efficient Estimation of Word Representations in Vector Spacecited by 4,346
  8. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  9. 2019The Curious Case of Neural Text Degenerationcited by 2,881
  10. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
  11. 2018Deep contextualized word representationscited by 2,285
  12. 2015A large annotated corpus for learning natural language inferencecited by 1,837
27 more
  1. 2016Enriching Word Vectors with Subword Informationcited by 1,578
  2. 2020Making Pre-trained Language Models Better Few-shot Learnerscited by 1,255
  3. 2018Hierarchical Neural Story Generationcited by 1,182
  4. 2017SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluationcited by 998
  5. 2019How Can We Know What Language Models Know?cited by 953
  6. 2020Exploiting Cloze Questions for Few Shot Text Classification and Natural Language Inferencecited by 934
  7. 2014Distributed Representations of Sentences and Documentscited by 677
  8. 2020It's Not Just Size That Matters: Small Language Models Are Also Few-Shot Learnerscited by 668
  9. 2017Supervised Learning of Universal Sentence Representations from Natural Language Inference Datacited by 428
  10. 2021Self-Diagnosis and Self-Debiasing: A Proposal for Reducing Corpus-Based Bias in NLPcited by 336
  11. 2015Skip-Thought Vectorscited by 321
  12. 2020DeCLUTR: Deep Contrastive Learning for Unsupervised Textual Representationscited by 249
  13. 2020CLEAR: Contrastive Learning for Sentence Representationcited by 186
  14. 2019Not Enough Data? Deep Learning to the Rescue!cited by 174
  15. 2020Data Augmentation using Pre-trained Transformer Modelscited by 168
  16. 2018Learning to Write with Cooperative Discriminatorscited by 119
  17. 2021Improving and Simplifying Pattern Exploiting Trainingcited by 97
  18. 2020An Unsupervised Sentence Embedding Method by Mutual Information Maximizationcited by 91
  19. 2020On the Sentence Embeddings from Pre-trained Language Modelscited by 83
  20. 2020The Turking Test: Can Language Models Understand Instructions?cited by 82
  21. 2020Learning from Task Descriptionscited by 78
  22. 2019Zero-shot Text Classification With Generative Language Modelscited by 55
  23. 2020Generative Data Augmentation for Commonsense Reasoningcited by 49
  24. 2017ParaNMT-50M: Pushing the Limits of Paraphrastic Sentence Embeddings with Millions of Machine Translationscited by 44
  25. 2020DARE: Data Augmented Relation Extraction with GPT-2cited by 43
  26. 2019Sentence Meta-Embeddings for Unsupervised Semantic Textual Similaritycited by 8
  27. 2019Argumentative Relation Classification as Plausibility Rankingcited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 10; 2022: 43; 2023: 61; 2024: 51; 2025: 16; 2026: 11.

192 papers.

Sort:

By area:

  1. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,072
  2. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
  3. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,222
  4. 2023CAMEL: Communicative Agents for "Mind" Exploration of Large Language Model Societycited by 1,348
  5. 2022Discovering Language Model Behaviors with Model-Written Evaluationscited by 805
  6. 2021Recent Advances in Natural Language Processing via Large Pre-Trained Language Models: A Surveycited by 457
  7. 2022Unnatural Instructions: Tuning Language Models with (Almost) No Human Laborcited by 395
  8. 2023ToolAlpaca: Generalized Tool Learning for Language Models with 3000 Simulated Casescited by 383
  9. 2022A Holistic Approach to Undesired Content Detection in the Real Worldcited by 337
  10. 2023Improving Text Embeddings with Large Language Modelscited by 295
  11. 2024A Survey on Knowledge Distillation of Large Language Modelscited by 276
  12. 2021Symbolic Knowledge Distillation: from General Language Models to Commonsense Modelscited by 270
38 more
  1. 2023Large Language Model as Attributed Training Data Generator: A Tale of Diversity and Biascited by 230
  2. 2022ZeroGen: Efficient Zero-shot Learning via Dataset Generationcited by 209
  3. 2022SGPT: GPT Sentence Embeddings for Semantic Searchcited by 204
  4. 2022WANLI: Worker and AI Collaboration for Natural Language Inference Dataset Creationcited by 200
  5. 2022Generating Training Data with Language Models: Towards Zero-Shot Language Understandingcited by 199
  6. 2023Understanding Social Reasoning in Language Models with Language Modelscited by 166
  7. 2023Symbolic Chain-of-Thought Distillation: Small Models Can Also "Think" Step-by-Stepcited by 144
  8. 2023ReviewerGPT? An Exploratory Study on Using Large Language Models for Paper Reviewingcited by 118
  9. 2023DetGPT: Detect What You Need via Reasoningcited by 112
  10. 2023Exploiting Asymmetry for Synthetic Training Data Generation: SynthIE and the Case of Information Extractioncited by 89
  11. 2023Controlled Text Generation with Natural Language Instructionscited by 87
  12. 2022AugESC: Dialogue Augmentation with Large Language Models for Emotional Support Conversationcited by 79
  13. 2023LLMaAA: Making Large Language Models as Active Annotatorscited by 78
  14. 2021What Changes Can Large-scale Language Models Bring? Intensive Study on HyperCLOVA: Billions-scale Korean Generative Pretrained Transformerscited by 75
  15. 2022Language Models Can Teach Themselves to Program Bettercited by 75
  16. 2022Differentially Private Language Models for Secure Data Sharingcited by 68
  17. 2022ProGen: Progressive Zero-shot Dataset Generation via In-context Feedbackcited by 67
  18. 2022InPars: Data Augmentation for Information Retrieval using Large Language Modelscited by 60
  19. 2022Contrastive Search Is What You Need For Neural Text Generationcited by 58
  20. 2022Revisiting DocRED -- Addressing the False Negative Problem in Relation Extractioncited by 57
  21. 2023VITATECS: A Diagnostic Dataset for Temporal Concept Understanding of Video-Language Modelscited by 55
  22. 2022ZeroPrompt: Scaling Prompt-Based Pretraining to 1,000 Tasks Improves Zero-Shot Generalizationcited by 54
  23. 2022Generating Data to Mitigate Spurious Correlations in Natural Language Inference Datasetscited by 54
  24. 2022Language Models in the Loop: Incorporating Prompting into Weak Supervisioncited by 51
  25. 2023FlexKBQA: A Flexible LLM-Powered Framework for Few-Shot Knowledge Base Question Answeringcited by 51
  26. 2023Dr.Spider: A Diagnostic Evaluation Benchmark towards Text-to-SQL Robustnesscited by 50
  27. 2022Self-Guided Noise-Free Data Generation for Efficient Zero-Shot Learningcited by 49
  28. 2024Internal Consistency and Self-Feedback in Large Language Models: A Surveycited by 49
  29. 2024Forcing Diffuse Distributions out of Language Modelscited by 48
  30. 2023LongForm: Effective Instruction Tuning with Reverse Instructionscited by 47
  31. 2023Faithful Persona-based Conversational Dataset Generation with Large Language Modelscited by 45
  32. 2023Distributed Inference and Fine-tuning of Large Language Models Over The Internetcited by 42
  33. 2024CodecLM: Aligning Language Models with Tailored Synthetic Datacited by 39
  34. 2021True Few-Shot Learning with Prompts -- A Real-World Perspectivecited by 38
  35. 2022GPS: Genetic Prompt Search for Efficient Few-shot Learningcited by 37
  36. 2022Tuning Language Models as Training Data Generators for Augmentation-Enhanced Few-Shot Learningcited by 36
  37. 2024Operationalizing Contextual Integrity in Privacy-Conscious Assistantscited by 36
  38. 2024A Synthetic Dataset for Personal Attribute Inferencecited by 35

50 most cited of 192 All 192 (JSON)