mlsanityData to 7 Oct 2026

Better Synthetic Data by Retrieving and Transforming Existing Datasets

arXiv 2404.14361 ↗First posted 22 Apr 2024cs.CLcited by 36

On the map: Synthetic data and self-improvement for LLMs → LLM synthetic data generation

Abstract (arXiv)

Despite recent advances in large language models, building dependable and deployable NLP models typically requires abundant, high-quality training data. However, task-specific data is not available for many use cases, and manually curating task-specific data is labor-intensive. Recent work has studied prompt-driven synthetic data generation using large language models, but these generated datasets tend to lack complexity and diversity. To address these limitations, we introduce a method, DataTune, to make better use of existing, publicly available datasets to improve automatic dataset generation. DataTune performs dataset transformation, enabling the repurposing of publicly available datasets into a format that is directly aligned with the specific requirements of target tasks. On a diverse set of language-based tasks from the BIG-Bench benchmark, we find that finetuning language models via DataTune improves over a few-shot prompting baseline by 49% and improves over existing methods that use synthetic or retrieved training data by 34%. We find that dataset transformation significantly increases the diversity and difficulty of generated data on many tasks. We integrate DataTune into an open-source repository to make this method accessible to the community: https://github.com/neulab/prompt2model.

Key facts. Cited by 36 papers in the map (data through 7 Oct 2026). Placed in LLM synthetic data generation, one of 13 directions in Synthetic data and self-improvement for LLMs. Builds on 21 of its 27 references that are in the map. First posted on arXiv 22 Apr 2024.

Builds on

21 of 27 references.

References · 21

  1. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  2. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  3. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  4. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  5. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
  6. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,970
  7. 2017Decoupled Weight Decay Regularizationcited by 1,722
  8. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,462
  9. 2023WizardLM: Empowering large pre-trained language models to follow complex instructionscited by 1,116
  10. 2023Quantifying Language Models' Sensitivity to Spurious Features in Prompt Design or: How I learned to start worrying about prompt formattingcited by 778
  11. 2021Datasets: A Community Library for Natural Language Processingcited by 526
  12. 2023Is ChatGPT Good at Search? Investigating Large Language Models as Re-Ranking Agentscited by 356
9 more
  1. 2020MiniLMv2: Multi-Head Self-Attention Relation Distillation for Compressing Pretrained Transformerscited by 240
  2. 2022Is GPT-3 a Good Data Annotator?cited by 208
  3. 2023Does Synthetic Data Generation of LLMs Help Clinical Text Mining?cited by 151
  4. 2023Few-shot Fine-tuning vs. In-context Learning: A Fair Comparison and Evaluationcited by 134
  5. 2020Exploring and Predicting Transferability across NLP Taskscited by 126
  6. 2022ProGen: Progressive Zero-shot Dataset Generation via In-context Feedbackcited by 67
  7. 2023Prompt2Model: Generating Deployable Models from Natural Language Instructionscited by 40
  8. 2023Incorporating Worker Perspectives into MTurk Annotation Practices for NLPcited by 17
  9. 2023DataFinder: Scientific Dataset Recommendation from Natural Language Descriptionscited by 15

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 17; 2025: 16; 2026: 3.

36 papers.

Sort:

By area:

  1. 2024Magpie: Alignment Data Synthesis from Scratch by Prompting Aligned LLMs with Nothingcited by 341
  2. 2024On LLMs-Driven Synthetic Data Generation, Curation, and Evaluation: A Surveycited by 240
  3. 2025Cartridges: Lightweight and general-purpose long context representations via self-studycited by 74
  4. 2025Self-Adapting Language Modelscited by 71
  5. 2024JiuZhang3.0: Efficiently Improving Mathematical Reasoning by Training Small Data Synthesis Modelscited by 44
  6. 2024A Survey on Data Synthesis and Augmentation for Large Language Modelscited by 40
  7. 2024Surveying the Effects of Quality, Diversity, and Complexity in Synthetic Data From Large Language Modelscited by 31
  8. 2025InSTA: Towards Internet-Scale Training For Agentscited by 31
  9. 2025CoT-Self-Instruct: Building high-quality synthetic prompts for reasoning and non-reasoning taskscited by 30
  10. 2024AutoDCWorkflow: LLM-based Data Cleaning Workflow Auto-Generation and Benchmarkcited by 23
  11. 2024DataGen: Unified Synthetic Dataset Generation via Large Language Modelscited by 22
  12. 2024Deep Learning based Visually Rich Document Content Understanding: A Surveycited by 21
24 more
  1. 2024CRAFT Your Dataset: Task-Specific Synthetic Dataset Generation Through Corpus Retrieval and Augmentationcited by 14
  2. 2024SynthesizRR: Generating Diverse Datasets with Retrieval Augmentationcited by 13
  3. 2025LLMs can be easily Confused by Instructional Distractionscited by 12
  4. 2024Boosting Reward Model with Preference-Conditional Multi-Aspect Synthetic Data Generationcited by 9
  5. 2025DatasetResearch: Benchmarking Agent Systems for Demand-Driven Dataset Discoverycited by 9
  6. 2024Hybrid Training Approaches for LLMs: Leveraging Real and Synthetic Data to Enhance Model Performance in Domain-Specific Applicationscited by 7
  7. 2024ChainBuddy: An AI Agent System for Generating LLM Pipelinescited by 6
  8. 2025LIFT: A Novel Framework for Enhancing Long-Context Understanding of LLMs via Long Input Fine-Tuningcited by 6
  9. 2024Training Task Experts through Retrieval Based Distillationcited by 3
  10. 2024Mastering the Craft of Data Synthesis for CodeLLMscited by 3
  11. 2025InCo-DPO: Balancing Distribution Shift and Data Quality for Enhanced Preference Optimizationcited by 3
  12. 2024AIDE: Attribute-Guided MultI-Hop Data Expansion for Data Scarcity in Task-Specific Fine-tuningcited by 2
  13. 2025We Need to Measure Data Diversity in NLP -- Better and Broadercited by 2
  14. 2025Hierarchical Level-Wise News Article Clustering via Multilingual Matryoshka Embeddingscited by 2
  15. 2025Rethinking the Role of Text Complexity in Language Model Pretrainingcited by 2
  16. 2025Concept Attractors in LLMs and their Applicationscited by 1
  17. 2024On the Necessity of World Knowledge for Mitigating Missing Labels in Extreme Classificationcited by 0
  18. 2025Step-Opt: Boosting Optimization Modeling in LLMs through Iterative Data Synthesis and Structured Validationcited by 0
  19. 2025SYNAPSE-G: Bridging Large Language Models and Graph Learning for Rare Event Classificationcited by 0
  20. 2025TopoAlign: A Framework for Aligning Code to Math via Topological Decompositioncited by 0
  21. 2025LOCUS: A System and Method for Low-Cost Customization for Universal Specializationcited by 0
  22. 2026Entropy-Based Data Selection for Language Modelscited by 0
  23. 2026PersonaTrace: Synthesizing Realistic Digital Footprints with LLM Agentscited by 0
  24. 2026ROBE: Reversed-Order-Biased-Experts for Extracting Extreme Long-tail Events from Historical Textscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Multi-Sample Prompting and Actor-Critic Prompt Optimization for Diverse Synthetic Data Generation
  2. 2026How Can We Synthesize High-Quality Pretraining Data? A Systematic Study of Prompt Design, Generator Model, and Source Data
  3. 2023TarGEN: Targeted Data Generation with Large Language Models
  4. 2025BARE: Leveraging Base Language Models for Few-Shot Synthetic Data Generation
  5. 2024Evaluating Language Models as Synthetic Data Generators
  6. 2025Synthetic Data Generation Using Large Language Models: Advances in Text and Code
  7. 2024Efficacy of Synthetic Data as a Benchmark
  8. 2026Want Better Synthetic Data? Steer It: Activation Steering for Low-Resource Language Generation