mlsanityData to 7 Oct 2026

CRAFT Your Dataset: Task-Specific Synthetic Dataset Generation Through Corpus Retrieval and Augmentation

arXiv 2409.02098 ↗First posted 3 Sep 2024cs.CL cs.AI cs.LGcited by 14

On the map: Synthetic data and self-improvement for LLMs → LLM synthetic data generation

Abstract (arXiv)

Building high-quality datasets for specialized tasks is a time-consuming and resource-intensive process that often requires specialized domain knowledge. We propose Corpus Retrieval and Augmentation for Fine-Tuning (CRAFT), a method for generating synthetic datasets, given a small number of user-written few-shots that demonstrate the task to be performed. Given these examples, CRAFT uses large-scale public web-crawled corpora and similarity-based document retrieval to find other relevant human-written documents. Lastly, instruction-tuned large language models (LLMs) augment the retrieved documents into custom-formatted task samples, which then can be used for fine-tuning. We demonstrate that CRAFT can efficiently generate large-scale task-specific training datasets for four diverse tasks: biology, medicine, and commonsense question-answering (QA), as well as summarization. Our experiments show that CRAFT-based models outperform or match general LLMs on QA tasks, while exceeding models trained on human-curated summarization data by 46 preference points. CRAFT outperforms other synthetic dataset generation methods such as Self- and Evol-Instruct, and remains robust even when the quality of the initial few-shots varies.

Key facts. Cited by 14 papers in the map (data through 7 Oct 2026). Placed in LLM synthetic data generation, one of 13 directions in Synthetic data and self-improvement for LLMs. Builds on 59 of its 100 references that are in the map. First posted on arXiv 3 Sep 2024.

Builds on

59 of 100 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024Better Synthetic Data by Retrieving and Transforming Existing Datasets22 Apr

Other references · 58

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  3. 2022Training language models to follow instructions with human feedbackcited by 15,712
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2024The Llama 3 Herd of Modelscited by 14,192
  6. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  7. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  8. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  9. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  10. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  11. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  12. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,383
38 more
  1. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  2. 2023Mistral 7Bcited by 3,065
  3. 2019The Curious Case of Neural Text Degenerationcited by 2,881
  4. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
  5. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,205
  6. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,881
  7. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,462
  8. 2020MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformerscited by 1,414
  9. 2017Get To The Point: Summarization with Pointer-Generator Networkscited by 1,281
  10. 2018Hierarchical Neural Story Generationcited by 1,182
  11. 2024Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preferencecited by 1,151
  12. 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,023
  13. 2022Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learningcited by 886
  14. 2023AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedbackcited by 738
  15. 2022A Survey on In-context Learningcited by 717
  16. 2016TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systemscited by 660
  17. 2020Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuningcited by 658
  18. 2023Textbooks Are All You Needcited by 561
  19. 2022MedMCQA : A Large-scale Multi-Subject Multi-Choice Dataset for Medical domain Question Answeringcited by 525
  20. 2018Scalable agent alignment via reward modeling: a research directioncited by 438
  21. 2023TinyStories: How Small Can Language Models Be and Still Speak Coherent English?cited by 425
  22. 2022Unnatural Instructions: Tuning Language Models with (Almost) No Human Laborcited by 395
  23. 2023Orca: Progressive Learning from Complex Explanation Traces of GPT-4cited by 369
  24. 2018Measuring the Intrinsic Dimension of Objective Landscapescited by 364
  25. 2021True Few-Shot Learning with Language Modelscited by 343
  26. 2020Evaluation of Text Generation: A Surveycited by 275
  27. 2018WikiHow: A Large Scale Text Summarization Datasetcited by 203
  28. 2021PAQ: 65 Million Probably-Asked Questions and What You Can Do With Themcited by 198
  29. 2022A Comprehensive Survey of Few-shot Learning: Evolution, Applications, Challenges, and Opportunitiescited by 162
  30. 2023Orca 2: Teaching Small Language Models How to Reasoncited by 157
  31. 2024Rephrasing the Web: A Recipe for Compute and Data-Efficient Language Modelingcited by 156
  32. 2022CommonsenseQA 2.0: Exposing the Limits of AI through Gamificationcited by 108
  33. 2024AgentInstruct: Toward Generative Teaching with Agentic Flowscited by 88
  34. 2022Dialog Inpainting: Turning Documents into Dialogscited by 56
  35. 2023Evaluation Metrics in the Era of GPT-4: Reliably Evaluating Large Language Models on Sequence to Sequence Taskscited by 53
  36. 2023Dynosaur: A Dynamic Growth Paradigm for Instruction-Tuning Data Curationcited by 38
  37. 2019Corpus Wide Argument Mining -- a Working Solutioncited by 36
  38. 2023Chain of Natural Language Inference for Reducing Large Language Model Ungrounded Hallucinationscited by 34

50 most cited of 58 All 58 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 8; 2026: 5.

14 papers.

Sort:

By area:

  1. 2025A Survey of Scientific Large Language Models: From Data Foundations to Agent Frontierscited by 34
  2. 2024Source2Synth: Synthetic Data Generation and Curation Grounded in Real Data Sourcescited by 27
  3. 2025CYCLE-INSTRUCT: Fully Seed-Free Instruction Tuning via Dual Self-Training and Cycle Consistencycited by 6
  4. 2025ImageChain: Advancing Sequential Image-to-Text Reasoning in Multimodal Large Language Modelscited by 4
  5. 2025AQuilt: Weaving Logic and Self-Inspection into Low-Cost, High-Relevance Data Synthesis for Specialist LLMscited by 4
  6. 2025More Data or Better Data? A Critical Analysis of Data Selection and Synthesis for Mathematical Reasoningcited by 4
  7. 2025ELTEX: A Framework for Domain-Driven Synthetic Data Generationcited by 2
  8. 2025Integrating Personality into Digital Humans: A Review of LLM-Driven Approaches for Virtual Realitycited by 2
  9. 2025Learning from Reasoning Failures via Synthetic Data Generationcited by 2
  10. 2026TuneShift-KD: Knowledge Distillation and Transfer for Fine-tuned Modelscited by 1
  11. 2026VERSE: Visual Embedding Reduction and Space Exploration. Clustering-Guided Insights for Training Data Enhancement in Visually-Rich Document Understandingcited by 0
  12. 2026Is Micro Domain-Adaptive Pre-Training Effective for Real-World Operations? Multi-Step Evaluation Reveals Potential and Bottleneckscited by 0
2 more
  1. 2026One Prompt Does Not Fit All: Self-Meta-Evolve for Personalized Information Extractioncited by 0
  2. 2026Mapping the RAG Landscape: A Four Axis Taxonomy of Efficiency, Defense, Interactivity, and Reasoningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Synthetic Data Generation Using Large Language Models: Advances in Text and Code
  2. 2023Synthetic Data Generation in Low-Resource Settings via Fine-Tuning of Large Language Models
  3. 2024Balancing Cost and Effectiveness of Synthetic Data Generation Strategies for LLMs
  4. 2023Synthetic Data Generation with Large Language Models for Text Classification: Potential and Limitations
  5. 2024Prompting-based Synthetic Data Generation for Few-Shot Question Answering
  6. 2025Synthesize-on-Graph: Knowledgeable Synthetic Data Generation for Continue Pre-training of Large Language Models
  7. 2024DataGen: Unified Synthetic Dataset Generation via Large Language Models
  8. 2025Scaling Laws of Synthetic Data for Language Models