mlsanityData to 7 Oct 2026

Source2Synth: Synthetic Data Generation and Curation Grounded in Real Data Sources

arXiv 2409.08239 ↗First posted 12 Sep 2024cs.CL cs.AIcited by 27

On the map: Synthetic data and self-improvement for LLMs → Synthetic reasoning data and bootstrapping

Abstract (arXiv)

Synthetic data generation has recently emerged as a promising approach for enhancing the capabilities of large language models (LLMs) without the need for expensive human annotations. However, existing methods often generate data that can be low quality or contrived. In this paper, we introduce Source2Synth, a scalable approach for synthetic data generation and curation that is grounded in real-world data sources. Source2Synth takes as input a custom data source and produces synthetic data examples with intermediate reasoning steps. Our method improves the dataset quality by discarding low-quality generations based on their answerability. We demonstrate the generality of this approach by applying it to two tasks that leverage two different types of data: multi-hop question answering (MHQA), where we test complex reasoning abilities leveraging documents, and tabular question answering (TQA), where we test tool usage leveraging tables. Our method improves performance by 25.51% for TQA on WikiSQL and 22.57% for MHQA on HotpotQA compared to the fine-tuned baselines.

Key facts. Cited by 27 papers in the map (data through 7 Oct 2026). Placed in Synthetic reasoning data and bootstrapping, one of 13 directions in Synthetic data and self-improvement for LLMs. Builds on 37 of its 49 references that are in the map. First posted on arXiv 12 Sep 2024.

Builds on

37 of 49 references.

References · 37

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  5. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  6. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  7. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,072
  8. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,686
  9. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
  10. 2019Language Models as Knowledge Bases?cited by 1,922
  11. 2022LaMDA: Language Models for Dialog Applicationscited by 1,218
  12. 2019How Can We Know What Language Models Know?cited by 953
25 more
  1. 2020Exploiting Cloze Questions for Few Shot Text Classification and Natural Language Inferencecited by 934
  2. 2022Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Taskscited by 858
  3. 2023ChatGPT Outperforms Crowd-Workers for Text-Annotation Taskscited by 849
  4. 2017Seq2SQL: Generating Structured Queries from Natural Language using Reinforcement Learningcited by 679
  5. 2023Can LLM Already Serve as A Database Interface? A BIg Bench for Large-Scale Database Grounded Text-to-SQLscited by 667
  6. 2015Compositional Semantic Parsing on Semi-Structured Tablescited by 509
  7. 2020TAPAS: Weakly Supervised Table Parsing via Pre-trainingcited by 432
  8. 2023Augmented Language Models: a Surveycited by 398
  9. 2020TaBERT: Pretraining for Joint Understanding of Textual and Tabular Datacited by 395
  10. 2022Unnatural Instructions: Tuning Language Models with (Almost) No Human Laborcited by 395
  11. 2023ToolAlpaca: Generalized Tool Learning for Language Models with 3000 Simulated Casescited by 383
  12. 2023Large Language Models as Tool Makerscited by 293
  13. 2022WANLI: Worker and AI Collaboration for Natural Language Inference Dataset Creationcited by 200
  14. 2023ART: Automatic multi-step reasoning and tool-use for large language modelscited by 193
  15. 2021Generating Datasets with Pretrained Language Modelscited by 192
  16. 2020Answering Complex Open-Domain Questions with Multi-Hop Dense Retrievalcited by 160
  17. 2023Self-Alignment with Instruction Backtranslationcited by 149
  18. 2020Few-Shot Text Generation with Pattern-Exploiting Trainingcited by 87
  19. 2023Dynosaur: A Dynamic Growth Paradigm for Instruction-Tuning Data Curationcited by 38
  20. 2024TOOLVERIFIER: Generalization to New Tools via Self-Verificationcited by 23
  21. 2024Better Alignment with Instruction Back-and-Forth Translationcited by 16
  22. 2024What are the Essential Factors in Crafting Effective Long Context Multi-Hop Instruction Datasets? Insights and Best Practicescited by 14
  23. 2024CRAFT Your Dataset: Task-Specific Synthetic Dataset Generation Through Corpus Retrieval and Augmentationcited by 14
  24. 2023Generate, Transform, Answer: Question Specific Tool Synthesis for Tabular Datacited by 11
  25. 2024Towards Faithful and Robust LLM Specialists for Evidence-Based Question-Answeringcited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 4; 2025: 14; 2026: 9.

27 papers.

Sort:

By area:

  1. 2024Explore Theory of Mind: Program-guided adversarial data generation for theory of mind reasoningcited by 30
  2. 2025CoT-Self-Instruct: Building high-quality synthetic prompts for reasoning and non-reasoning taskscited by 30
  3. 2025Compute as Teacher: Turning Inference Compute Into Reference-Free Supervisioncited by 17
  4. 2026Autodata: An agentic data scientist to create high quality synthetic datacited by 12
  5. 2024MDCure: A Scalable Pipeline for Multi-Document Instruction-Followingcited by 10
  6. 2025A Survey on Efficient Large Language Model Training: From Data-centric Perspectivescited by 9
  7. 2025SyncLoop: A Multimodal Dual-Loop Framework for Self-Improving Mathematical Reasoningcited by 8
  8. 2024Forewarned is Forearmed: Leveraging LLMs for Data Synthesis through Failure-Inducing Explorationcited by 7
  9. 2025Aligning Instruction Tuning with Pre-trainingcited by 7
  10. 2025TreeSynth: Synthesizing Diverse Data from Scratch via Tree-Guided Subspace Partitioningcited by 6
  11. 2025Generative Data Refinement: Just Ask for Better Datacited by 6
  12. 2025Escaping Model Collapse via Synthetic Data Verification: Near-term Improvements and Long-term Convergencecited by 5
15 more
  1. 2024Compound-QA: A Benchmark for Evaluating LLMs on Compound Questionscited by 4
  2. 2025HopWeaver: Cross-Document Synthesis of High-Quality and Authentic Multi-Hop Questionscited by 2
  3. 2025Is Your Model Fairly Certain? Uncertainty-Aware Fairness Evaluation for LLMscited by 2
  4. 2026Beyond Rephrasing: Book-Level Organization Improves Synthetic Textbook Data for Mid-Trainingcited by 2
  5. 2025SQL-Exchange: Transforming SQL Queries Across Domainscited by 1
  6. 2025Semantic Bridge: Universal Multi-Hop Question Generation via AMR-Driven Graph Synthesiscited by 1
  7. 2025LLM Unlearning on Noisy Forget Sets: A Study of Incomplete, Rewritten, and Watermarked Datacited by 1
  8. 2026TuneShift-KD: Knowledge Distillation and Transfer for Fine-tuned Modelscited by 1
  9. 2026Masked Diffusion Language Models are Strong and Steerable Text-Based World Models for Agentic RLcited by 1
  10. 2025Large Language Model's Multi-Capability Alignment in Biomedical Domaincited by 0
  11. 2026Entropy-Based Data Selection for Language Modelscited by 0
  12. 2026Context-Instrumental Data Distillation for Kubernetes Manifest Generation: Method and Experimental Evaluationcited by 0
  13. 2026CORTEX: High-Quality Cross-Domain Organization of Web-Scale Corpora through Ontological Corpus Graphcited by 0
  14. 2026Secret Scanner Agent: Extracting Secrets and Access Context from Unstructured Documentscited by 0
  15. 2026A Multi-Agent Pipeline for Source-Grounded Synthetic Note Generation from Longitudinal Structured EHRcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024TARGA: Targeted Synthetic Data Generation for Practical Reasoning over Structured Data
  2. 2025Synthetic Data Generation Using Large Language Models: Advances in Text and Code
  3. 2024Balancing Cost and Effectiveness of Synthetic Data Generation Strategies for LLMs
  4. 2025Synthesize-on-Graph: Knowledgeable Synthetic Data Generation for Continue Pre-training of Large Language Models
  5. 2026ORACLE: Optimizing Reasoning Abilities of Large Language Models via Constraint-Led Synthetic Data Elicitation
  6. 2025SING-SQL: A Synthetic Data Generation Framework for In-Domain Text-to-SQL Translation
  7. 2026Learning from Synthetic Data Improves Multi-hop Reasoning
  8. 2024Prompting-based Synthetic Data Generation for Few-Shot Question Answering