# Source2Synth: Synthetic Data Generation and Curation Grounded in Real Data Sources

arXiv: https://arxiv.org/abs/2409.08239 · HTML: https://mlsanity.com/paper/2409.08239/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- Placed in Synthetic reasoning data and bootstrapping, one of 13 directions in Synthetic data and self-improvement for LLMs.
- Builds on 37 of its 49 references that are in the map.
- First posted on arXiv 12 Sep 2024.

## Placed in

- Synthetic data and self-improvement for LLMs / [Synthetic reasoning data and bootstrapping](https://mlsanity.com/synth/synthetic-reasoning-data-and-bootstrapping/)

## Builds on (most cited first, 30 of 37)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Llama 2: Open Foundation and Fine-Tuned Chat Models](https://mlsanity.com/paper/2307.09288/) (2023)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [PaLM: Scaling Language Modeling with Pathways](https://mlsanity.com/paper/2204.02311/) (2022)
- [Toolformer: Language Models Can Teach Themselves to Use Tools](https://mlsanity.com/paper/2302.04761/) (2023)
- [HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering](https://mlsanity.com/paper/1809.09600/) (2018)
- [Self-Instruct: Aligning Language Models with Self-Generated Instructions](https://mlsanity.com/paper/2212.10560/) (2022)
- [Language Models as Knowledge Bases?](https://mlsanity.com/paper/1909.01066/) (2019)
- [LaMDA: Language Models for Dialog Applications](https://mlsanity.com/paper/2201.08239/) (2022)
- [How Can We Know What Language Models Know?](https://mlsanity.com/paper/1911.12543/) (2019)
- [Exploiting Cloze Questions for Few Shot Text Classification and Natural Language Inference](https://mlsanity.com/paper/2001.07676/) (2020)
- [ChatGPT Outperforms Crowd-Workers for Text-Annotation Tasks](https://mlsanity.com/paper/2303.15056/) (2023)
- [Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Tasks](https://mlsanity.com/paper/2204.07705/) (2022)
- [Can LLM Already Serve as A Database Interface? A BIg Bench for Large-Scale Database Grounded Text-to-SQLs](https://mlsanity.com/paper/2305.03111/) (2023)
- [Seq2SQL: Generating Structured Queries from Natural Language using Reinforcement Learning](https://mlsanity.com/paper/1709.00103/) (2017)
- [Compositional Semantic Parsing on Semi-Structured Tables](https://mlsanity.com/paper/1508.00305/) (2015)
- [TAPAS: Weakly Supervised Table Parsing via Pre-training](https://mlsanity.com/paper/2004.02349/) (2020)
- [Augmented Language Models: a Survey](https://mlsanity.com/paper/2302.07842/) (2023)
- [TaBERT: Pretraining for Joint Understanding of Textual and Tabular Data](https://mlsanity.com/paper/2005.08314/) (2020)
- [Unnatural Instructions: Tuning Language Models with (Almost) No Human Labor](https://mlsanity.com/paper/2212.09689/) (2022)
- [ToolAlpaca: Generalized Tool Learning for Language Models with 3000 Simulated Cases](https://mlsanity.com/paper/2306.05301/) (2023)
- [Large Language Models as Tool Makers](https://mlsanity.com/paper/2305.17126/) (2023)
- [WANLI: Worker and AI Collaboration for Natural Language Inference Dataset Creation](https://mlsanity.com/paper/2201.05955/) (2022)
- [ART: Automatic multi-step reasoning and tool-use for large language models](https://mlsanity.com/paper/2303.09014/) (2023)
- [Generating Datasets with Pretrained Language Models](https://mlsanity.com/paper/2104.07540/) (2021)
- [Answering Complex Open-Domain Questions with Multi-Hop Dense Retrieval](https://mlsanity.com/paper/2009.12756/) (2020)
- [Self-Alignment with Instruction Backtranslation](https://mlsanity.com/paper/2308.06259/) (2023)
- [Few-Shot Text Generation with Pattern-Exploiting Training](https://mlsanity.com/paper/2012.11926/) (2020)

## Built on by (most cited first, 28 of 28)

- [Explore Theory of Mind: Program-guided adversarial data generation for theory of mind reasoning](https://mlsanity.com/paper/2412.12175/) (2024)
- [CoT-Self-Instruct: Building high-quality synthetic prompts for reasoning and non-reasoning tasks](https://mlsanity.com/paper/2507.23751/) (2025)
- [Compute as Teacher: Turning Inference Compute Into Reference-Free Supervision](https://mlsanity.com/paper/2509.14234/) (2025)
- [Autodata: An agentic data scientist to create high quality synthetic data](https://mlsanity.com/paper/2606.25996/) (2026)
- [MDCure: A Scalable Pipeline for Multi-Document Instruction-Following](https://mlsanity.com/paper/2410.23463/) (2024)
- [A Survey on Efficient Large Language Model Training: From Data-centric Perspectives](https://mlsanity.com/paper/2510.25817/) (2025)
- [SyncLoop: A Multimodal Dual-Loop Framework for Self-Improving Mathematical Reasoning](https://mlsanity.com/paper/2507.16518/) (2025)
- [Forewarned is Forearmed: Leveraging LLMs for Data Synthesis through Failure-Inducing Exploration](https://mlsanity.com/paper/2410.16736/) (2024)
- [Aligning Instruction Tuning with Pre-training](https://mlsanity.com/paper/2501.09368/) (2025)
- [TreeSynth: Synthesizing Diverse Data from Scratch via Tree-Guided Subspace Partitioning](https://mlsanity.com/paper/2503.17195/) (2025)
- [Generative Data Refinement: Just Ask for Better Data](https://mlsanity.com/paper/2509.08653/) (2025)
- [Escaping Model Collapse via Synthetic Data Verification: Near-term Improvements and Long-term Convergence](https://mlsanity.com/paper/2510.16657/) (2025)
- [Compound-QA: A Benchmark for Evaluating LLMs on Compound Questions](https://mlsanity.com/paper/2411.10163/) (2024)
- [Beyond Rephrasing: Book-Level Organization Improves Synthetic Textbook Data for Mid-Training](https://mlsanity.com/paper/2607.28109/) (2026)
- [HopWeaver: Cross-Document Synthesis of High-Quality and Authentic Multi-Hop Questions](https://mlsanity.com/paper/2505.15087/) (2025)
- [Is Your Model Fairly Certain? Uncertainty-Aware Fairness Evaluation for LLMs](https://mlsanity.com/paper/2505.23996/) (2025)
- [LLM Unlearning on Noisy Forget Sets: A Study of Incomplete, Rewritten, and Watermarked Data](https://mlsanity.com/paper/2510.09007/) (2025)
- [SQL-Exchange: Transforming SQL Queries Across Domains](https://mlsanity.com/paper/2508.07087/) (2025)
- [Semantic Bridge: Universal Multi-Hop Question Generation via AMR-Driven Graph Synthesis](https://mlsanity.com/paper/2508.10013/) (2025)
- [TuneShift-KD: Knowledge Distillation and Transfer for Fine-tuned Models](https://mlsanity.com/paper/2603.24518/) (2026)
- [Masked Diffusion Language Models are Strong and Steerable Text-Based World Models for Agentic RL](https://mlsanity.com/paper/2607.16204/) (2026)
- [Large Language Model's Multi-Capability Alignment in Biomedical Domain](https://mlsanity.com/paper/2508.04278/) (2025)
- [Entropy-Based Data Selection for Language Models](https://mlsanity.com/paper/2602.17465/) (2026)
- [Context-Instrumental Data Distillation for Kubernetes Manifest Generation: Method and Experimental Evaluation](https://mlsanity.com/paper/2605.25835/) (2026)
- [CORTEX: High-Quality Cross-Domain Organization of Web-Scale Corpora through Ontological Corpus Graph](https://mlsanity.com/paper/2606.30175/) (2026)
- [Secret Scanner Agent: Extracting Secrets and Access Context from Unstructured Documents](https://mlsanity.com/paper/2607.09011/) (2026)
- [A Multi-Agent Pipeline for Source-Grounded Synthetic Note Generation from Longitudinal Structured EHR](https://mlsanity.com/paper/2609.22164/) (2026)
- [FALCON: A Model and Dataset Agnostic Framework for Synthetic Data Generation for NL2SQL Pairs](https://mlsanity.com/paper/2610.03625/) (2026)

## Abstract (arXiv)

Synthetic data generation has recently emerged as a promising approach for enhancing the capabilities of large language models (LLMs) without the need for expensive human annotations. However, existing methods often generate data that can be low quality or contrived. In this paper, we introduce Source2Synth, a scalable approach for synthetic data generation and curation that is grounded in real-world data sources. Source2Synth takes as input a custom data source and produces synthetic data examples with intermediate reasoning steps. Our method improves the dataset quality by discarding low-quality generations based on their answerability. We demonstrate the generality of this approach by applying it to two tasks that leverage two different types of data: multi-hop question answering (MHQA), where we test complex reasoning abilities leveraging documents, and tabular question answering (TQA), where we test tool usage leveraging tables. Our method improves performance by 25.51% for TQA on WikiSQL and 22.57% for MHQA on HotpotQA compared to the fine-tuned baselines.