# MetaSynth: Meta-Prompting-Driven Agentic Scaffolds for Diverse Synthetic Data Generation

arXiv: https://arxiv.org/abs/2504.12563 · HTML: https://mlsanity.com/paper/2504.12563/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Placed in Web, search, and research agents, one of 16 directions in LLM agents and tool use.
- Builds on 17 of its 34 references that are in the map.
- First posted on arXiv 17 Apr 2025.

## Placed in

- LLM agents and tool use / [Web, search, and research agents](https://mlsanity.com/agents/web-search-and-research-agents/)
- Synthetic data and self-improvement for LLMs / [LLM synthetic data generation](https://mlsanity.com/synth/llm-synthetic-data-generation/)

## Builds on (most cited first, 17 of 17)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Enhancing Chat Language Models by Scaling High-quality Instructional Conversations](https://mlsanity.com/paper/2305.14233/) (2023)
- [TinyStories: How Small Can Language Models Be and Still Speak Coherent English?](https://mlsanity.com/paper/2305.07759/) (2023)
- [Scaling Synthetic Data Creation with 1,000,000,000 Personas](https://mlsanity.com/paper/2406.20094/) (2024)
- [The Curse of Recursion: Training on Generated Data Makes Models Forget](https://mlsanity.com/paper/2305.17493/) (2023)
- [ConvFinQA: Exploring the Chain of Numerical Reasoning in Conversational Finance Question Answering](https://mlsanity.com/paper/2210.03849/) (2022)
- [Will we run out of data? Limits of LLM scaling based on human-generated data](https://mlsanity.com/paper/2211.04325/) (2022)
- [Meta-Prompting: Enhancing Language Models with Task-Agnostic Scaffolding](https://mlsanity.com/paper/2401.12954/) (2024)
- [Concise Thoughts: Impact of Output Length on LLM Reasoning and Cost](https://mlsanity.com/paper/2407.19825/) (2024)
- [Impact of News on the Commodity Market: Dataset and Results](https://mlsanity.com/paper/2009.04202/) (2020)
- [Standardizing the Measurement of Text Diversity: A Tool and a Comparative Analysis of Scores](https://mlsanity.com/paper/2403.00553/) (2024)
- [Detection and Measurement of Syntactic Templates in Generated Text](https://mlsanity.com/paper/2407.00211/) (2024)
- [GenQA: Generating Millions of Instructions from a Handful of Prompts](https://mlsanity.com/paper/2406.10323/) (2024)
- [Synthesizing Post-Training Data for LLMs through Multi-Agent Simulation](https://mlsanity.com/paper/2410.14251/) (2024)
- [Demystifying Domain-adaptive Post-training for Financial LLMs](https://mlsanity.com/paper/2501.04961/) (2025)
- [Explore-Instruct: Enhancing Domain-Specific Instruction Coverage through Active Exploration](https://mlsanity.com/paper/2310.09168/) (2023)

## Built on by (most cited first, 10 of 10)

- [Cartridges: Lightweight and general-purpose long context representations via self-study](https://mlsanity.com/paper/2506.06266/) (2025)
- [Building a Foundational Guardrail for General Agentic Systems via Synthetic Data](https://mlsanity.com/paper/2510.09781/) (2025)
- [HunyuanOCR-1.5: Making Lightweight OCR VLMs Faster and Better](https://mlsanity.com/paper/2607.04884/) (2026)
- [LLM-Based Social Simulations Require a Boundary](https://mlsanity.com/paper/2506.19806/) (2025)
- [More Data or Better Data? A Critical Analysis of Data Selection and Synthesis for Mathematical Reasoning](https://mlsanity.com/paper/2510.07169/) (2025)
- [ChemOrch: Empowering LLMs with Chemical Intelligence via Synthetic Instructions](https://mlsanity.com/paper/2509.16543/) (2025)
- [Language Models Need Sleep: Learning to Self-Modify and Consolidate Memories](https://mlsanity.com/paper/2606.03979/) (2026)
- [PolyMicros: Bootstrapping a Foundation Model for Polycrystalline Material Structure](https://mlsanity.com/paper/2506.11055/) (2025)
- [Performance Trade-offs of Optimizing Small Language Models for E-Commerce](https://mlsanity.com/paper/2510.21970/) (2025)
- [SecProbe: Adaptive Evaluation of Coding Agents on Cybersecurity Vulnerabilities](https://mlsanity.com/paper/2609.33763/) (2026)

## Abstract (arXiv)

Recent smaller language models such Phi-3.5 and Phi-4 rely on synthetic data generated using larger Language models. Questions remain about leveraging synthetic data for other use cases, such as adapting LLMs to specific domains. A key limitation of synthetic data is low diversity, which negatively impacts its downstream applicability for improving other models. To address this, we propose MetaSynth, a method for generating synthetic data that enhances diversity through meta-prompting, where a language model orchestrates multiple "expert" LLM agents to collaboratively generate data. Using only 25 million tokens of synthetic data generated with MetaSynth, we successfully adapt a well-trained LLM (Mistral-7B-v0.3) to two specialized domains-Finance and Biomedicine-without compromising the capabilities of the resulting model in general tasks. In addition, we evaluate the diversity of our synthetic data using seven automated metrics, and find that it approaches the diversity of LLM pre-training corpora. Continually pre-training Mistral-7B-v0.3 with MetaSynth notably outperforms the base LLM, showing improvements of up to 4.08% in Finance and 13.75% in Biomedicine. The same model shows degraded performance when trained on data generated using a template prompt, even when the template includes prior generations and varying In-Context exemplars of real data. Our findings suggest that a few million tokens of diverse synthetic data without mixing any real data, is sufficient for effective domain adaptation when using MetaSynth.