# Synthesizing Post-Training Data for LLMs through Multi-Agent Simulation

arXiv: https://arxiv.org/abs/2410.14251 · HTML: https://mlsanity.com/paper/2410.14251/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- Placed in General LLM agent frameworks and benchmarks, one of 16 directions in LLM agents and tool use.
- Builds on 48 of its 69 references that are in the map.
- First posted on arXiv 18 Oct 2024.

## Placed in

- LLM agents and tool use / [General LLM agent frameworks and benchmarks](https://mlsanity.com/agents/general-llm-agent-frameworks-and-benchmarks/)

## Builds on (most cited first, 30 of 48)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [Measuring Mathematical Problem Solving With the MATH Dataset](https://mlsanity.com/paper/2103.03874/) (2021)
- [Let's Verify Step by Step](https://mlsanity.com/paper/2305.20050/) (2023)
- [Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2204.05862/) (2022)
- [Program Synthesis with Large Language Models](https://mlsanity.com/paper/2108.07732/) (2021)
- [Generative Agents: Interactive Simulacra of Human Behavior](https://mlsanity.com/paper/2304.03442/) (2023)
- [Universal and Transferable Adversarial Attacks on Aligned Language Models](https://mlsanity.com/paper/2307.15043/) (2023)
- [GPQA: A Graduate-Level Google-Proof Q&A Benchmark](https://mlsanity.com/paper/2311.12022/) (2023)
- [Self-Instruct: Aligning Language Models with Self-Generated Instructions](https://mlsanity.com/paper/2212.10560/) (2022)
- [Voyager: An Open-Ended Embodied Agent with Large Language Models](https://mlsanity.com/paper/2305.16291/) (2023)
- [Qwen2 Technical Report](https://mlsanity.com/paper/2407.10671/) (2024)
- [CAMEL: Communicative Agents for "Mind" Exploration of Large Language Model Society](https://mlsanity.com/paper/2303.17760/) (2023)
- [WizardLM: Empowering large pre-trained language models to follow complex instructions](https://mlsanity.com/paper/2304.12244/) (2023)
- [LIMA: Less Is More for Alignment](https://mlsanity.com/paper/2305.11206/) (2023)
- [Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluators](https://mlsanity.com/paper/2404.04475/) (2024)
- [Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Tasks](https://mlsanity.com/paper/2204.07705/) (2022)
- [BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Dataset](https://mlsanity.com/paper/2307.04657/) (2023)
- [Enhancing Chat Language Models by Scaling High-quality Instructional Conversations](https://mlsanity.com/paper/2305.14233/) (2023)
- [OpenAssistant Conversations -- Democratizing Large Language Model Alignment](https://mlsanity.com/paper/2304.07327/) (2023)
- [Cross-Task Generalization via Natural Language Crowdsourcing Instructions](https://mlsanity.com/paper/2104.08773/) (2021)
- [WildChat: 1M ChatGPT Interaction Logs in the Wild](https://mlsanity.com/paper/2405.01470/) (2024)
- [Using Large Language Models to Simulate Multiple Humans and Replicate Human Subject Studies](https://mlsanity.com/paper/2208.10264/) (2022)
- [From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipeline](https://mlsanity.com/paper/2406.11939/) (2024)
- [Scaling Synthetic Data Creation with 1,000,000,000 Personas](https://mlsanity.com/paper/2406.20094/) (2024)

## Built on by (most cited first, 24 of 24)

- [APIGen-MT: Agentic Pipeline for Multi-Turn Data Generation via Simulated Agent-Human Interplay](https://mlsanity.com/paper/2504.03601/) (2025)
- [From Individual to Society: A Survey on Social Simulation Driven by Large Language Model-based Agents](https://mlsanity.com/paper/2412.03563/) (2024)
- [LoopTool: Closing the Data-Training Loop for Robust LLM Tool Calls](https://mlsanity.com/paper/2511.09148/) (2025)
- [MASLab: A Unified and Comprehensive Codebase for LLM-based Multi-Agent Systems](https://mlsanity.com/paper/2505.16988/) (2025)
- [MetaSynth: Meta-Prompting-Driven Agentic Scaffolds for Diverse Synthetic Data Generation](https://mlsanity.com/paper/2504.12563/) (2025)
- [Behavior Injection: Preparing Language Models for Reinforcement Learning](https://mlsanity.com/paper/2505.18917/) (2025)
- [Towards On-Policy Data Evolution for Visual-Native Multimodal Deep Search Agents](https://mlsanity.com/paper/2605.10832/) (2026)
- [XtraGPT: Context-Aware and Controllable Academic Paper Revision via Human-AI Collaboration](https://mlsanity.com/paper/2505.11336/) (2025)
- [On the Importance of Task Complexity in Evaluating LLM-Based Multi-Agent Systems](https://mlsanity.com/paper/2510.04311/) (2025)
- [Combatting Dimensional Collapse in LLM Pre-Training Data via Diversified File Selection](https://mlsanity.com/paper/2504.20644/) (2025)
- [Beyond Quantity: Trajectory Diversity Scaling for Code Agents](https://mlsanity.com/paper/2602.03219/) (2026)
- [TopoCurate:Modeling Interaction Topology for Tool-Use Agent Training](https://mlsanity.com/paper/2603.01714/) (2026)
- [Bohdi: Heterogeneous LLM Fusion with Automatic Data Exploration](https://mlsanity.com/paper/2506.15721/) (2025)
- [Beyond Syntax: Action Semantics Learning for App Agents](https://mlsanity.com/paper/2506.17697/) (2025)
- [FURINA: A Fully Customizable Role-Playing Benchmark via Scalable Multi-Agent Collaboration Pipeline](https://mlsanity.com/paper/2510.06800/) (2025)
- [EmboMatrix: A Scalable Training-Ground for Embodied Decision-Making](https://mlsanity.com/paper/2510.12072/) (2025)
- [Refine-n-Judge: Curating High-Quality Preference Chains for LLM-Fine-Tuning](https://mlsanity.com/paper/2508.01543/) (2025)
- [Beyond Query-Level Comparison: Fine-Grained Reinforcement Learning for Text-to-SQL with Automated Interpretable Critiques](https://mlsanity.com/paper/2511.22258/) (2025)
- [MathAgent: Adversarial Evolution of Constraint Graphs for Mathematical Reasoning Data Synthesis](https://mlsanity.com/paper/2604.11188/) (2026)
- [Declarative by Design, Assistable Only by Convention: Benchmarking Multi-Agent Frameworks for AI-Assistability](https://mlsanity.com/paper/2602.11198/) (2026)
- [Know When To Fold 'Em: Token-Efficient LLM Synthetic Data Generation via Multi-Stage In-Flight Rejection](https://mlsanity.com/paper/2605.14062/) (2026)
- [Better Literary Translation: A Multi-Aspect Data Generation and LLM Training Approach](https://mlsanity.com/paper/2606.05924/) (2026)
- [Playing social deduction games with reinforcement fine-tuned large language models](https://mlsanity.com/paper/2610.04261/) (2026)
- [Know What You Don't Know: Agentic Data Synthesis by Evolving Generation Space for Closed-Loop Learning](https://mlsanity.com/paper/2610.12593/) (2026)

## Abstract (arXiv)

Post-training is essential for enabling large language models (LLMs) to follow human instructions. However, its effectiveness depends on high-quality instruction data, which is challenging to obtain in the real world due to privacy concerns, data scarcity, and high annotation costs. To fill this gap, inspired by the recent success of using LLMs to simulate human society, we propose MATRIX, a multi-agent simulator that automatically generates diverse text-based scenarios, capturing a wide range of real-world human needs in a realistic and scalable manner. Leveraging these outputs, we introduce a novel scenario-driven instruction generator MATRIX-Gen for controllable and highly realistic data synthesis. Extensive experiments demonstrate that our framework effectively generates both general and domain-specific data. On AlpacaEval 2 and Arena-Hard benchmarks, Llama-3-8B-Base, post-trained on datasets synthesized by MATRIX-Gen with just 20K instruction-response pairs, outperforms Meta's Llama-3-8B-Instruct model, which was trained on over 10M pairs.