# From Self-Evolving Synthetic Data to Verifiable-Reward RL: Post-Training Multi-turn Interactive Tool-Using Agents

arXiv: https://arxiv.org/abs/2601.22607 · HTML: https://mlsanity.com/paper/2601.22607/

## Key facts

- Cited by 17 papers in the map (data through 7 Oct 2026).
- Builds on 26 of its 35 references that are in the map.
- First posted on arXiv 30 Jan 2026.

## Builds on (most cited first, 26 of 26)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [ReAct: Synergizing Reasoning and Acting in Language Models](https://mlsanity.com/paper/2210.03629/) (2022)
- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [Toolformer: Language Models Can Teach Themselves to Use Tools](https://mlsanity.com/paper/2302.04761/) (2023)
- [Self-Instruct: Aligning Language Models with Self-Generated Instructions](https://mlsanity.com/paper/2212.10560/) (2022)
- [ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIs](https://mlsanity.com/paper/2307.16789/) (2023)
- [Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learning](https://mlsanity.com/paper/2503.09516/) (2025)
- [τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains](https://mlsanity.com/paper/2406.12045/) (2024)
- [WizardLM: Empowering large pre-trained language models to follow complex instructions](https://mlsanity.com/paper/2304.12244/) (2023)
- [DeepSeek-V3.2: Pushing the Frontier of Open Large Language Models](https://mlsanity.com/paper/2512.02556/) (2025)
- [τ2-Bench: Evaluating Conversational Agents in a Dual-Control Environment](https://mlsanity.com/paper/2506.07982/) (2025)
- [Magpie: Alignment Data Synthesis from Scratch by Prompting Aligned LLMs with Nothing](https://mlsanity.com/paper/2406.08464/) (2024)
- [AReaL: A Large-Scale Asynchronous Reinforcement Learning System for Language Reasoning](https://mlsanity.com/paper/2505.24298/) (2025)
- [APIGen: Automated Pipeline for Generating Verifiable and Diverse Function-Calling Datasets](https://mlsanity.com/paper/2406.18518/) (2024)
- [APIGen-MT: Agentic Pipeline for Multi-Turn Data Generation via Simulated Agent-Human Interplay](https://mlsanity.com/paper/2504.03601/) (2025)
- [Beyond Ten Turns: Unlocking Long-Horizon Agentic Search with Large-Scale Asynchronous RL](https://mlsanity.com/paper/2508.07976/) (2025)
- [ACEBench: Who Wins the Match Point in Tool Usage?](https://mlsanity.com/paper/2501.12851/) (2025)
- [Simulating Environments with Reasoning Models for Agent Training](https://mlsanity.com/paper/2511.01824/) (2025)
- [CoT-Self-Instruct: Building high-quality synthetic prompts for reasoning and non-reasoning tasks](https://mlsanity.com/paper/2507.23751/) (2025)
- [Toward Ultra-Long-Horizon Agentic Science: Cognitive Accumulation for Machine Learning Engineering](https://mlsanity.com/paper/2601.10402/) (2026)
- [ArenaRL: Scaling RL for Open-Ended Agents via Tournament-based Relative Ranking](https://mlsanity.com/paper/2601.06487/) (2026)
- [Scaling Agentic Reinforcement Learning for Tool-Integrated Reasoning in VLMs](https://mlsanity.com/paper/2511.19773/) (2025)
- [Eigen-1: Adaptive Multi-Agent Refinement with Monitor-Based RAG for Scientific Reasoning](https://mlsanity.com/paper/2509.21193/) (2025)
- [WorkForceAgent-R1: Incentivizing Reasoning Capability in LLM-based Web Agents via Reinforcement Learning](https://mlsanity.com/paper/2505.22942/) (2025)
- [Dynamic Skill Adaptation for Large Language Models](https://mlsanity.com/paper/2412.19361/) (2024)

## Built on by (most cited first, 17 of 17)

- [AgentDoG 1.5: A Lightweight and Scalable Alignment Framework for AI Agent Safety and Security](https://mlsanity.com/paper/2605.29801/) (2026)
- [Skill Self-Play: Pushing the Frontier of LLM Capability with Co-Evolving Skills](https://mlsanity.com/paper/2607.22529/) (2026)
- [IHBench: Evaluating Post-Interruption Recovery in Voice Agents with Structured Workflows](https://mlsanity.com/paper/2606.19595/) (2026)
- [TRACE: Capability-Targeted Agentic Training](https://mlsanity.com/paper/2604.05336/) (2026)
- [EVE-Agent: Evidence-Verifiable Self-Evolving Agents](https://mlsanity.com/paper/2605.22905/) (2026)
- [What Makes Good Agentic Data? An ACE Lens on Data Generation for LLM Agents](https://mlsanity.com/paper/2608.27260/) (2026)
- [WRIT: Write-Read Intensive Trajectory Synthesis for Multi-Turn User-Facing Agents](https://mlsanity.com/paper/2606.02908/) (2026)
- [WebGrader: Training LLMs for Web Development with Self-Evolving Programmatic Grader](https://mlsanity.com/paper/2608.06474/) (2026)
- [Building Multi-Task Agentic LLMs via Two-Phase Distillation](https://mlsanity.com/paper/2606.30044/) (2026)
- [AgentOmnia: Scaling Agentic Models for Full-Scenario Applications](https://mlsanity.com/paper/2607.23124/) (2026)
- [Ignorance or Incompetence? Constructing Knowledge-Gated, Verifiable Tasks for LLM Agents](https://mlsanity.com/paper/2608.30322/) (2026)
- [Know When to Stop, Where to Restart: Accelerating Multi-Turn Agentic On-Policy Distillation](https://mlsanity.com/paper/2609.14636/) (2026)
- [UnifiedPlayers: Enhance Tool-Integrated Reasoning in Agentic Reinforcement Learning](https://mlsanity.com/paper/2609.20089/) (2026)
- [Verifiable Hidden Dynamics Play: Generating Agentic RL Environments from Solved Mechanisms](https://mlsanity.com/paper/2609.27321/) (2026)
- [Rufus-Air: An Open LLM Post-Training Recipe](https://mlsanity.com/paper/2609.29421/) (2026)
- [SALMONN-duo: Adaptive Dual-System Coordination for Full-Duplex Voice Agents](https://mlsanity.com/paper/2609.34247/) (2026)
- [SynCo: Data Synthesis Co-Training for Self-Evolving LLMs via Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2610.11345/) (2026)

## Abstract (arXiv)

Interactive tool-using agents must solve real-world tasks via multi-turn interaction with both humans and external environments, requiring dialogue state tracking, multi-step tool execution, while following complex instructions. Post-training such agents is challenging because synthesis for high-quality multi-turn tool-use data is difficult to scale, and reinforcement learning (RL) could face noisy signals caused by user simulation, leading to degraded training efficiency. We propose a unified framework that combines a self-evolving data agent with verifier-based RL. Our system, EigenData, is a hierarchical multi-agent engine that synthesizes tool-grounded dialogues together with executable per-instance checkers, and improves generation reliability via closed-loop self-evolving process that updates prompts and workflow. Building on the synthetic data, we develop an RL recipe that first fine-tunes the user model and then applies GRPO-style training with trajectory-level group-relative advantages and dynamic filtering, yielding consistent improvements beyond SFT. Evaluated on tau^2-bench, our best model reaches 73.0% pass^1 on Airline and 98.3% pass^1 on Telecom, matching or exceeding frontier models. Overall, our results suggest a scalable pathway for bootstrapping complex tool-using behaviors without expensive human annotation.