What Makes Good Agentic Data? An ACE Lens on Data Generation for LLM Agents
On the map: LLM agents and tool use → General LLM agent frameworks and benchmarks
Abstract (arXiv)
LLM agents increasingly rely on generated interaction data to learn how to interact with external environments. Agentic data generation must maintain consistency among environments, tasks, interactions, and success signals while producing experience that is useful rather than merely abundant. Existing work spans many agent domains, but domain-centered organization and heterogeneous evaluation often obscure common generation mechanisms and conflate candidate construction with verification and selection. This work develops a two-level framework for the field. First, we represent agentic data as a common factorized object (E,q,τ,v), comprising an environment specification, task signal, interaction realization, and optional verifier. We organize generation paradigms by their primary anchor and dependency structure. Second, we formulate generation as constrained distribution design through the Accuracy-Complexity-divErsity (ACE) lens. Accuracy establishes the feasible support of grounded and internally consistent data. Within this support, Complexity places learning mass relative to the capability of a declared learner and execution configuration, while divErsity controls coverage and redundancy of data. Using this framework, we explore how prior work verifies generated experience, constructs and calibrates difficulty, and expands behavioral coverage. The literature reveals a shift toward execution-grounded accuracy, learner-relative complexity, and diversity beyond surface variation or dataset size. We further discuss broader directions and emerging trends in agentic data generation through the ACE lens, including their implications for scaling, data sources, training regimes and adaptive learning. Overall, the central challenge is not simply to generate more data, but to continually allocate valid, informative, and non-redundant experience as agents and environments evolve.
Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Placed in General LLM agent frameworks and benchmarks, one of 16 directions in LLM agents and tool use. Builds on 133 of its 146 references that are in the map. First posted on arXiv 27 Aug 2026.
Builds on
133 of 146 references.
Closest ancestors · 24
Same direction, newest first
- 2026Can Agents Generalize to the Open World? Unveiling the Fragility of Static Training in Tool Use1 Jul
- 2026Agent-World: Scaling Real-World Environment Synthesis for Evolving General Agent Intelligence20 Apr
- 2026HiL-Bench (Human-in-Loop Benchmark): Do Agents Know When to Ask for Help?10 Apr
- 2026LOGIGEN: Logic-Driven Generation of Verifiable Agentic Tasks28 Feb
- 2026Agent World Model: Infinity Synthetic Environments for Agentic Reinforcement Learning10 Feb
- 2026Understanding Agent Scaling in LLM-Based Multi-Agent Systems via Diversity3 Feb
- 2026Agentic Proposing: Enhancing Large Language Model Reasoning via Compositional Skill Synthesis3 Feb
- 2026DeepPlanning: Benchmarking Long-Horizon Agentic Planning with Verifiable Constraints26 Jan
- 2026ACE-Router: Generalizing History-Aware Routing from MCP Tools to the Agent Web13 Jan
- 2025Youtu-LLM: Unlocking the Native Agentic Potential for Lightweight Large Language Models31 Dec
- 2025GenEnv: Difficulty-Aligned Co-Evolution Between LLM Agents and Environment Simulators22 Dec
- 2025AgentEvolver: Towards Efficient Self-Evolving Agent System13 Nov
12 more
- 2025AgentFrontier: Expanding the Capability Frontier of LLM Agents with ZPD-Guided Data Synthesis28 Oct
- 2025Scaling Synthetic Task Generation for Agents via Exploration29 Sep
- 2025ToolACE-MT: Non-Autoregressive Generation for Agentic Multi-Turn Interaction18 Aug
- 2025Chain-of-Agents: End-to-End Agent Foundation Models via Multi-Agent Distillation and Agentic RL6 Aug
- 2025AgentSynth: Scalable Task Generation for Generalist Computer-Use Agents17 Jun
- 2025APIGen-MT: Agentic Pipeline for Multi-Turn Data Generation via Simulated Agent-Human Interplay4 Apr
- 2025Plan-and-Act: Improving Planning of Agents for Long-Horizon Tasks12 Mar
- 2025Learn-by-interact: A Data-Centric Framework for Self-Adaptive Agents in Realistic Environments18 Jan
- 2024AgentGen: Enhancing Planning Abilities for Large Language Model based Agent via Environment and Task Generation1 Aug
- 2024CRAB: Cross-environment Agent Benchmark for Multimodal Language Model Agents1 Jul
- 2024Agent-FLAN: Designing Data and Methods of Effective Agent Tuning for Large Language Models19 Mar
- 2023SOTOPIA: Interactive Evaluation for Social Intelligence in Language Agents18 Oct
Other references · 109
- 2023ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIscited by 1,947
- 2017Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real Worldcited by 1,498
- 2025Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learningcited by 1,444
- 2023Gorilla: Large Language Model Connected with Massive APIscited by 1,332
- 2024OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environmentscited by 1,195
- 2025τ2-Bench: Evaluating Conversational Agents in a Dual-Control Environmentcited by 627
- 2024AndroidWorld: A Dynamic Benchmarking Environment for Autonomous Agentscited by 468
- 2024AFlow: Automating Agentic Workflow Generationcited by 432
- 2024RoboCasa: Large-Scale Simulation of Everyday Tasks for Generalist Robotscited by 407
- 2024AppWorld: A Controllable World of Apps and People for Benchmarking Interactive Coding Agentscited by 401
- 2023ToolAlpaca: Generalized Tool Learning for Language Models with 3000 Simulated Casescited by 393
- 2025ReTool: Reinforcement Learning for Strategic Tool Use in LLMscited by 392
38 more
- 2022ProcTHOR: Large-Scale Embodied AI Using Procedural Generationcited by 380
- 2024Training Software Engineering Agents and Verifiers with SWE-Gymcited by 336
- 2025R1-Searcher: Incentivizing the Search Capability in LLMs via Reinforcement Learningcited by 336
- 2022The Vendi Score: A Diversity Evaluation Metric for Machine Learningcited by 311
- 2024ToolSandbox: A Stateful, Conversational, Interactive Evaluation Benchmark for LLM Tool Use Capabilitiescited by 295
- 2025SWE-smith: Scaling Data for Software Engineering Agentscited by 285
- 2023ToRA: A Tool-Integrated Reasoning Agent for Mathematical Problem Solvingcited by 282
- 2023Holodeck: Language Guided Generation of 3D Embodied AI Environmentscited by 268
- 2017Automatic Goal Generation for Reinforcement Learning Agentscited by 259
- 2024DeepSeek-Prover: Advancing Theorem Proving in LLMs through Large-Scale Synthetic Datacited by 234
- 2023RoboGen: Towards Unleashing Infinite Data for Automated Robot Learning via Generative Simulationcited by 218
- 2024ToolACE: Winning the Points of LLM Function Callingcited by 216
- 2024APIGen: Automated Pipeline for Generating Verifiable and Diverse Function-Calling Datasetscited by 198
- 2019Active Domain Randomizationcited by 170
- 2024OS-Genesis: Automating GUI Agent Trajectory Construction via Reverse Task Synthesiscited by 163
- 2025R2E-Gym: Procedural Environments and Hybrid Verifiers for Scaling Open-Weights SWE Agentscited by 158
- 2023MathCoder: Seamless Code Integration in LLMs for Enhanced Mathematical Reasoningcited by 156
- 2023TaskBench: Benchmarking Large Language Models for Task Automationcited by 144
- 2025SWE-rebench: An Automated Pipeline for Task Collection and Decontaminated Evaluation of Software Engineering Agentscited by 142
- 2024AgentTrek: Agent Trajectory Synthesis via Guiding Replay with Web Tutorialscited by 130
- 2023Generative agent-based modeling with actions grounded in physical, social, or digital space using Concordiacited by 126
- 2023GenSim: Generating Robotic Simulation Tasks via Large Language Modelscited by 125
- 2025MCP-Bench: Benchmarking Tool-Using LLM Agents with Complex Real-World Tasks via MCP Serverscited by 109
- 2021Scalable Evaluation of Multi-Agent Reinforcement Learning with Melting Potcited by 99
- 2025Gold-medalist Performance in Solving Olympiad Geometry with AlphaGeometry2cited by 97
- 2024MARIO: MAth Reasoning with code Interpreter Output -- A Reproducible Pipelinecited by 81
- 2025Explorer: Scaling Exploration-driven Web Trajectory Synthesis for Multimodal Web Agentscited by 72
- 2025TOUCAN: Synthesizing 1.5M Tool-Agentic Data from Real-World MCP Environmentscited by 69
- 2024Seal-Tools: Self-Instruct Tool Learning Dataset for Agent Tuning and Detailed Benchmarkcited by 63
- 2026EnvScaler: Scaling Tool-Interactive Environments for LLM Agent via Programmatic Synthesiscited by 63
- 2025WebEvolver: Enhancing Web Agent Self-Improvement with Coevolving World Modelcited by 61
- 2025WebSailor-V2: Bridging the Chasm to Proprietary Agents via Synthetic Data and Scalable Reinforcement Learningcited by 61
- 2025Simulating Environments with Reasoning Models for Agent Trainingcited by 57
- 2025ToolHop: A Query-Driven Benchmark for Evaluating Large Language Models in Multi-Hop Tool Usecited by 50
- 2024Facilitating Multi-turn Function Calling for LLMs via Compositional Instruction Tuningcited by 42
- 2025MCPMark: A Benchmark for Stress-Testing Realistic and Comprehensive MCP Usecited by 39
- 2025Magnet: Multi-turn Tool-use Data Synthesis and Distillation via Graph Translationcited by 37
- 2025ToolDial: Multi-turn Dialogue Generation Method for Tool-Augmented Language Modelscited by 34
50 most cited of 109 All 109 (JSON)
Built on by
Citing papers per year: 2026: 3.
3 papers.
Similar, not cited
Close in wording, no citation either way
- 2026Agentic Data Engineering for LLMs: System Design and a Controlled Empirical Study
- 2026Know What You Don't Know: Agentic Data Synthesis by Evolving Generation Space for Closed-Loop Learning
- 2026From Failure to Mastery: Generating Hard Samples for Tool-use Agents
- 2026SkillsVote: Lifecycle Governance of Agent Skills from Collection, Recommendation to Evolution
- 2026Agentic CLEAR: Automating Multi-Level Evaluation of LLM Agents
- 2026TRACE: A Self-Evolving Skill Bank for Consistent, Limit-Aware LLM Agents
- 2026A Framework for Evaluating Agentic Skills at Scale
- 2025FABRIC: Framework for Agent-Based Realistic Intelligence Creation