# Execution-First Synthetic Tool-Use Trace Generation for LLM Agents arXiv: https://arxiv.org/abs/2607.29175 · HTML: https://mlsanity.com/paper/2607.29175/ ## Key facts - Cited by 2 papers in the map (data through 14 Oct 2026). - Placed in Software engineering coding agents, one of 16 directions in LLM agents and tool use. - Builds on 29 of its 50 references that are in the map. - First posted on arXiv 31 Jul 2026. ## Placed in - LLM agents and tool use / [Software engineering coding agents](https://mlsanity.com/agents/software-engineering-coding-agents/) ## Builds on (most cited first, 29 of 29) - [ReAct: Synergizing Reasoning and Acting in Language Models](https://mlsanity.com/paper/2210.03629/) (2022) - [Toolformer: Language Models Can Teach Themselves to Use Tools](https://mlsanity.com/paper/2302.04761/) (2023) - [A Survey on Large Language Model based Autonomous Agents](https://mlsanity.com/paper/2308.11432/) (2023) - [ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIs](https://mlsanity.com/paper/2307.16789/) (2023) - [SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineering](https://mlsanity.com/paper/2405.15793/) (2024) - [The Rise and Potential of Large Language Model Based Agents: A Survey](https://mlsanity.com/paper/2309.07864/) (2023) - [τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains](https://mlsanity.com/paper/2406.12045/) (2024) - [CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquing](https://mlsanity.com/paper/2305.11738/) (2023) - [API-Bank: A Comprehensive Benchmark for Tool-Augmented LLMs](https://mlsanity.com/paper/2304.08244/) (2023) - [ToolAlpaca: Generalized Tool Learning for Language Models with 3000 Simulated Cases](https://mlsanity.com/paper/2306.05301/) (2023) - [ToolSandbox: A Stateful, Conversational, Interactive Evaluation Benchmark for LLM Tool Use Capabilities](https://mlsanity.com/paper/2408.04682/) (2024) - [A Survey on Data Selection for LLM Instruction Tuning](https://mlsanity.com/paper/2402.05123/) (2024) - [TOUCAN: Synthesizing 1.5M Tool-Agentic Data from Real-World MCP Environments](https://mlsanity.com/paper/2510.01179/) (2025) - [ToolNet: Connecting Large Language Models with Massive Tools via Tool Graph](https://mlsanity.com/paper/2403.00839/) (2024) - [AgentSynth: Scalable Task Generation for Generalist Computer-Use Agents](https://mlsanity.com/paper/2506.14205/) (2025) - [Middleware for LLMs: Tools Are Instrumental for Language Agents in Complex Environments](https://mlsanity.com/paper/2402.14672/) (2024) - [ToolMind Technical Report: A Large-Scale, Reasoning-Enhanced Tool-Use Dataset](https://mlsanity.com/paper/2511.15718/) (2025) - [Benchmarking LLM Tool-Use in the Wild](https://mlsanity.com/paper/2604.06185/) (2026) - [Autodata: An agentic data scientist to create high quality synthetic data](https://mlsanity.com/paper/2606.25996/) (2026) - [AgentLens: Revealing The Lucky Pass Problem in SWE-Agent Evaluation](https://mlsanity.com/paper/2605.12925/) (2026) - [Which Reasoning Trajectories Teach Students to Reason Better? A Simple Metric of Informative Alignment](https://mlsanity.com/paper/2601.14249/) (2026) - [ToolFailBench: Diagnosing Tool-Use Failures in LLM Agents](https://mlsanity.com/paper/2607.04686/) (2026) - [From Failure to Mastery: Generating Hard Samples for Tool-use Agents](https://mlsanity.com/paper/2601.01498/) (2026) - [How to Fine-Tune a Reasoning Model? A Teacher-Student Cooperation Framework to Synthesize Student-Consistent SFT Data](https://mlsanity.com/paper/2604.14164/) (2026) - [Harnessing the Power of David against Goliath: Exploring Instruction Data Generation without Using Closed-Source Models](https://mlsanity.com/paper/2308.12711/) (2023) - [ABC-Bench: Benchmarking Agentic Backend Coding in Real-World Development](https://mlsanity.com/paper/2601.11077/) (2026) - [Why Johnny Can't Use Agents: Industry Aspirations vs. User Realities with AI Agents](https://mlsanity.com/paper/2509.14528/) (2025) - [ToolGrad: Efficient Tool-use Dataset Generation with Textual "Gradients"](https://mlsanity.com/paper/2508.04086/) (2025) - [ISE: An Execution-Grounded Recipe for Multi-Turn OS-Agent Trajectories](https://mlsanity.com/paper/2606.11520/) (2026) ## Built on by (most cited first, 2 of 2) - [What Makes Good Agentic Data? An ACE Lens on Data Generation for LLM Agents](https://mlsanity.com/paper/2608.27260/) (2026) - [Davinci-Tool: Scaling Agentic Tool-Use Supervision with Verifiable Executable Environments](https://mlsanity.com/paper/2610.16867/) (2026) ## Abstract (arXiv) Agentic software-engineering and industrial systems increasingly operate through executable workflows rather than code genera- tion alone: they search artifacts, invoke tools, inspect structured observations, and query databases. Training these agents requires supervision data that captures valid tool interactions and executable workflows. However, traditional query-first data synthesis can fail because plausible user requests may not correspond to valid tool sequences, compatible parameters, or available data. To address this limitation, we propose SyntheticAgentTraceQA, an execution- first framework for generating scalable supervision data for tool- augmented agents. Our framework first constructs high-level work- flow structures, maps them to available tools through dependency- aware assignment, executes and validates the resulting traces in con- trolled environments, and only then synthesizes natural-language user tasks, teacher-generated reasoning annotations, and reference answers. We evaluate the framework across four tool ecosystems and use the resulting data to fine-tune and evaluate Qwen model variants. The results show that execution-grounded supervision improves tool execution behavior, reference-trace agreement, and answer-generation performance on the evaluated tasks. Further analysis reveals a supervision trade-off: masked supervision, which excludes reasoning annotations from the training objective, im- proves final-answer metrics, whereas full supervision, computing loss over the complete assistant output including reasoning tokens, underperforms on answer quality and does not consistently im- prove reference-trace agreement, particularly at the 9B scale. These findings highlight the importance of designing synthetic supervi- sion according to the desired capabilities of tool-augmented agents.