mlsanityData to 14 Oct 2026

ISE: An Execution-Grounded Recipe for Multi-Turn OS-Agent Trajectories

arXiv 2606.11520 ↗First posted 9 Jun 2026cs.CL cs.AI cs.LGcited by 1

On the map: LLM agents and tool use → Tool learning and function calling

Abstract (arXiv)

Training capable OS agents requires data that simultaneously captures structured user intents, multi-turn task delegation, and grounded tool execution--properties absent from existing datasets. We propose ISE (Intent -> Simulate -> Execute), a three-stage synthesis paradigm that addresses these gaps jointly. Stage 1 constructs roughly 50000 structured intents via a 4D framework (Persona x Domain x Task x Complexity); after deduplication the pool contains 43956 unique intents and attains a Vendi Score of 61.57 over the entire pool on mpnet-base-v2 embeddings (cosine kernel, q=1). Stage 2 drives multi-turn user-agent interaction through a role-locked user simulator that grounds each user turn in actual execution outcomes, producing 23132 complete trajectories averaging 8.12 user turns and 68.24 total dialogue turns. Stage 3 runs every tool call inside a live, isolated OS workspace, generating authentic failure-recovery dynamics instead of simulated responses. Fine-tuning on ISETrace improves ClawEval pass@1 from 19.3 to 37.7 using Qwen3-8B on agent tool-use tasks with a standard protocol. This result outperforms zero-shot GPT-4o and the larger Qwen3-32B base model which is four times bigger. An ablation on Stage 2 proves multi-turn simulation brings a large portion of the performance gain. We release all source code and dataset at https://github.com/Valiere01/ISE-Trace.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Tool learning and function calling, one of 16 directions in LLM agents and tool use. Builds on 25 of its 29 references that are in the map. First posted on arXiv 9 Jun 2026.

Builds on

25 of 29 references.

Closest ancestors · 6

Same direction, newest first

  1. 2026EnvFactory: Scaling Tool-Use Agents via Executable Environments Synthesis and Robust RL18 May
  2. 2026EigenData: A Self-Evolving Multi-Agent Platform for Function-Calling Data Synthesis, Auditing, and Repair5 Mar
  3. 2025TOUCAN: Synthesizing 1.5M Tool-Agentic Data from Real-World MCP Environments1 Oct
  4. 2024APIGen: Automated Pipeline for Generating Verifiable and Diverse Function-Calling Datasets26 Jun
  5. 2023ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIs31 Jul
  6. 2023Gorilla: Large Language Model Connected with Massive APIs24 May

Other references · 19

  1. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,724
  2. 2024τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domainscited by 1,402
  3. 2023WizardLM: Empowering large pre-trained language models to follow complex instructionscited by 1,125
  4. 2023AgentTuning: Enabling Generalized Agent Abilities for LLMscited by 338
  5. 2022The Vendi Score: A Diversity Evaluation Metric for Machine Learningcited by 311
  6. 2025APIGen-MT: Agentic Pipeline for Multi-Turn Data Generation via Simulated Agent-Human Interplaycited by 167
  7. 2024OS-Genesis: Automating GUI Agent Trajectory Construction via Reverse Task Synthesiscited by 163
  8. 2024AgentTrek: Agent Trajectory Synthesis via Guiding Replay with Web Tutorialscited by 130
  9. 2024AgentInstruct: Toward Generative Teaching with Agentic Flowscited by 89
  10. 2026Mind the Sim2Real Gap in User Simulation for Agentic Taskscited by 54
  11. 2025TaskCraft: Automated Generation of Agentic Taskscited by 51
  12. 2025ToolMind Technical Report: A Large-Scale, Reasoning-Enhanced Tool-Use Datasetcited by 30
7 more
  1. 2026TermiGen: High-Fidelity Environment and Robust Trajectory Synthesis for Terminal Agentscited by 29
  2. 2026CLI-Gym: Scalable CLI Task Generation via Agentic Environment Inversioncited by 23
  3. 2026OpenMobile: Building Open Mobile Agents with Task and Trajectory Synthesiscited by 11
  4. 2026ToolCUA: Towards Optimal GUI-Tool Path Orchestration for Computer Use Agentscited by 11
  5. 2026Video2GUI: Synthesizing Large-Scale Interaction Trajectories for Generalized GUI Agent Pretrainingcited by 7
  6. 2026Controllable and Verifiable Tool-Use Data Synthesis for Agentic Reinforcement Learningcited by 6
  7. 2026CoVe: Training Interactive Tool-Use Agents via Constraint-Guided Verificationcited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

Similar, not cited

Close in wording, no citation either way

  1. 2026WRIT: Write-Read Intensive Trajectory Synthesis for Multi-Turn User-Facing Agents
  2. 2026Trajectory2Task: Training Robust Tool-Calling Agents with Synthesized Yet Verifiable Data for Complex User Intents
  3. 2025VeriOS: Query-Driven Proactive Human-Agent-GUI Interaction for Trustworthy OS Agents
  4. 2026Emerging from Ground: Addressing Intent Deviation in Tool-Using Agents via Deriving Real Calls into Virtual Trajectories
  5. 2025Close the Loop: Synthesizing Infinite Tool-Use Data via Multi-Agent Role-Playing
  6. 2026RideWay: Benchmarking Efficient Task Completion for Tool-Using Language Agents
  7. 2026Beyond Oracle Communication: Benchmarking Interactive Intent Alignment Under Miscommunication and Evolving User Intent
  8. 2026When Users Change Their Minds: Measuring and Repairing Intent Drift in LLM Agents