mlsanityData to 14 Oct 2026

AgentSynth: Scalable Task Generation for Generalist Computer-Use Agents

arXiv 2506.14205 ↗First posted 17 Jun 2025cs.CLcited by 46

On the map: LLM agents and tool use → General LLM agent frameworks and benchmarks

Abstract (arXiv)

We introduce AgentSynth, a scalable and cost-efficient pipeline for automatically synthesizing high-quality tasks and trajectory datasets for generalist computer-use agents. Leveraging information asymmetry, AgentSynth constructs subtasks that are simple during generation but significantly more challenging when composed into long-horizon tasks, enabling the creation of over 6,000 diverse and realistic tasks. A key strength of AgentSynth is its ability to precisely modulate task complexity by varying the number of subtasks. Empirical evaluations show that state-of-the-art LLM agents suffer a steep performance drop, from 18% success at difficulty level 1 to just 4% at level 6, highlighting the benchmark's difficulty and discriminative power. Moreover, our pipeline achieves a low average cost of 0.60 per trajectory, orders of magnitude cheaper than human annotations. Our code and data are available at https://github.com/sunblaze-ucb/AgentSynth

Key facts. Cited by 46 papers in the map (data through 14 Oct 2026). Placed in General LLM agent frameworks and benchmarks, one of 16 directions in LLM agents and tool use. Builds on 26 of its 39 references that are in the map. First posted on arXiv 17 Jun 2025.

Builds on

26 of 39 references.

Closest ancestors · 3

Same direction, newest first

  1. 2025Learn-by-interact: A Data-Centric Framework for Self-Adaptive Agents in Realistic Environments18 Jan
  2. 2024TheAgentCompany: Benchmarking LLM Agents on Consequential Real World Tasks18 Dec
  3. 2024τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains17 Jun

Other references · 23

  1. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
  2. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,373
  3. 2024SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineeringcited by 1,737
  4. 2022WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agentscited by 1,251
  5. 2023Mind2Web: Towards a Generalist Agent for the Webcited by 1,242
  6. 2023WizardLM: Empowering large pre-trained language models to follow complex instructionscited by 1,125
  7. 2023Large Language Models Cannot Self-Correct Reasoning Yetcited by 1,017
  8. 2024Scaling Synthetic Data Creation with 1,000,000,000 Personascited by 417
  9. 2024WorkArena: How Capable Are Web Agents at Solving Common Knowledge Work Tasks?cited by 363
  10. 2018Reinforcement Learning on Web Interfaces Using Workflow-Guided Explorationcited by 284
  11. 2025An Illusion of Progress? Assessing the Current State of Web Agentscited by 174
  12. 2024OS-Genesis: Automating GUI Agent Trajectory Construction via Reverse Task Synthesiscited by 163
11 more
  1. 2025Agent S2: A Compositional Generalist-Specialist Framework for Computer Use Agentscited by 159
  2. 2024AgentTrek: Agent Trajectory Synthesis via Guiding Replay with Web Tutorialscited by 130
  3. 2025AgentRewardBench: Evaluating Automatic Evaluations of Web Agent Trajectoriescited by 108
  4. 2024WorkArena++: Towards Compositional Planning and Reasoning-based Common Knowledge Work Taskscited by 90
  5. 2025NaturalReasoning: Reasoning in the Wild with 2.8M Challenging Questionscited by 84
  6. 2025Explorer: Scaling Exploration-driven Web Trajectory Synthesis for Multimodal Web Agentscited by 72
  7. 2024NNetNav: Unsupervised Learning of Browser Agents Through Environment Interaction in the Wildcited by 59
  8. 2024AutoBencher: Towards Declarative Benchmark Constructioncited by 44
  9. 2025Go-Browse: Training Web Agents with Structured Explorationcited by 33
  10. 2023Exposing Limitations of Language Model Agents in Sequential-Task Compositions on the Webcited by 24
  11. 2019Synthetic Datasets for Neural Program Synthesiscited by 15

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 10; 2026: 36.

46 papers.

Sort:

By area:

  1. 2026Agent World Model: Infinity Synthetic Environments for Agentic Reinforcement Learningcited by 66
  2. 2025OdysseyBench: Evaluating LLM Agents on Long-Horizon Complex Office Application Workflowscited by 57
  3. 2025Agentic Web: Weaving the Next Web with AI Agentscited by 45
  4. 2025Scaling Agent Learning via Experience Synthesiscited by 44
  5. 2026InfiniteWeb: Scalable Web Environment Synthesis for GUI Agent Trainingcited by 27
  6. 2025Scaling Synthetic Task Generation for Agents via Explorationcited by 23
  7. 2026WebGym: Scaling Training Environments for Visual Web Agents with Realistic Taskscited by 19
  8. 2026Proactive Agent Research Environment: Simulating Active Users to Evaluate Proactive Assistantscited by 15
  9. 2025GUI-ReWalk: Massive Data Generation for GUI Agent via Stochastic Exploration and Intent-Aware Reasoningcited by 13
  10. 2026OpenMobile: Building Open Mobile Agents with Task and Trajectory Synthesiscited by 11
  11. 2025DatasetResearch: Benchmarking Agent Systems for Demand-Driven Dataset Discoverycited by 9
  12. 2026Agentic Environment Engineering for Large Language Models: A Survey of Environment Modeling, Synthesis, Evaluation, and Applicationcited by 8
34 more
  1. 2026SCALECUA: Scaling Computer Use Agents with Verifiable Task Synthesis and Efficient Online RLcited by 7
  2. 2026NexForge: Scaling Agent Capabilities through Requirement-Driven Task Synthesis for LLMscited by 7
  3. 2026OpenForgeRL: Train Harness-native Agents in Any Environmentcited by 7
  4. 2025Malice in Agentland: Down the Rabbit Hole of Backdoors in the AI Supply Chaincited by 6
  5. 2025SynthTools: A Framework for Scaling Synthetic Tools for Agent Developmentcited by 5
  6. 2026The World Won't Stay Still: Programmable Evolution for Agent Benchmarkscited by 5
  7. 2026Anchor: Mitigating Artifact Drift in Agent Benchmark Generationcited by 5
  8. 2026Scaling Large Reasoning Models beyond Human Supervision: A Path toward Superintelligencecited by 5
  9. 2026Learning with Challenges: Adaptive Difficulty-Aware Data Generation for Mobile GUI Agent Trainingcited by 4
  10. 2026Agentic Reward Modeling: Verifying GUI Agent via Progressive Trajectory-Grounded Interactioncited by 4
  11. 2025Matrix: Peer-to-Peer Multi-Agent Synthetic Data Generation Frameworkcited by 3
  12. 2026Internalizing the Future: A Unified Agentic Training Paradigm for World Model Planningcited by 3
  13. 2026What Makes Good Agentic Data? An ACE Lens on Data Generation for LLM Agentscited by 3
  14. 2025Learning from Online Videos at Inference Time for Computer-Use Agentscited by 2
  15. 2026Scalable Environments Drive Generalizable Agentscited by 2
  16. 2026ProCUA-SFT Technical Reportcited by 2
  17. 2026Execution-First Synthetic Tool-Use Trace Generation for LLM Agentscited by 2
  18. 2026OmnilingualGAIA2: Evaluating the Multilingual Gap in Frontier AI Agentscited by 2
  19. 2026InfoSynth: Information-Guided Benchmark Synthesis for LLMscited by 1
  20. 2026Learn from Weaknesses: Automated Domain Specialization for Small Computer-Use Agentscited by 1
  21. 2026ChainWorld: Composing Long-Horizon Desktop Workloads from Atomic OSWorld Taskscited by 1
  22. 2026FedGUI: Benchmarking Federated GUI Agents across Heterogeneous Platforms, Devices, and Operating Systemscited by 0
  23. 2026AgentOmnia: Scaling Agentic Models for Full-Scenario Applicationscited by 0
  24. 2026FinCUABuild: Can Agents Build Reliable Benchmarks for Dynamic Financial Computer Use?cited by 0
  25. 2026AutoGym: Blueprint-First Generation of Verifiable Agent Gymscited by 0
  26. 2026FinancialAuditBench: Benchmark Construction under Differential Privacy Using Real-World Priorscited by 0
  27. 2026WebPageBench: Event-Level Verification and Controlled UI-Variant Generation for Web Agentscited by 0
  28. 2026GraphForge: Training Working Agents with Graph-Anchored Workspace Synthesiscited by 0
  29. 2026AutoGUIWorld: Image Generators as Visual World Models for GUI Agentcited by 0
  30. 2026Scaling Verifiable Environments for Long-horizon Work Agentscited by 0
  31. 2026Terrarium: Growing Personal Assistants via Scalable Synthesis of App Worldscited by 0
  32. 2026Dream2Discern: Learning from Hard Latent Variants for Runtime Failure Prediction in Multi-Agent Systemscited by 0
  33. 2026Interlock: From Local Correctness to Task Completion in Long-Horizon Agentscited by 0
  34. 2026Explorer-Zero: Improving Self-Play for Long-Horizon Tasks with Decomposable Verifiable Rewardscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025TaskCraft: Automated Generation of Agentic Tasks
  2. 2026DIVE: Scaling Diversity in Agentic Task Synthesis for Generalizable Tool Use
  3. 2026Meta-Task: Turning Terminal Task Synthesis into a Terminal Task for Scalable Agent Training
  4. 2026AOrchestra: Automating Sub-Agent Creation for Agentic Orchestration
  5. 2026Structured Distillation of Web Agent Capabilities Enables Generalization
  6. 2026Trace2Skill: Distill Trajectory-Local Lessons into Transferable Agent Skills
  7. 2026AgentIF-OneDay: A Task-level Instruction-Following Benchmark for General AI Agents in Daily Scenarios
  8. 2025Interaction as Intelligence Part II: Asynchronous Human-Agent Rollout for Long-Horizon Task Training