mlsanityData to 7 Oct 2026

AgentTrek: Agent Trajectory Synthesis via Guiding Replay with Web Tutorials

arXiv 2412.09605 ↗First posted 12 Dec 2024cs.CLcited by 123

On the map: LLM agents and tool use → Web, search, and research agents

Abstract (arXiv)

Graphical User Interface (GUI) agents can automate complex tasks across digital environments, but their development is hindered by the scarcity of high-quality trajectory data for training. Existing approaches rely on expensive human annotation, making them unsustainable at scale. We propose AgentTrek, a scalable data synthesis pipeline that generates web agent trajectories by leveraging publicly available tutorials. Our three-stage method: (1) automatically harvests and filters tutorial-like texts from the internet using a specialized classification model, (2) transforms these texts into structured task specifications with step-by-step instructions, and (3) employs a visual-language model (VLM) agent to execute these instructions in real environments, while a VLM-based evaluator verifies trajectory correctness. The synthesized trajectories encompass multiple modalities, including text-based HTML observations with function-calling API actions, and vision-based screenshot observations with pixel-level actions. This multimodal data, enriched with chain-of-thought reasoning, enables agents to achieve state-of-the-art performance on both textual web browsing benchmarks (e.g., WebArena) and visual web grounding and browsing benchmarks (e.g., ScreenSpot Web and Multimodal Mind2Web). Furthermore, our fully automated approach significantly reduces data collection costs, achieving a cost of just 0.55 per high-quality trajectory without human annotators. Our work demonstrates that guided replay using web tutorials is a practical and scalable strategy for training advanced GUI agents, paving the way for more capable and autonomous digital assistants.

Key facts. Cited by 123 papers in the map (data through 7 Oct 2026). Placed in Web, search, and research agents, one of 16 directions in LLM agents and tool use. Builds on 22 of its 28 references that are in the map. First posted on arXiv 12 Dec 2024.

Builds on

22 of 28 references.

Closest ancestors · 8

Same direction, newest first

  1. 2024NNetNav: Unsupervised Learning of Browser Agents Through Environment Interaction in the Wild3 Oct
  2. 2024AutoWebGLM: A Large Language Model-based Web Navigating Agent4 Apr
  3. 2024WorkArena: How Capable Are Web Agents at Solving Common Knowledge Work Tasks?12 Mar
  4. 2024WebVoyager: Building an End-to-End Web Agent with Large Multimodal Models25 Jan
  5. 2024GPT-4V(ision) is a Generalist Web Agent, if Grounded3 Jan
  6. 2023WebArena: A Realistic Web Environment for Building Autonomous Agents25 Jul
  7. 2023Mind2Web: Towards a Generalist Agent for the Web9 Jun
  8. 2022WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agents4 Jul

Other references · 14

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,151
  3. 2023ToolLLM: Facilitating Large Language Models to Master 16000+ Real-world APIscited by 1,880
  4. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,722
  5. 2024OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environmentscited by 1,134
  6. 2016Bag of Tricks for Efficient Text Classificationcited by 869
  7. 2023Instruction Tuning with GPT-4cited by 676
  8. 2024SeeClick: Harnessing GUI Grounding for Advanced Visual GUI Agentscited by 553
  9. 2023Android in the Wild: A Large-Scale Dataset for Android Device Controlcited by 314
  10. 2022ZeroGen: Efficient Zero-shot Learning via Dataset Generationcited by 209
  11. 2024Trial and Error: Exploration-Based Trajectory Optimization for LLM Agentscited by 194
  12. 2024OmniParser for Pure Vision Based GUI Agentcited by 193
2 more
  1. 2024Autonomous Evaluation and Refinement of Digital Agentscited by 136
  2. 2024ScreenAgent: A Vision Language Model-driven Computer Control Agentcited by 101

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 46; 2026: 76.

123 papers.

Sort:

By area:

  1. 2025SWE-smith: Scaling Data for Software Engineering Agentscited by 255
  2. 2025An Illusion of Progress? Assessing the Current State of Web Agentscited by 165
  3. 2025OpenCUA: Open Foundations for Computer-Use Agentscited by 151
  4. 2025Inducing Programmatic Skills for Agentic Taskscited by 103
  5. 2025A Survey of WebAgents: Towards Next-Generation AI Agents for Web Automation with Large Foundation Modelscited by 87
  6. 2025Explorer: Scaling Exploration-driven Web Trajectory Synthesis for Multimodal Web Agentscited by 68
  7. 2025SEAgent: Self-Evolving Computer Use Agent with Autonomous Learning from Experiencecited by 68
  8. 2025OdysseyBench: Evaluating LLM Agents on Long-Horizon Complex Office Application Workflowscited by 54
  9. 2026Agent World Model: Infinity Synthetic Environments for Agentic Reinforcement Learningcited by 51
  10. 2024Proposer-Agent-Evaluator(PAE): Autonomous Skill Discovery For Foundation Model Internet Agentscited by 50
  11. 2025Earth-Agent: Unlocking the Full Landscape of Earth Observation with Agentscited by 42
  12. 2025AgentSynth: Scalable Task Generation for Generalist Computer-Use Agentscited by 41
38 more
  1. 2025TongUI: Internet-Scale Trajectories from Multimodal Web Tutorials for Generalized GUI Agentscited by 40
  2. 2025Scaling Agent Learning via Experience Synthesiscited by 40
  3. 2025Thinking vs. Doing: Agents that Reason by Scaling Test-Time Interactioncited by 34
  4. 2025GenEnv: Difficulty-Aligned Co-Evolution Between LLM Agents and Environment Simulatorscited by 34
  5. 2026OSWorld 2.0: Benchmarking Computer Use Agents on Long-Horizon Real-World Taskscited by 29
  6. 2026InfiniteWeb: Scalable Web Environment Synthesis for GUI Agent Trainingcited by 26
  7. 2025Symbiotic Cooperation for Web Agents: Harnessing Complementary Strengths of Large and Small LLMscited by 24
  8. 2026Gym-Anything: Turn any Software into an Agent Environmentcited by 24
  9. 2026WebWorld: A Large-Scale World Model for Web Agent Trainingcited by 23
  10. 2025VideoAgentTrek: Computer Use Pretraining from Unlabeled Videoscited by 22
  11. 2025ProgRM: Build Better GUI Agents with Progress Rewardscited by 21
  12. 2026CUA-Gym: Scaling Verifiable Training Environments and Tasks for Computer-Use Agentscited by 21
  13. 2026AutoWebWorld: Synthesizing Infinite Verifiable Web Environments via Finite State Machinescited by 20
  14. 2025WorldGUI: An Interactive Benchmark for Desktop GUI Automation from Any Starting Pointcited by 19
  15. 2026OS-Symphony: A Holistic Framework for Robust and Generalist Computer-Using Agentcited by 19
  16. 2025WebSynthesis: World-Model-Guided MCTS for Efficient WebUI-Trajectory Synthesiscited by 18
  17. 2025Efficient Agent Training for Computer Usecited by 17
  18. 2025LLMs as Scalable, General-Purpose Simulators For Evolving Digital Agent Trainingcited by 17
  19. 2025WebLeaper: Empowering Efficiency and Efficacy in WebAgent via Enabling Info-Rich Seekingcited by 17
  20. 2025Test-Time Adaptation for LLM Agents via Environment Interactioncited by 17
  21. 2025Agent Data Protocol: Unifying Datasets for Diverse, Effective Fine-tuning of LLM Agentscited by 16
  22. 2025Scalable Video-to-Dataset Generation for Cross-Platform Mobile Agentscited by 14
  23. 2025WebCoT: Enhancing Web Agent Reasoning by Reconstructing Chain-of-Thought in Reflection, Branching, and Rollbackcited by 14
  24. 2025Retrieval-augmented GUI Agents with Generative Guidelinescited by 14
  25. 2026GUI-Libra: Training Native GUI Agents to Reason and Act with Action-aware Supervision and Partially Verifiable RLcited by 14
  26. 2026CoEvolve: Training LLM Agents via Agent-Data Mutual Evolutioncited by 14
  27. 2025Breaking the Data Barrier -- Building GUI Agents Through Task Generalizationcited by 13
  28. 2025MEMTRACK: Evaluating Long-Term Memory and State Tracking in Multi-Platform Dynamic Agent Environmentscited by 13
  29. 2025EchoTrail-GUI: Building Actionable Memory for GUI Agents via Critic-Guided Self-Explorationcited by 13
  30. 2025GUI-ReWalk: Massive Data Generation for GUI Agent via Stochastic Exploration and Intent-Aware Reasoningcited by 12
  31. 2025SynthAgent: Adapting Web Agents with Synthetic Supervisioncited by 12
  32. 2025WebSTAR: Scalable Data Synthesis for Computer Use Agents with Step-Level Filteringcited by 11
  33. 2026Who&When Pro: Can LLMs Really Attribute Failures in AI Agents?cited by 11
  34. 2025R-WoM: Retrieval-augmented World Model For Computer-use Agentscited by 10
  35. 2025WebDevJudge: Evaluating (M)LLMs as Critiques for Web Development Qualitycited by 10
  36. 2025Watch and Learn: Learning to Use Computers from Online Videoscited by 9
  37. 2025WebAggregator: Enhancing Compositional Reasoning Capabilities of Deep Research Agent Foundation Modelscited by 9
  38. 2026OpenComputer: Verifiable Software Worlds for Computer-Use Agentscited by 9

50 most cited of 123 All 123 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026MolmoWeb: Open Visual Web Agent and Open Data for the Open Web
  2. 2024OS-Genesis: Automating GUI Agent Trajectory Construction via Reverse Task Synthesis
  3. 2026HATS: Hardness-Aware Trajectory Synthesis for GUI Agents
  4. 2025AgentRewardBench: Evaluating Automatic Evaluations of Web Agent Trajectories
  5. 2025GUI-360^circ: A Comprehensive Dataset and Benchmark for Computer-Using Agents
  6. 2026MAG: A Web-Agent Benchmark and Harness for Multimodal Action and Guide Generation
  7. 2026WRIT: Write-Read Intensive Trajectory Synthesis for Multi-Turn User-Facing Agents
  8. 2026SCAFFOLD: Self-Improving Web Agents via Recursive Parametric Skill Abstraction