mlsanityData to 14 Oct 2026

OS-Genesis: Automating GUI Agent Trajectory Construction via Reverse Task Synthesis

arXiv 2412.19723 ↗First posted 27 Dec 2024cs.AI cs.CL cs.CVcited by 163

On the map: LLM agents and tool use → GUI and mobile agents

Abstract (arXiv)

Graphical User Interface (GUI) agents powered by Vision-Language Models (VLMs) have demonstrated human-like computer control capability. Despite their utility in advancing digital automation, a critical bottleneck persists: collecting high-quality trajectory data for training. Common practices for collecting such data rely on human supervision or synthetic data generation through executing pre-defined tasks, which are either resource-intensive or unable to guarantee data quality. Moreover, these methods suffer from limited data diversity and significant gaps between synthetic data and real-world environments. To address these challenges, we propose OS-Genesis, a novel GUI data synthesis pipeline that reverses the conventional trajectory collection process. Instead of relying on pre-defined tasks, OS-Genesis enables agents first to perceive environments and perform step-wise interactions, then retrospectively derive high-quality tasks to enable trajectory-level exploration. A trajectory reward model is then employed to ensure the quality of the generated trajectories. We demonstrate that training GUI agents with OS-Genesis significantly improves their performance on highly challenging online benchmarks. In-depth analysis further validates OS-Genesis's efficiency and its superior data quality and diversity compared to existing synthesis methods. Our codes, data, and checkpoints are available at https://qiushisun.github.io/OS-Genesis-Home/.

Key facts. Cited by 163 papers in the map (data through 14 Oct 2026). Placed in GUI and mobile agents, one of 16 directions in LLM agents and tool use. Builds on 40 of its 56 references that are in the map. First posted on arXiv 27 Dec 2024.

Builds on

40 of 56 references.

Closest ancestors · 14

Same direction, newest first

  1. 2024The Dawn of GUI Agent: A Preliminary Case Study with Claude 3.5 Computer Use15 Nov
  2. 2024OS-ATLAS: A Foundation Action Model for Generalist GUI Agents30 Oct
  3. 2024Navigating the Digital World as Humans Do: Universal Visual Grounding for GUI Agents7 Oct
  4. 2024AMEX: Android Multi-annotation Expo Dataset for Mobile GUI Agents3 Jul
  5. 2024GUICourse: From General Vision Language Models to Versatile GUI Agents17 Jun
  6. 2024On the Effects of Data Scale on UI Control Agents6 Jun
  7. 2024OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments11 Apr
  8. 2024Android in the Zoo: Chain-of-Action-Thought for GUI Agents5 Mar
  9. 2024OmniACT: A Dataset and Benchmark for Enabling Multimodal Generalist Autonomous Agents for Desktop and Web27 Feb
  10. 2024OS-Copilot: Towards Generalist Computer Agents with Self-Improvement12 Feb
  11. 2024ScreenAgent: A Vision Language Model-driven Computer Control Agent9 Feb
  12. 2024Mobile-Agent: Autonomous Multi-Modal Mobile Device Agent with Visual Perception29 Jan
2 more
  1. 2024SeeClick: Harnessing GUI Grounding for Advanced Visual GUI Agents17 Jan
  2. 2023AppAgent: Multimodal Agents as Smartphone Users21 Dec

Other references · 26

  1. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,170
  2. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
  3. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,234
  4. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,724
  5. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,565
  6. 2023WebArena: A Realistic Web Environment for Building Autonomous Agentscited by 2,006
  7. 2022WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agentscited by 1,251
  8. 2023Mind2Web: Towards a Generalist Agent for the Webcited by 1,242
  9. 2024GPT-4V(ision) is a Generalist Web Agent, if Groundedcited by 563
  10. 2024AndroidWorld: A Dynamic Benchmarking Environment for Autonomous Agentscited by 468
  11. 2018Reinforcement Learning on Web Interfaces Using Workflow-Guided Explorationcited by 284
  12. 2024GUIOdyssey: A Comprehensive Dataset for Cross-App GUI Navigation on Mobile Devicescited by 194
14 more
  1. 2023Synapse: Trajectory-as-Exemplar Prompting with Memory for Computer Controlcited by 187
  2. 2024WebLINX: Real-World Website Navigation with Multi-Turn Dialoguecited by 180
  3. 2024Autonomous Evaluation and Refinement of Digital Agentscited by 137
  4. 2024AutoWebGLM: A Large Language Model-based Web Navigating Agentcited by 130
  5. 2024MMToM-QA: Multimodal Theory of Mind Question Answeringcited by 85
  6. 2023Corex: Pushing the Boundaries of Complex Reasoning through Multi-Model Collaborationcited by 70
  7. 2024Vision-Language Models Can Self-Improve Reasoning via Reflectioncited by 69
  8. 2023Language Models, Agent Models, and World Models: The LAW for Machine Reasoning and Planningcited by 60
  9. 2024Large Language Models Can Self-Improve At Web Agent Taskscited by 55
  10. 2024AgentOhana: Design Unified Data and Training Pipeline for Effective Agent Learningcited by 53
  11. 2024OpenWebVoyager: Building Multimodal Web Agents via Iterative Real-World Exploration, Feedback and Optimizationcited by 46
  12. 2024BAGEL: Bootstrapping Agents by Guiding Exploration with Languagecited by 34
  13. 2024AgentStore: Scalable Integration of Heterogeneous Agents As Specialized Generalist Computer Assistantcited by 27
  14. 2023Make Prompt-based Black-Box Tuning Colorful: Boosting Model Generalization from Three Orthogonal Perspectivescited by 5

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 76; 2026: 87.

163 papers.

Sort:

By area:

  1. 2025From System 1 to System 2: A Survey of Reasoning Large Language Modelscited by 320
  2. 2025Mobile-Agent-v3: Fundamental Agents for GUI Automationcited by 190
  3. 2025A Survey of Self-Evolving Agents: What, When, How, and Where to Evolve on the Path to Artificial Super Intelligencecited by 170
  4. 2025OpenCUA: Open Foundations for Computer-Use Agentscited by 160
  5. 2025From LLM Reasoning to Autonomous AI Agents: A Comprehensive Reviewcited by 149
  6. 2025Scaling Computer-Use Grounding via User Interface Decomposition and Synthesiscited by 139
  7. 2025GUI-Actor: Coordinate-Free Visual Grounding for GUI Agentscited by 95
  8. 2025UI-Venus Technical Report: Building High-performance UI Agents with RFTcited by 93
  9. 2025SEAgent: Self-Evolving Computer Use Agent with Autonomous Learning from Experiencecited by 69
  10. 2025MMBench-GUI: Hierarchical Multi-Platform Evaluation Framework for GUI Agentscited by 67
  11. 2025GUI-G2: Gaussian Reward Modeling for GUI Groundingcited by 63
  12. 2025ScaleCUA: Scaling Open-Source Computer Use Agents with Cross-Platform Datacited by 60
38 more
  1. 2025ScienceBoard: Evaluating Multimodal Autonomous Agents in Realistic Scientific Workflowscited by 52
  2. 2025MobileGUI-RL: Advancing Mobile GUI Agent through Reinforcement Learning in Online Environmentcited by 52
  3. 2025ComputerRL: Scaling End-to-End Online Reinforcement Learning for Computer Use Agentscited by 50
  4. 2026Code as Agent Harnesscited by 47
  5. 2025AgentSynth: Scalable Task Generation for Generalist Computer-Use Agentscited by 46
  6. 2025ZeroGUI: Automating Online GUI Learning at Zero Human Costcited by 44
  7. 2025AgentCPM-GUI: Building Mobile-Use Agents with Reinforcement Fine-Tuningcited by 44
  8. 2025CapArena: Benchmarking and Analyzing Detailed Image Captioning in the LLM Eracited by 41
  9. 2026UI-Venus-1.5 Technical Reportcited by 41
  10. 2025UI-Genie: A Self-Improving Approach for Iteratively Boosting MLLM-based Mobile GUI Agentscited by 34
  11. 2025Go-Browse: Training Web Agents with Structured Explorationcited by 33
  12. 2025LLM-Powered GUI Agents in Phone Automation: Surveying Progress and Prospectscited by 32
  13. 2025UI-S1: Advancing GUI Automation via Semi-online Reinforcement Learningcited by 32
  14. 2026InfiniteWeb: Scalable Web Environment Synthesis for GUI Agent Trainingcited by 27
  15. 2026Gym-Anything: Turn any Software into an Agent Environmentcited by 27
  16. 2025GUI-Shepherd: Reliable Process Reward and Verification for Long-Sequence GUI Taskscited by 26
  17. 2025OS-Kairos: Adaptive Interaction for MLLM-Powered GUI Agentscited by 25
  18. 2025Genius: A Generalizable and Purely Unsupervised Self-Training Framework For Advanced Reasoningcited by 24
  19. 2026WebXSkill: Skill Learning for Autonomous Web Agentscited by 24
  20. 2025GUI-Reflection: Empowering Multimodal GUI Models with Self-Reflection Behaviorcited by 23
  21. 2026MobileDreamer: Generative Sketch World Model for GUI Agentcited by 23
  22. 2025VideoAgentTrek: Computer Use Pretraining from Unlabeled Videoscited by 22
  23. 2025ProgRM: Build Better GUI Agents with Progress Rewardscited by 21
  24. 2025WebSynthesis: World-Model-Guided MCTS for Efficient WebUI-Trajectory Synthesiscited by 21
  25. 2026OS-Symphony: A Holistic Framework for Robust and Generalist Computer-Using Agentcited by 21
  26. 2025Agent-RewardBench: Towards a Unified Benchmark for Reward Modeling across Perception, Planning, and Safety in Real-World Multimodal Agentscited by 20
  27. 2025OS-Sentinel: Towards Safety-Enhanced Mobile GUI Agents via Hybrid Validation in Realistic Workflowscited by 19
  28. 2025Efficient Agent Training for Computer Usecited by 18
  29. 2025LLMs as Scalable, General-Purpose Simulators For Evolving Digital Agent Trainingcited by 18
  30. 2025WebLeaper: Empowering Efficiency and Efficacy in WebAgent via Enabling Info-Rich Seekingcited by 18
  31. 2025WebCoT: Enhancing Web Agent Reasoning by Reconstructing Chain-of-Thought in Reflection, Branching, and Rollbackcited by 17
  32. 2025Can LLMs Help You at Work? A Sandbox for Evaluating LLM Agents in Enterprise Environmentscited by 17
  33. 2025LPO: Towards Accurate GUI Agent Interaction via Location Preference Optimizationcited by 16
  34. 2026GUI-Libra: Training Native GUI Agents to Reason and Act with Action-aware Supervision and Partially Verifiable RLcited by 16
  35. 2025EchoTrail-GUI: Building Actionable Memory for GUI Agents via Critic-Guided Self-Explorationcited by 15
  36. 2025SynthAgent: Adapting Web Agents with Synthetic Supervisioncited by 14
  37. 2026Don't Act Blindly: Robust GUI Automation via Action-Effect Verification and Self-Correctioncited by 14
  38. 2025Standard Benchmarks Fail -- Auditing LLM Agents in Finance Must Prioritize Riskcited by 13

50 most cited of 163 All 163 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Learn from Move: the Next Step for GUI Agents
  2. 2025TongUI: Internet-Scale Trajectories from Multimodal Web Tutorials for Generalized GUI Agents
  3. 2026GUI-GENESIS: Automated Synthesis of Efficient Environments with Verifiable Rewards for GUI Agent Post-Training
  4. 2026Towards Omni-dimensional GUI Agent Navigation with Masked Trajectory Prediction
  5. 2025Guiding VLM Agents with Process Rewards at Inference Time for GUI Navigation
  6. 2026GUI-CC: Benchmarking Contextual Consistency of GUI World Models as Agent Environments
  7. 2024AgentTrek: Agent Trajectory Synthesis via Guiding Replay with Web Tutorials
  8. 2025UIPro: Unleashing Superior Interaction Capability For GUI Agents