# Efficient Agent Training for Computer Use

arXiv: https://arxiv.org/abs/2505.13909 · HTML: https://mlsanity.com/paper/2505.13909/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- Placed in GUI and mobile agents, one of 16 directions in LLM agents and tool use.
- Builds on 26 of its 56 references that are in the map.
- First posted on arXiv 20 May 2025.

## Placed in

- LLM agents and tool use / [GUI and mobile agents](https://mlsanity.com/agents/gui-and-mobile-agents/)

## Builds on (most cited first, 26 of 26)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [ReAct: Synergizing Reasoning and Acting in Language Models](https://mlsanity.com/paper/2210.03629/) (2022)
- [Self-Instruct: Aligning Language Models with Self-Generated Instructions](https://mlsanity.com/paper/2212.10560/) (2022)
- [Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters](https://mlsanity.com/paper/2408.03314/) (2024)
- [s1: Simple test-time scaling](https://mlsanity.com/paper/2501.19393/) (2025)
- [Large Language Monkeys: Scaling Inference Compute with Repeated Sampling](https://mlsanity.com/paper/2407.21787/) (2024)
- [Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?](https://mlsanity.com/paper/2504.13837/) (2025)
- [UI-TARS: Pioneering Automated GUI Interaction with Native Agents](https://mlsanity.com/paper/2501.12326/) (2025)
- [LIMO: Less is More for Reasoning](https://mlsanity.com/paper/2502.03387/) (2025)
- [Navigating the Digital World as Humans Do: Universal Visual Grounding for GUI Agents](https://mlsanity.com/paper/2410.05243/) (2024)
- [OS-ATLAS: A Foundation Action Model for Generalist GUI Agents](https://mlsanity.com/paper/2410.23218/) (2024)
- [Training Software Engineering Agents and Verifiers with SWE-Gym](https://mlsanity.com/paper/2412.21139/) (2024)
- [Aguvis: Unified Pure Vision Agents for Autonomous GUI Interaction](https://mlsanity.com/paper/2412.04454/) (2024)
- [Windows Agent Arena: Evaluating Multi-Modal OS Agents at Scale](https://mlsanity.com/paper/2409.08264/) (2024)
- [OS-Copilot: Towards Generalist Computer Agents with Self-Improvement](https://mlsanity.com/paper/2402.07456/) (2024)
- [OmniParser for Pure Vision Based GUI Agent](https://mlsanity.com/paper/2408.00203/) (2024)
- [On the Effects of Data Scale on UI Control Agents](https://mlsanity.com/paper/2406.03679/) (2024)
- [GUIOdyssey: A Comprehensive Dataset for Cross-App GUI Navigation on Mobile Devices](https://mlsanity.com/paper/2406.08451/) (2024)
- [Agent S: An Open Agentic Framework that Uses Computers Like a Human](https://mlsanity.com/paper/2410.08164/) (2024)
- [OS-Genesis: Automating GUI Agent Trajectory Construction via Reverse Task Synthesis](https://mlsanity.com/paper/2412.19723/) (2024)
- [AgentTrek: Agent Trajectory Synthesis via Guiding Replay with Web Tutorials](https://mlsanity.com/paper/2412.09605/) (2024)
- [O1 Replication Journey -- Part 2: Surpassing O1-preview through Simple Distillation, Big Progress or Bitter Lesson?](https://mlsanity.com/paper/2411.16489/) (2024)
- [Large Language Models Can Self-Improve At Web Agent Tasks](https://mlsanity.com/paper/2405.20309/) (2024)
- [Synatra: Turning Indirect Knowledge into Direct Demonstrations for Digital Agents at Scale](https://mlsanity.com/paper/2409.15637/) (2024)
- [PC-Agent: A Hierarchical Multi-Agent Collaboration Framework for Complex Task Automation on PC](https://mlsanity.com/paper/2502.14282/) (2025)
- [AgentStore: Scalable Integration of Heterogeneous Agents As Specialized Generalist Computer Assistant](https://mlsanity.com/paper/2410.18603/) (2024)

## Built on by (most cited first, 18 of 18)

- [Thinking with Images for Multimodal Reasoning: Foundations, Methods, and Future Frontiers](https://mlsanity.com/paper/2506.23918/) (2025)
- [ScaleCUA: Scaling Open-Source Computer Use Agents with Cross-Platform Data](https://mlsanity.com/paper/2509.15221/) (2025)
- [ComputerRL: Scaling End-to-End Online Reinforcement Learning for Computer Use Agents](https://mlsanity.com/paper/2508.14040/) (2025)
- [Gym-Anything: Turn any Software into an Agent Environment](https://mlsanity.com/paper/2604.06126/) (2026)
- [AutoWebWorld: Synthesizing Infinite Verifiable Web Environments via Finite State Machines](https://mlsanity.com/paper/2602.14296/) (2026)
- [From Question Answering to Task Completion: A Survey on Agent System and Harness Design](https://mlsanity.com/paper/2606.20683/) (2026)
- [Autonomous Continual Learning for Environment Adaptation of Computer-Use Agents](https://mlsanity.com/paper/2602.10356/) (2026)
- [Surfer 2: The Next Generation of Cross-Platform Computer Use Agents](https://mlsanity.com/paper/2510.19949/) (2025)
- [Learning from Failure: Inference-Time Self-Improvement for Computer-Use Agents](https://mlsanity.com/paper/2606.31270/) (2026)
- [NaturalGAIA: A Verifiable Benchmark and Hierarchical Framework for Long-Horizon GUI Tasks](https://mlsanity.com/paper/2508.01330/) (2025)
- [Scalable Environments Drive Generalizable Agents](https://mlsanity.com/paper/2605.18181/) (2026)
- [ProCUA-SFT Technical Report](https://mlsanity.com/paper/2606.17321/) (2026)
- [Instruction Agent: Enhancing Agent with Expert Demonstration](https://mlsanity.com/paper/2509.07098/) (2025)
- [Learn from Weaknesses: Automated Domain Specialization for Small Computer-Use Agents](https://mlsanity.com/paper/2605.28775/) (2026)
- [ChainWorld: Composing Long-Horizon Desktop Workloads from Atomic OSWorld Tasks](https://mlsanity.com/paper/2606.21654/) (2026)
- [WM-R1: Training GUI Agents to Reason and leverage World Models with Reinforcement Learning](https://mlsanity.com/paper/2608.27508/) (2026)
- [Efficient Test-Time Adaptation through Human-AI Interaction](https://mlsanity.com/paper/2609.04141/) (2026)
- [Imagine to Act: High-Fidelity Data Synthesis via Image Editing World Model for Scalable GUI Agent Training](https://mlsanity.com/paper/2610.05861/) (2026)

## Abstract (arXiv)

Scaling up high-quality trajectory data has long been a critical bottleneck for developing human-like computer use agents. We introduce PC Agent-E, an efficient agent training framework that significantly reduces reliance on large-scale human demonstrations. Starting with just 312 human-annotated computer use trajectories, we further augment them by synthesizing diverse alternative action decisions with Claude 3.7 Sonnet. Trained on these enriched trajectories, our PC Agent-E model achieved a remarkable 141 relative improvement, and even surpassed the Claude 3.7 Sonnet by 10% in relative terms on WindowsAgentArena-V2, an improved benchmark we also released. By integrating robust human computer use skills with automated AI data synthesis capabilities, our method not only brought substantial improvements over training on human trajectories alone, but also significantly surpassed direct distillation from Claude 3.7 Sonnet. Code, data and models are available at https://github.com/GAIR-NLP/PC-Agent-E