# OSExpert: Computer-Use Agents Learning Professional Skills via Exploration

arXiv: https://arxiv.org/abs/2603.07978 · HTML: https://mlsanity.com/paper/2603.07978/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Placed in GUI and mobile agents, one of 16 directions in LLM agents and tool use.
- Builds on 18 of its 29 references that are in the map.
- First posted on arXiv 9 Mar 2026.

## Placed in

- LLM agents and tool use / [GUI and mobile agents](https://mlsanity.com/agents/gui-and-mobile-agents/)

## Builds on (most cited first, 18 of 18)

- [Efficient Memory Management for Large Language Model Serving with PagedAttention](https://mlsanity.com/paper/2309.06180/) (2023)
- [WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agents](https://mlsanity.com/paper/2207.01206/) (2022)
- [OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments](https://mlsanity.com/paper/2404.07972/) (2024)
- [TextWorld: A Learning Environment for Text-based Games](https://mlsanity.com/paper/1806.11532/) (2018)
- [Reinforcement Learning on Web Interfaces Using Workflow-Guided Exploration](https://mlsanity.com/paper/1802.08802/) (2018)
- [Windows Agent Arena: Evaluating Multi-Modal OS Agents at Scale](https://mlsanity.com/paper/2409.08264/) (2024)
- [UI-TARS-2 Technical Report: Advancing GUI Agent with Multi-Turn Reinforcement Learning](https://mlsanity.com/paper/2509.02544/) (2025)
- [Agent S: An Open Agentic Framework that Uses Computers Like a Human](https://mlsanity.com/paper/2410.08164/) (2024)
- [Interactive Fiction Games: A Colossal Adventure](https://mlsanity.com/paper/1909.05398/) (2019)
- [Agent S2: A Compositional Generalist-Specialist Framework for Computer Use Agents](https://mlsanity.com/paper/2504.00906/) (2025)
- [Mobile-Agent-E: Self-Evolving Mobile Assistant for Complex Tasks](https://mlsanity.com/paper/2501.11733/) (2025)
- [OfficeBench: Benchmarking Language Agents across Multiple Applications for Office Automation](https://mlsanity.com/paper/2407.19056/) (2024)
- [SEAgent: Self-Evolving Computer Use Agent with Autonomous Learning from Experience](https://mlsanity.com/paper/2508.04700/) (2025)
- [WebEvolver: Enhancing Web Agent Self-Improvement with Coevolving World Model](https://mlsanity.com/paper/2504.21024/) (2025)
- [OSWorld-Human: Benchmarking the Efficiency of Computer-Use Agents](https://mlsanity.com/paper/2506.16042/) (2025)
- [OS Agents: A Survey on MLLM-based Agents for General Computing Devices Use](https://mlsanity.com/paper/2508.04482/) (2025)
- [macOSWorld: A Multilingual Interactive Benchmark for GUI Agents](https://mlsanity.com/paper/2506.04135/) (2025)
- [BEARCUBS: A benchmark for computer-using web agents](https://mlsanity.com/paper/2503.07919/) (2025)

## Built on by (most cited first, 7 of 7)

- [MementoGUI: Learning Agentic Multimodal Memory Control for Long-Horizon GUI Agents](https://mlsanity.com/paper/2605.18652/) (2026)
- [Augmenting Interface Usability Heuristics for Reliable Computer-Use Agents](https://mlsanity.com/paper/2605.02729/) (2026)
- [VISUALSKILL: Multimodal Skills for Computer-Use Agents](https://mlsanity.com/paper/2606.18448/) (2026)
- [Learn from Weaknesses: Automated Domain Specialization for Small Computer-Use Agents](https://mlsanity.com/paper/2605.28775/) (2026)
- [BioInsight: Multi-Agent Orchestration for Interactive Biomedical Knowledge Discovery](https://mlsanity.com/paper/2606.20997/) (2026)
- [Brick-Composer: Using MLLMs for Assembly with Diverse Bricks](https://mlsanity.com/paper/2606.05445/) (2026)
- [Towards a Systems Foundation for Agentic Skills: Architecture, Lifecycle, and Security](https://mlsanity.com/paper/2608.29596/) (2026)

## Abstract (arXiv)

General-purpose computer-use agents have shown impressive performance across diverse digital environments. However, our new benchmark, OSExpert-Eval, indicates they remain far less helpful than human experts. Although inference-time scaling enables adaptation, these agents complete complex tasks inefficiently with degraded performance, transfer poorly to unseen UIs, and struggle with fine-grained action sequences. To solve the problem, we introduce a GUI-based depth-first search (GUI-DFS) exploration algorithm to comprehensively explore and verify an environment's unit functions. The agent then exploits compositionality between unit skills to self-construct a curriculum for composite tasks. To support fine-grained actions, we curate a database of action primitives for agents to discover during exploration; these are saved as a skill set once the exploration is complete. We use the learned skills to improve the agent's performance and efficiency by (1) enriching agents with ready-to-use procedural knowledge, allowing them to plan only once for long trajectories and generate accurate actions, and (2) enabling them to end inference-time scaling earlier by realizing their boundary of capabilities. Extensive experiments show that our environment-learned agent takes a meaningful step toward expert-level computer use, achieving a around 20 percent performance gain on OSExpert-Eval and closing the efficiency gap to humans by around 80 percent