OSExpert: Computer-Use Agents Learning Professional Skills via Exploration
On the map: LLM agents and tool use → GUI and mobile agents
Abstract (arXiv)
General-purpose computer-use agents have shown impressive performance across diverse digital environments. However, our new benchmark, OSExpert-Eval, indicates they remain far less helpful than human experts. Although inference-time scaling enables adaptation, these agents complete complex tasks inefficiently with degraded performance, transfer poorly to unseen UIs, and struggle with fine-grained action sequences. To solve the problem, we introduce a GUI-based depth-first search (GUI-DFS) exploration algorithm to comprehensively explore and verify an environment's unit functions. The agent then exploits compositionality between unit skills to self-construct a curriculum for composite tasks. To support fine-grained actions, we curate a database of action primitives for agents to discover during exploration; these are saved as a skill set once the exploration is complete. We use the learned skills to improve the agent's performance and efficiency by (1) enriching agents with ready-to-use procedural knowledge, allowing them to plan only once for long trajectories and generate accurate actions, and (2) enabling them to end inference-time scaling earlier by realizing their boundary of capabilities. Extensive experiments show that our environment-learned agent takes a meaningful step toward expert-level computer use, achieving a around 20 percent performance gain on OSExpert-Eval and closing the efficiency gap to humans by around 80 percent
Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Placed in GUI and mobile agents, one of 16 directions in LLM agents and tool use. Builds on 18 of its 29 references that are in the map. First posted on arXiv 9 Mar 2026.
Builds on
18 of 29 references.
Closest ancestors · 9
Same direction, newest first
- 2025UI-TARS-2 Technical Report: Advancing GUI Agent with Multi-Turn Reinforcement Learning2 Sep
- 2025OS Agents: A Survey on MLLM-based Agents for General Computing Devices Use6 Aug
- 2025OSWorld-Human: Benchmarking the Efficiency of Computer-Use Agents19 Jun
- 2025macOSWorld: A Multilingual Interactive Benchmark for GUI Agents4 Jun
- 2025Agent S2: A Compositional Generalist-Specialist Framework for Computer Use Agents1 Apr
- 2025Mobile-Agent-E: Self-Evolving Mobile Assistant for Complex Tasks20 Jan
- 2024Agent S: An Open Agentic Framework that Uses Computers Like a Human10 Oct
- 2024Windows Agent Arena: Evaluating Multi-Modal OS Agents at Scale12 Sep
- 2024OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments11 Apr
Other references · 9
- 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
- 2022WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agentscited by 1,197
- 2018TextWorld: A Learning Environment for Text-based Gamescited by 304
- 2018Reinforcement Learning on Web Interfaces Using Workflow-Guided Explorationcited by 282
- 2019Interactive Fiction Games: A Colossal Adventurecited by 165
- 2025SEAgent: Self-Evolving Computer Use Agent with Autonomous Learning from Experiencecited by 68
- 2024OfficeBench: Benchmarking Language Agents across Multiple Applications for Office Automationcited by 66
- 2025WebEvolver: Enhancing Web Agent Self-Improvement with Coevolving World Modelcited by 56
- 2025BEARCUBS: A benchmark for computer-using web agentscited by 29
Built on by
Citing papers per year: 2026: 7.
7 papers.
- 2026MementoGUI: Learning Agentic Multimodal Memory Control for Long-Horizon GUI Agentscited by 9
- 2026Augmenting Interface Usability Heuristics for Reliable Computer-Use Agentscited by 3
- 2026VISUALSKILL: Multimodal Skills for Computer-Use Agentscited by 3
- 2026Learn from Weaknesses: Automated Domain Specialization for Small Computer-Use Agentscited by 1
- 2026BioInsight: Multi-Agent Orchestration for Interactive Biomedical Knowledge Discoverycited by 1
- 2026Brick-Composer: Using MLLMs for Assembly with Diverse Brickscited by 0
- 2026Towards a Systems Foundation for Agentic Skills: Architecture, Lifecycle, and Securitycited by 0
- 2026Towards a Systems Foundation for Agentic Skills: Architecture, Lifecycle, and Security30 Aug 2026
- 2026BioInsight: Multi-Agent Orchestration for Interactive Biomedical Knowledge Discovery19 Jun 2026
- 2026VISUALSKILL: Multimodal Skills for Computer-Use Agents16 Jun 2026
- 2026Brick-Composer: Using MLLMs for Assembly with Diverse Bricks3 Jun 2026
- 2026Learn from Weaknesses: Automated Domain Specialization for Small Computer-Use Agents27 May 2026
- 2026MementoGUI: Learning Agentic Multimodal Memory Control for Long-Horizon GUI Agents18 May 2026
- 2026Augmenting Interface Usability Heuristics for Reliable Computer-Use Agents4 May 2026
Similar, not cited
Close in wording, no citation either way
- 2026MMG2Skill: Can Agents Distill In-the-Wild Guides into Self-Evolving Skills?
- 2025Mirage-1: Augmenting and Updating GUI Agent with Hierarchical Multimodal Skills
- 2026ExpVoyager: Direct Experience Navigation for Dynamic Agent Skill Synthesis
- 2025GUI Exploration Lab: Enhancing Screen Navigation in Agents via Multi-Turn Reinforcement Learning
- 2026OmegaUse: Building a General-Purpose GUI Agent for Autonomous Task Execution
- 2025UltraCUA: A Foundation Model for Computer Use Agents with Hybrid Action
- 2026Empowering GUI Agents via Autonomous Experience Exploration and Hindsight Experience Utilization for Task Planning
- 2026SKT: Skill-Use Training at Scale via Verified Synthetic Data Generation