# DocOS: Towards Proactive Document-Guided Actions in GUI Agents

arXiv: https://arxiv.org/abs/2605.18048 · HTML: https://mlsanity.com/paper/2605.18048/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in GUI and mobile agents, one of 16 directions in LLM agents and tool use.
- Builds on 31 of its 39 references that are in the map.
- First posted on arXiv 18 May 2026.

## Placed in

- LLM agents and tool use / [GUI and mobile agents](https://mlsanity.com/agents/gui-and-mobile-agents/)

## Builds on (most cited first, 30 of 31)

- [WebArena: A Realistic Web Environment for Building Autonomous Agents](https://mlsanity.com/paper/2307.13854/) (2023)
- [WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agents](https://mlsanity.com/paper/2207.01206/) (2022)
- [OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments](https://mlsanity.com/paper/2404.07972/) (2024)
- [A Survey on Multimodal Large Language Models](https://mlsanity.com/paper/2306.13549/) (2023)
- [UI-TARS: Pioneering Automated GUI Interaction with Native Agents](https://mlsanity.com/paper/2501.12326/) (2025)
- [OS-ATLAS: A Foundation Action Model for Generalist GUI Agents](https://mlsanity.com/paper/2410.23218/) (2024)
- [Aguvis: Unified Pure Vision Agents for Autonomous GUI Interaction](https://mlsanity.com/paper/2412.04454/) (2024)
- [ScreenSpot-Pro: GUI Grounding for Professional High-Resolution Computer Use](https://mlsanity.com/paper/2504.07981/) (2025)
- [Mobile-Agent-v3: Fundamental Agents for GUI Automation](https://mlsanity.com/paper/2508.15144/) (2025)
- [OmniACT: A Dataset and Benchmark for Enabling Multimodal Generalist Autonomous Agents for Desktop and Web](https://mlsanity.com/paper/2402.17553/) (2024)
- [WebCanvas: Benchmarking Web Agents in Online Environments](https://mlsanity.com/paper/2406.12373/) (2024)
- [GUI Agents with Foundation Models: A Comprehensive Survey](https://mlsanity.com/paper/2411.04890/) (2024)
- [Towards Conversational Recommendation over Multi-Type Dialogs](https://mlsanity.com/paper/2005.03954/) (2020)
- [GUI Agents: A Survey](https://mlsanity.com/paper/2412.13501/) (2024)
- [AutoGLM: Autonomous Foundation Agents for GUIs](https://mlsanity.com/paper/2411.00820/) (2024)
- [MAI-UI Technical Report: Real-World Centric Foundation GUI Agents](https://mlsanity.com/paper/2512.22047/) (2025)
- [Mind2Web 2: Evaluating Agentic Search with Agent-as-a-Judge](https://mlsanity.com/paper/2506.21506/) (2025)
- [MobileAgentBench: An Efficient and User-Friendly Benchmark for Mobile LLM Agents](https://mlsanity.com/paper/2406.08184/) (2024)
- [DuRecDial 2.0: A Bilingual Parallel Corpus for Conversational Recommendation](https://mlsanity.com/paper/2109.08877/) (2021)
- [GUI-World: A Video Benchmark and Dataset for Multimodal GUI-oriented Understanding](https://mlsanity.com/paper/2406.10819/) (2024)
- [TongUI: Internet-Scale Trajectories from Multimodal Web Tutorials for Generalized GUI Agents](https://mlsanity.com/paper/2504.12679/) (2025)
- [CRAB: Cross-environment Agent Benchmark for Multimodal Language Model Agents](https://mlsanity.com/paper/2407.01511/) (2024)
- [Step-GUI Technical Report](https://mlsanity.com/paper/2512.15431/) (2025)
- [VideoGUI: A Benchmark for GUI Automation from Instructional Videos](https://mlsanity.com/paper/2406.10227/) (2024)
- [GUI-explorer: Autonomous Exploration and Mining of Transition-aware Knowledge for GUI Agent](https://mlsanity.com/paper/2505.16827/) (2025)
- [Retrieval-augmented GUI Agents with Generative Guidelines](https://mlsanity.com/paper/2509.24183/) (2025)
- [MidMed: Towards Mixed-Type Dialogues for Medical Consultation](https://mlsanity.com/paper/2306.02923/) (2023)
- [Where to Go for the Holidays: Towards Mixed-Type Dialogs for Clarification of User Goals](https://mlsanity.com/paper/2204.07299/) (2022)
- [XIFBench: Evaluating Large Language Models on Multilingual Instruction Following](https://mlsanity.com/paper/2503.07539/) (2025)
- [TransBench: Breaking Barriers for Transferable Graphical User Interface Agents in Dynamic Digital Environments](https://mlsanity.com/paper/2505.17629/) (2025)

## Built on by (most cited first, 1 of 1)

- [Behavior2Trip: Towards Personalized Travel Planning via User Behavior Trajectory](https://mlsanity.com/paper/2608.26807/) (2026)

## Abstract (arXiv)

While Graphical User Interface (GUI) agents have shown promising performance in automated device interaction, they primarily depend on static parametric knowledge from pre-training or instruction tuning. This reliance fundamentally limits their ability to handle long-tailed tasks that require explicit procedural knowledge absent from model parameters, often forcing agents to resort to inefficient and brittle trial-and-error exploration. To mitigate this limitation, we introduce Proactive Document-Guided Action for GUI agents in dynamic, open-web environments, a novel paradigm that mirrors human problem-solving by enabling agents to autonomously search for relevant documentation to resolve long-tailed tasks. To evaluate agents' capability in this paradigm, we propose DocOS, a benchmark designed to assess document-guided problem solving in fully interactive environments. DocOS requires agents to autonomously navigate a web browser, locate relevant online documentation, comprehend procedural instructions, and faithfully ground them into executable GUI actions. Extensive experiments reveal that progress is strictly constrained by dual bottlenecks: agents struggle to reliably locate relevant information during proactive search and frequently fail to faithfully ground retrieved instructions into precise actions, pointing toward document-guided interaction as a crucial pathway for enabling self-evolving GUI agents in dynamic environments.