mlsanityData to 14 Oct 2026

DocOS: Towards Proactive Document-Guided Actions in GUI Agents

arXiv 2605.18048 ↗First posted 18 May 2026cs.AIcited by 1

On the map: LLM agents and tool use → GUI and mobile agents

Abstract (arXiv)

While Graphical User Interface (GUI) agents have shown promising performance in automated device interaction, they primarily depend on static parametric knowledge from pre-training or instruction tuning. This reliance fundamentally limits their ability to handle long-tailed tasks that require explicit procedural knowledge absent from model parameters, often forcing agents to resort to inefficient and brittle trial-and-error exploration. To mitigate this limitation, we introduce Proactive Document-Guided Action for GUI agents in dynamic, open-web environments, a novel paradigm that mirrors human problem-solving by enabling agents to autonomously search for relevant documentation to resolve long-tailed tasks. To evaluate agents' capability in this paradigm, we propose DocOS, a benchmark designed to assess document-guided problem solving in fully interactive environments. DocOS requires agents to autonomously navigate a web browser, locate relevant online documentation, comprehend procedural instructions, and faithfully ground them into executable GUI actions. Extensive experiments reveal that progress is strictly constrained by dual bottlenecks: agents struggle to reliably locate relevant information during proactive search and frequently fail to faithfully ground retrieved instructions into precise actions, pointing toward document-guided interaction as a crucial pathway for enabling self-evolving GUI agents in dynamic environments.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in GUI and mobile agents, one of 16 directions in LLM agents and tool use. Builds on 31 of its 39 references that are in the map. First posted on arXiv 18 May 2026.

Builds on

31 of 39 references.

Closest ancestors · 15

Same direction, newest first

  1. 2025MAI-UI Technical Report: Real-World Centric Foundation GUI Agents26 Dec
  2. 2025Step-GUI Technical Report17 Dec
  3. 2025Mobile-Agent-v3: Fundamental Agents for GUI Automation21 Aug
  4. 2025TransBench: Breaking Barriers for Transferable Graphical User Interface Agents in Dynamic Digital Environments23 May
  5. 2025GUI-explorer: Autonomous Exploration and Mining of Transition-aware Knowledge for GUI Agent22 May
  6. 2025TongUI: Internet-Scale Trajectories from Multimodal Web Tutorials for Generalized GUI Agents17 Apr
  7. 2025ScreenSpot-Pro: GUI Grounding for Professional High-Resolution Computer Use4 Apr
  8. 2025UI-TARS: Pioneering Automated GUI Interaction with Native Agents21 Jan
  9. 2024GUI Agents: A Survey18 Dec
  10. 2024GUI Agents with Foundation Models: A Comprehensive Survey7 Nov
  11. 2024OS-ATLAS: A Foundation Action Model for Generalist GUI Agents30 Oct
  12. 2024AutoGLM: Autonomous Foundation Agents for GUIs28 Oct
3 more
  1. 2024MobileAgentBench: An Efficient and User-Friendly Benchmark for Mobile LLM Agents12 Jun
  2. 2024OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments11 Apr
  3. 2024OmniACT: A Dataset and Benchmark for Enabling Multimodal Generalist Autonomous Agents for Desktop and Web27 Feb

Other references · 16

  1. 2023WebArena: A Realistic Web Environment for Building Autonomous Agentscited by 2,006
  2. 2022WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agentscited by 1,251
  3. 2023A Survey on Multimodal Large Language Modelscited by 909
  4. 2024Aguvis: Unified Pure Vision Agents for Autonomous GUI Interactioncited by 278
  5. 2024WebCanvas: Benchmarking Web Agents in Online Environmentscited by 121
  6. 2020Towards Conversational Recommendation over Multi-Type Dialogscited by 107
  7. 2025Mind2Web 2: Evaluating Agentic Search with Agent-as-a-Judgecited by 76
  8. 2021DuRecDial 2.0: A Bilingual Parallel Corpus for Conversational Recommendationcited by 46
  9. 2024GUI-World: A Video Benchmark and Dataset for Multimodal GUI-oriented Understandingcited by 43
  10. 2024CRAB: Cross-environment Agent Benchmark for Multimodal Language Model Agentscited by 39
  11. 2024VideoGUI: A Benchmark for GUI Automation from Instructional Videoscited by 35
  12. 2025Retrieval-augmented GUI Agents with Generative Guidelinescited by 15
4 more
  1. 2023MidMed: Towards Mixed-Type Dialogues for Medical Consultationcited by 13
  2. 2022Where to Go for the Holidays: Towards Mixed-Type Dialogs for Clarification of User Goalscited by 12
  3. 2025XIFBench: Evaluating Large Language Models on Multilingual Instruction Followingcited by 11
  4. 2025What Limits Virtual Agent Application? OmniBench: A Scalable Multi-Dimensional Benchmark for Essential Virtual Agent Capabilitiescited by 8

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

Similar, not cited

Close in wording, no citation either way

  1. 2026OmegaUse: Building a General-Purpose GUI Agent for Autonomous Task Execution
  2. 2026Learn from Move: the Next Step for GUI Agents
  3. 2025A Survey on (M)LLM-Based GUI Agents
  4. 2024Agent S: An Open Agentic Framework that Uses Computers Like a Human
  5. 2025Learning, Reasoning, Refinement: A Framework for Kahneman's Dual-System Intelligence in GUI Agents
  6. 2026Learning Reliable GUI Agents under Imperfect Priors
  7. 2023You Only Look at Screens: Multimodal Chain-of-Action Agents
  8. 2025Computer-Use Agents as Judges for Generative User Interface