mlsanityData to 14 Oct 2026

Navi-plus: Managing Ambiguous GUI Navigation Tasks with Follow-up Questions

arXiv 2503.24180 ↗First posted 31 Mar 2025cs.CV cs.HCcited by 8

Not on the map. Why?

Abstract (arXiv)

Graphical user interfaces (GUI) automation agents are emerging as powerful tools, enabling humans to accomplish increasingly complex tasks on smart devices. However, users often inadvertently omit key information when conveying tasks, which hinders agent performance in the current agent paradigm that does not support immediate user intervention. To address this issue, we introduce a Self-Correction GUI Navigation task that incorporates interactive information completion capabilities within GUI agents. We developed the Navi-plus dataset with GUI follow-up question-answer pairs, alongside a Dual-Stream Trajectory Evaluation method to benchmark this new capability. Our results show that agents equipped with the ability to ask GUI follow-up questions can fully recover their performance when faced with ambiguous user tasks.

Key facts. Cited by 8 papers in the map (data through 14 Oct 2026). Builds on 39 of its 49 references that are in the map. First posted on arXiv 31 Mar 2025.

Builds on

39 of 49 references.

References · 39

  1. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  2. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
  3. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,182
  4. 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scalingcited by 1,599
  5. 2023Mind2Web: Towards a Generalist Agent for the Webcited by 1,242
  6. 2024How Far Are We to GPT-4V? Closing the Gap to Commercial Multimodal Models with Open-Source Suitescited by 1,050
  7. 2023CogAgent: A Visual Language Model for GUI Agentscited by 704
  8. 2025UI-TARS: Pioneering Automated GUI Interaction with Native Agentscited by 610
  9. 2024SeeClick: Harnessing GUI Grounding for Advanced Visual GUI Agentscited by 567
  10. 2024GPT-4V(ision) is a Generalist Web Agent, if Groundedcited by 563
  11. 2024OS-ATLAS: A Foundation Action Model for Generalist GUI Agentscited by 378
  12. 2023ToRA: A Tool-Integrated Reasoning Agent for Mathematical Problem Solvingcited by 282
27 more
  1. 2024Aguvis: Unified Pure Vision Agents for Autonomous GUI Interactioncited by 278
  2. 2023Patch n' Pack: NaViT, a Vision Transformer for any Aspect Ratio and Resolutioncited by 229
  3. 2024GUIOdyssey: A Comprehensive Dataset for Cross-App GUI Navigation on Mobile Devicescited by 194
  4. 2024WebLINX: Real-World Website Navigation with Multi-Turn Dialoguecited by 180
  5. 2024Ferret-UI: Grounded Mobile UI Understanding with Multimodal LLMscited by 176
  6. 2025UI-R1: Enhancing Efficient Action Prediction of GUI Agents by Reinforcement Learningcited by 168
  7. 2024OS-Genesis: Automating GUI Agent Trajectory Construction via Reverse Task Synthesiscited by 163
  8. 2024Android in the Zoo: Chain-of-Action-Thought for GUI Agentscited by 159
  9. 2024OmniACT: A Dataset and Benchmark for Enabling Multimodal Generalist Autonomous Agents for Desktop and Webcited by 157
  10. 2021FILM: Following Instructions in Language with Modular Methodscited by 121
  11. 2024WebCanvas: Benchmarking Web Agents in Online Environmentscited by 121
  12. 2021Screen2Words: Automatic Mobile UI Summarization with Multimodal Learningcited by 120
  13. 2024GUICourse: From General Vision Language Models to Versatile GUI Agentscited by 114
  14. 2024AMEX: Android Multi-annotation Expo Dataset for Mobile GUI Agentscited by 113
  15. 2019Help, Anna! Visual Navigation with Natural Multimodal Assistance via Retrospective Curiosity-Encouraging Imitation Learningcited by 100
  16. 2021WebSRC: A Dataset for Web-Based Structural Reading Comprehensioncited by 100
  17. 2020Widget Captioning: Generating Natural Language Description for Mobile User Interface Elementscited by 95
  18. 2025InfiGUIAgent: A Multimodal Generalist GUI Agent with Native Reasoning and Reflectioncited by 67
  19. 2022A Dataset for Interactive Vision-Language Navigation with Unknown Command Feasibilitycited by 61
  20. 2024Ferret-UI 2: Mastering Universal User Interface Understanding Across Platformscited by 51
  21. 2024On the Multi-turn Instruction Following for Conversational Web Agentscited by 43
  22. 2023L2MAC: Large Language Model Automatic Computer for Extensive Code Generationcited by 30
  23. 2024CLARINET: Augmenting Language Models to Ask Clarification Questions for Retrievalcited by 20
  24. 2025SpiritSight Agent: Advanced GUI Agent with One Lookcited by 18
  25. 2025MP-GUI: Modality Perception with MLLMs for GUI Understandingcited by 12
  26. 2024I Need Help! Evaluating LLM's Ability to Ask for Users' Support: A Case Study on Text-to-SQL Generationcited by 11
  27. 2024Large Language Models for Orchestrating Bimanual Robotscited by 10

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 5; 2026: 3.

8 papers.

  1. 2026PersonalAlign: Hierarchical Implicit Intent Alignment for Personalized GUI Agent with Long-Term User-Centric Recordscited by 16
  2. 2025VeriOS: Query-Driven Proactive Human-Agent-GUI Interaction for Trustworthy OS Agentscited by 15
  3. 2025Quick on the Uptake: Eliciting Implicit Intents from Human Demonstrations for Personalized Mobile-Use Agentscited by 11
  4. 2025Say One Thing, Do Another? Diagnosing Reasoning-Execution Gaps in VLM-Powered Mobile-Use Agentscited by 4
  5. 2025Interaction-Driven Browsing: A Human-in-the-Loop Conceptual Framework Informed by Human Web Browsing for Browser-Using Agentscited by 3
  6. 2025Agent-ScanKit: Unraveling Memory and Reasoning of Multimodal Agents via Sensitivity Perturbationscited by 3
  7. 2026Software Engineering for and with GUI Agentcited by 1
  8. 2026ADeptS-Bench: Measuring the Trustworthiness of Computer Use Agents Across Devicescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Falcon-UI: Understanding GUI Before Following User Instructions
  2. 2025A Survey on (M)LLM-Based GUI Agents
  3. 2025ReInAgent: A Context-Aware GUI Agent Enabling Human-in-the-Loop Mobile Task Navigation
  4. 2026DocOS: Towards Proactive Document-Guided Actions in GUI Agents
  5. 2026Learning Reliable GUI Agents under Imperfect Priors
  6. 2026EvoGUI: An Evolution-Aware Benchmark for GUI State-Transition Understanding
  7. 2025GUI-explorer: Autonomous Exploration and Mining of Transition-aware Knowledge for GUI Agent
  8. 2024GUI Agents: A Survey