mlsanityData to 7 Oct 2026

Think Twice, Click Once: Enhancing GUI Grounding via Fast and Slow Systems

arXiv 2503.06470 ↗First posted 9 Mar 2025cs.AI cs.CL cs.CVcited by 23

Not on the map. Why?

Abstract (arXiv)

Humans can flexibly switch between different modes of thinking based on task complexity: from rapid intuitive judgments to in-depth analytical understanding. However, current Graphical User Interface (GUI) grounding systems which locate interface elements based on natural language instructions rely solely on immediate prediction without reasoning, struggling to understand complex interface layouts with nested structures and hierarchical relationships, limiting their effectiveness on complex interfaces. Inspired by human dual-system cognition, we present Focus, a novel GUI grounding framework that combines fast prediction with systematic analysis. The framework dynamically switches between rapid and deliberate processing through an adaptive system switching based on task complexity, optimizing both efficiency and accuracy. Focus decomposes grounding into progressive stages: interface summarization, visual focused analysis, and precise coordinate prediction. This structured decomposition enables systematic understanding of both interface layouts and visual relationships. Extensive experiments show that Focus achieves state-of-the-art performance using only 300K of the training data with a 2B parameter model compared to existing approaches. Focus demonstrates superior performance particularly in complex GUI scenarios, achieving 77.4% average accuracy on ScreenSpot and 13.3% on the more challenging ScreenSpot-Pro. Our analysis reveals the effectiveness of this dual-system approach while demonstrating its potential for improving complex GUI interaction scenarios.

Key facts. Cited by 23 papers in the map (data through 7 Oct 2026). Builds on 32 of its 42 references that are in the map. First posted on arXiv 9 Mar 2025.

Builds on

32 of 42 references.

References · 32

  1. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,151
  2. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,274
  3. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
  4. 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,534
  5. 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,179
  6. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,722
  7. 2024DeepSeek-VL: Towards Real-World Vision-Language Understandingcited by 718
  8. 2023CogAgent: A Visual Language Model for GUI Agentscited by 692
  9. 2024Do NOT Think That Much for 2+3=? On the Overthinking of o1-Like LLMscited by 564
  10. 2024DeepSeek-VL2: Mixture-of-Experts Vision-Language Models for Advanced Multimodal Understandingcited by 559
  11. 2024SeeClick: Harnessing GUI Grounding for Advanced Visual GUI Agentscited by 553
  12. 2023AppAgent: Multimodal Agents as Smartphone Userscited by 407
20 more
  1. 2024Navigating the Digital World as Humans Do: Universal Visual Grounding for GUI Agentscited by 373
  2. 2024OS-ATLAS: A Foundation Action Model for Generalist GUI Agentscited by 370
  3. 2023Set-of-Mark Prompting Unleashes Extraordinary Visual Grounding in GPT-4Vcited by 291
  4. 2024Mobile-Agent: Autonomous Multi-Modal Mobile Device Agent with Visual Perceptioncited by 283
  5. 2024Aguvis: Unified Pure Vision Agents for Autonomous GUI Interactioncited by 271
  6. 2024ShowUI: One Vision-Language-Action Model for GUI Visual Agentcited by 226
  7. 2024OS-Copilot: Towards Generalist Computer Agents with Self-Improvementcited by 203
  8. 2024Mobile-Agent-v2: Mobile Device Operation Assistant with Effective Navigation via Multi-Agent Collaborationcited by 187
  9. 2024Ferret-UI: Grounded Mobile UI Understanding with Multimodal LLMscited by 173
  10. 2020PP-OCR: A Practical Ultra Lightweight OCR Systemcited by 147
  11. 2024UFO: A UI-Focused Agent for Windows OS Interactioncited by 147
  12. 2024Aria-UI: Visual Grounding for GUI Instructionscited by 136
  13. 2024AutoWebGLM: A Large Language Model-based Web Navigating Agentcited by 130
  14. 2025Mobile-Agent-E: Self-Evolving Mobile Assistant for Complex Taskscited by 128
  15. 2024GUICourse: From General Vision Language Models to Versatile GUI Agentscited by 112
  16. 2024ScreenAgent: A Vision Language Model-driven Computer Control Agentcited by 101
  17. 2024CoCo-Agent: A Comprehensive Cognitive MLLM Agent for Smartphone GUI Automationcited by 70
  18. 2025InfiGUIAgent: A Multimodal Generalist GUI Agent with Native Reasoning and Reflectioncited by 67
  19. 2024The Dawn of GUI Agent: A Preliminary Case Study with Claude 3.5 Computer Usecited by 62
  20. 2024PC Agent: While You Sleep, AI Works -- A Cognitive Journey into Digital Worldcited by 12

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 12; 2026: 11.

23 papers.

Sort:

By area:

  1. 2025UI-Venus Technical Report: Building High-performance UI Agents with RFTcited by 90
  2. 2025GUI-Actor: Coordinate-Free Visual Grounding for GUI Agentscited by 89
  3. 2025A Survey on (M)LLM-Based GUI Agentscited by 61
  4. 2025GUI-G2: Gaussian Reward Modeling for GUI Groundingcited by 60
  5. 2026UI-Venus-1.5 Technical Reportcited by 39
  6. 2025DiMo-GUI: Advancing Test-time Scaling in GUI Grounding via Modality-Aware Visual Reasoningcited by 29
  7. 2025Test-Time Reinforcement Learning for GUI Grounding via Region Consistencycited by 20
  8. 2026OmegaUse: Building a General-Purpose GUI Agent for Autonomous Task Executioncited by 13
  9. 2026ClawGUI: A Unified Framework for Training, Evaluating, and Deploying GUI Agentscited by 13
  10. 2025PAL-UI: Planning with Active Look-back for Vision-Based GUI Agentscited by 11
  11. 2026FocusUI: Efficient UI Grounding via Position-Preserving Visual Token Selectioncited by 9
  12. 2026Continual GUI Agentscited by 7
11 more
  1. 2025BTL-UI: Blink-Think-Link Reasoning Model for GUI Agentcited by 5
  2. 2025AndroidControl-Curated: Revealing the True Potential of GUI Agents through Benchmark Purificationcited by 5
  3. 2026UI-Zoomer: Uncertainty-Driven Adaptive Zoom-In for GUI Groundingcited by 5
  4. 2025Mind the Gap: Bridging Thought Leap for Improved Chain-of-Thought Tuningcited by 2
  5. 2025MEGA-GUI: Multi-stage Enhanced Grounding Agents for GUI Elementscited by 2
  6. 2025iSHIFT: Lightweight Slow-Fast GUI Agent with Adaptive Perceptioncited by 2
  7. 2026GAIA: A Data Flywheel System for Training GUI Test-Time Scaling Critic Modelscited by 2
  8. 2026Software Engineering for and with GUI Agentcited by 1
  9. 2026Perceive Before Reasoning: A Pre-Reasoning Perception Framework for Efficient and Reliable Proactive Mobile Agentscited by 0
  10. 2026Efficient GUI Agents: A Systems Survey of Observation, Memory, Action, and Runtime Optimizationcited by 0
  11. 2026Learning from Reliable Negatives: Confidence-Anchored Test-Time Adaptation for GUI Groundingcited by 0