BTL-UI: Blink-Think-Link Reasoning Model for GUI Agent
On the map: LLM reasoning → Multimodal reasoning
also in 1 more
Abstract (arXiv)
In the field of AI-driven human-GUI interaction automation, while rapid advances in multimodal large language models and reinforcement fine-tuning techniques have yielded remarkable progress, a fundamental challenge persists: their interaction logic significantly deviates from natural human-GUI communication patterns. To fill this gap, we propose "Blink-Think-Link" (BTL), a brain-inspired framework for human-GUI interaction that mimics the human cognitive process between users and graphical interfaces. The system decomposes interactions into three biologically plausible phases: (1) Blink - rapid detection and attention to relevant screen areas, analogous to saccadic eye movements; (2) Think - higher-level reasoning and decision-making, mirroring cognitive planning; and (3) Link - generation of executable commands for precise motor control, emulating human action selection mechanisms. Additionally, we introduce two key technical innovations for the BTL framework: (1) Blink Data Generation - an automated annotation pipeline specifically optimized for blink data, and (2) BTL Reward -- the first rule-based reward mechanism that enables reinforcement learning driven by both process and outcome. Building upon this framework, we develop a GUI agent model named BTL-UI, which demonstrates competitive performance across both static GUI understanding and dynamic interaction tasks in comprehensive benchmarks. These results provide conclusive empirical validation of the framework's efficacy in developing advanced GUI Agents.
Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Placed in Multimodal reasoning, one of 17 directions in LLM reasoning. Builds on 29 of its 36 references that are in the map. First posted on arXiv 19 Sep 2025.
Builds on
29 of 36 references.
Closest ancestors · 4
Same direction, newest first
- 2025InfiGUI-R1: Advancing Multimodal GUI Agents from Reactive Actors to Deliberative Reasoners19 Apr
- 2025VLM-R1: A Stable and Generalizable R1-style Large Vision-Language Model10 Apr
- 2025Visual-RFT: Visual Reinforcement Fine-Tuning3 Mar
- 2024Aguvis: Unified Pure Vision Agents for Autonomous GUI Interaction5 Dec
Other references · 25
- 2017Proximal Policy Optimization Algorithmscited by 15,215
- 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,330
- 2025Qwen2.5-VL Technical Reportcited by 5,587
- 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,234
- 2025InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Modelscited by 1,834
- 2023CogAgent: A Visual Language Model for GUI Agentscited by 704
- 2025UI-TARS: Pioneering Automated GUI Interaction with Native Agentscited by 610
- 2024SeeClick: Harnessing GUI Grounding for Advanced Visual GUI Agentscited by 567
- 2024Navigating the Digital World as Humans Do: Universal Visual Grounding for GUI Agentscited by 382
- 2024OS-ATLAS: A Foundation Action Model for Generalist GUI Agentscited by 378
- 2025ScreenSpot-Pro: GUI Grounding for Professional High-Resolution Computer Usecited by 254
- 2025GUI-R1 : A Generalist R1-Style Vision-Language Action Model For GUI Agentscited by 244
13 more
- 2024OmniParser for Pure Vision Based GUI Agentcited by 198
- 2024GUIOdyssey: A Comprehensive Dataset for Cross-App GUI Navigation on Mobile Devicescited by 194
- 2024Aria-UI: Visual Grounding for GUI Instructionscited by 140
- 2024GUI Agents with Foundation Models: A Comprehensive Surveycited by 111
- 2024GUI Agents: A Surveycited by 103
- 2024AutoGLM: Autonomous Foundation Agents for GUIscited by 100
- 2025Vision-R1: Evolving Human-Free Alignment in Large Vision-Language Models via Vision-Guided Reinforcement Learningcited by 69
- 2024MobileFlow: A Multimodal LLM For Mobile GUI Agentcited by 32
- 2023VisionTasker: Mobile Task Automation Using Vision Based UI Understanding and LLM Task Planningcited by 28
- 2025Think Twice, Click Once: Enhancing GUI Grounding via Fast and Slow Systemscited by 23
- 2024Falcon-UI: Understanding GUI Before Following User Instructionscited by 22
- 2025VEM: Environment-Free Exploration for Training GUI Agent with Value Environment Modelcited by 19
- 2024Lightweight Neural App Controlcited by 12
Built on by
Citing papers per year: 2025: 1; 2026: 4.
5 papers.
- 2026Learn where to Click from Yourself: On-Policy Self-Distillation for GUI Groundingcited by 11
- 2025Say One Thing, Do Another? Diagnosing Reasoning-Execution Gaps in VLM-Powered Mobile-Use Agentscited by 4
- 2026GUI Agents with Reinforcement Learning: Toward Digital Inhabitantscited by 3
- 2026Plover: Steering GUI Agents through Plan-Centric Interactioncited by 3
- 2026SwipeGen: Bridging the Execution Gap in GUI Agents via Human-like Swipe Synthesiscited by 0
- 2026Plover: Steering GUI Agents through Plan-Centric Interaction16 Jul 2026
- 2026Learn where to Click from Yourself: On-Policy Self-Distillation for GUI Grounding1 May 2026
- 2026GUI Agents with Reinforcement Learning: Toward Digital Inhabitants30 Apr 2026
- 2026SwipeGen: Bridging the Execution Gap in GUI Agents via Human-like Swipe Synthesis26 Jan 2026
- 2025Say One Thing, Do Another? Diagnosing Reasoning-Execution Gaps in VLM-Powered Mobile-Use Agents2 Oct 2025
Similar, not cited
Close in wording, no citation either way
- 2025Learning, Reasoning, Refinement: A Framework for Kahneman's Dual-System Intelligence in GUI Agents
- 2025Mano Technical Report
- 2025ZeroGUI: Automating Online GUI Learning at Zero Human Cost
- 2025UI-S1: Advancing GUI Automation via Semi-online Reinforcement Learning
- 2023ASSISTGUI: Task-Oriented Desktop Graphical User Interface Automation
- 2025History-Aware Reasoning for GUI Agents
- 2025A Survey on GUI Agents with Foundation Models Enhanced by Reinforcement Learning
- 2026GUIDE: A Benchmark for Understanding and Assisting Users in Open-Ended GUI Tasks