iSHIFT: Lightweight Slow-Fast GUI Agent with Adaptive Perception
On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models
also in 1 more
Abstract (arXiv)
Multimodal Large Language Models (MLLMs) show strong potential for interpreting and interacting with complex, pixel-rich Graphical User Interface (GUI) environments. However, building agents that are both efficient for high-level tasks and precise for fine-grained interactions remains challenging. GUI agents must perform routine actions efficiently while also handling tasks that demand exact visual grounding, yet existing approaches struggle when accuracy depends on identifying specific interface elements. These MLLMs also remain large and cannot adapt their reasoning depth to the task at hand. In this work, we introduce iSHIFT: Implicit Slow-fast Hybrid Inference with Flexible Tokens, a lightweight agent that integrates latent thinking (implicit chain-of-thought) with a perception control module. iSHIFT enables an MLLM to switch between a slow mode, which leverages detailed visual grounding for high precision and a fast mode that uses global cues for efficiency. Special perception tokens guide attention to relevant screen regions, allowing the model to decide both how to reason and where to focus. Despite its compact 2.5B size, iSHIFT matches state-of-the-art performance on multiple benchmark datasets.
Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 33 of its 40 references that are in the map. First posted on arXiv 26 Dec 2025.
Builds on
33 of 40 references.
Closest ancestors · 5
Same direction, newest first
- 2024OS-ATLAS: A Foundation Action Model for Generalist GUI Agents30 Oct
- 2024GUICourse: From General Vision Language Models to Versatile GUI Agents17 Jun
- 2024V-Zen: Efficient GUI Understanding and Precise Grounding With A Novel Multimodal LLM24 May
- 2023CogAgent: A Visual Language Model for GUI Agents14 Dec
- 2023GPT-4V in Wonderland: Large Multimodal Models for Zero-Shot Smartphone GUI Navigation13 Nov
Other references · 28
- 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
- 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
- 2023DINOv2: Learning Robust Visual Features without Supervisioncited by 7,764
- 2025Qwen2.5-VL Technical Reportcited by 5,418
- 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,151
- 2023Mind2Web: Towards a Generalist Agent for the Webcited by 1,210
- 2024Training Large Language Models to Reason in a Continuous Latent Spacecited by 763
- 2025UI-TARS: Pioneering Automated GUI Interaction with Native Agentscited by 584
- 2024SeeClick: Harnessing GUI Grounding for Advanced Visual GUI Agentscited by 553
- 2024GPT-4V(ision) is a Generalist Web Agent, if Groundedcited by 547
- 2024AndroidWorld: A Dynamic Benchmarking Environment for Autonomous Agentscited by 450
- 2024Mobile-Agent: Autonomous Multi-Modal Mobile Device Agent with Visual Perceptioncited by 283
16 more
- 2024Aguvis: Unified Pure Vision Agents for Autonomous GUI Interactioncited by 271
- 2024ShowUI: One Vision-Language-Action Model for GUI Visual Agentcited by 226
- 2024GUIOdyssey: A Comprehensive Dataset for Cross-App GUI Navigation on Mobile Devicescited by 190
- 2024On the Effects of Data Scale on UI Control Agentscited by 187
- 2023You Only Look at Screens: Multimodal Chain-of-Action Agentscited by 185
- 2024AMEX: Android Multi-annotation Expo Dataset for Mobile GUI Agentscited by 110
- 2024GUI Agents: A Surveycited by 101
- 2024VisualWebBench: How Far Have Multimodal LLMs Evolved in Web Page Understanding and Grounding?cited by 98
- 2021UIBert: Learning Generic Multimodal Representations for UI Understandingcited by 93
- 2023From Pixels to UI Actions: Learning to Follow Instructions via Graphical User Interfacescited by 67
- 2024Visual Agents as Fast and Slow Thinkerscited by 53
- 2024Agents Thinking Fast and Slow: A Talker-Reasoner Architecturecited by 35
- 2025Introducing Visual Perception Token into Multimodal Large Language Modelcited by 26
- 2025Think Twice, Click Once: Enhancing GUI Grounding via Fast and Slow Systemscited by 23
- 2024Falcon-UI: Understanding GUI Before Following User Instructionscited by 22
- 2025SpiritSight Agent: Advanced GUI Agent with One Lookcited by 18
Built on by
Citing papers per year: 2026: 2.
2 papers.
Similar, not cited
Close in wording, no citation either way
- 2025MGA: Memory-Driven GUI Agent for Observation-Centric Interaction
- 2025UI-AGILE: Advancing GUI Agents with Effective Reinforcement Learning and Precise Inference-Time Grounding
- 2025An Efficient Training Pipeline for Reasoning Graphical User Interface Agents
- 2025Breaking the Data Barrier -- Building GUI Agents Through Task Generalization
- 2026Towards GUI Agents: Vision-Language Diffusion Models for GUI Grounding
- 2025MEGA-GUI: Multi-stage Enhanced Grounding Agents for GUI Elements
- 2025MPR-GUI: Benchmarking and Enhancing Multilingual Perception and Reasoning in GUI Agents
- 2024Navigating the Digital World as Humans Do: Universal Visual Grounding for GUI Agents