# InfiGUI-R1: Advancing Multimodal GUI Agents from Reactive Actors to Deliberative Reasoners

arXiv: https://arxiv.org/abs/2504.14239 · HTML: https://mlsanity.com/paper/2504.14239/

## Key facts

- Cited by 140 papers in the map (data through 14 Oct 2026).
- Placed in Multimodal reasoning, one of 17 directions in LLM reasoning.
- Builds on 27 of its 52 references that are in the map.
- First posted on arXiv 19 Apr 2025.

## Placed in

- LLM reasoning / [Multimodal reasoning](https://mlsanity.com/lr/multimodal-reasoning/)
- Multimodal LLMs / vision-language models / [Embodied and 3D vision-language models](https://mlsanity.com/mllm/embodied-and-3d-vision-language-models/)
- LLM agents and tool use / [GUI and mobile agents](https://mlsanity.com/agents/gui-and-mobile-agents/)

## Builds on (most cited first, 27 of 27)

- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Qwen2.5-VL Technical Report](https://mlsanity.com/paper/2502.13923/) (2025)
- [HybridFlow: A Flexible and Efficient RLHF Framework](https://mlsanity.com/paper/2409.19256/) (2024)
- [Kosmos-2: Grounding Multimodal Large Language Models to the World](https://mlsanity.com/paper/2306.14824/) (2023)
- [CogAgent: A Visual Language Model for GUI Agents](https://mlsanity.com/paper/2312.08914/) (2023)
- [UI-TARS: Pioneering Automated GUI Interaction with Native Agents](https://mlsanity.com/paper/2501.12326/) (2025)
- [SeeClick: Harnessing GUI Grounding for Advanced Visual GUI Agents](https://mlsanity.com/paper/2401.10935/) (2024)
- [AndroidWorld: A Dynamic Benchmarking Environment for Autonomous Agents](https://mlsanity.com/paper/2405.14573/) (2024)
- [Understanding the planning of LLM agents: A survey](https://mlsanity.com/paper/2402.02716/) (2024)
- [AppAgent: Multimodal Agents as Smartphone Users](https://mlsanity.com/paper/2312.13771/) (2023)
- [Navigating the Digital World as Humans Do: Universal Visual Grounding for GUI Agents](https://mlsanity.com/paper/2410.05243/) (2024)
- [OS-ATLAS: A Foundation Action Model for Generalist GUI Agents](https://mlsanity.com/paper/2410.23218/) (2024)
- [ScreenSpot-Pro: GUI Grounding for Professional High-Resolution Computer Use](https://mlsanity.com/paper/2504.07981/) (2025)
- [GUI-R1 : A Generalist R1-Style Vision-Language Action Model For GUI Agents](https://mlsanity.com/paper/2504.10458/) (2025)
- [Windows Agent Arena: Evaluating Multi-Modal OS Agents at Scale](https://mlsanity.com/paper/2409.08264/) (2024)
- [AutoDroid: LLM-powered Task Automation in Android](https://mlsanity.com/paper/2308.15272/) (2023)
- [Ferret-UI: Grounded Mobile UI Understanding with Multimodal LLMs](https://mlsanity.com/paper/2404.05719/) (2024)
- [UI-R1: Enhancing Efficient Action Prediction of GUI Agents by Reinforcement Learning](https://mlsanity.com/paper/2503.21620/) (2025)
- [Agent S2: A Compositional Generalist-Specialist Framework for Computer Use Agents](https://mlsanity.com/paper/2504.00906/) (2025)
- [InfiAgent-DABench: Evaluating Agents on Data Analysis Tasks](https://mlsanity.com/paper/2401.05507/) (2024)
- [Aria-UI: Visual Grounding for GUI Instructions](https://mlsanity.com/paper/2412.16256/) (2024)
- [Lawformer: A Pre-trained Language Model for Chinese Legal Long Documents](https://mlsanity.com/paper/2105.03887/) (2021)
- [Widget Captioning: Generating Natural Language Description for Mobile User Interface Elements](https://mlsanity.com/paper/2010.04295/) (2020)
- [InfiGUIAgent: A Multimodal Generalist GUI Agent with Native Reasoning and Reflection](https://mlsanity.com/paper/2501.04575/) (2025)
- [Chain-of-Action: Faithful and Multimodal Question Answering through Large Language Models](https://mlsanity.com/paper/2403.17359/) (2024)

## Built on by (most cited first, 30 of 140)

- [The Landscape of Agentic Reinforcement Learning for LLMs: A Survey](https://mlsanity.com/paper/2509.02547/) (2025)
- [Mobile-Agent-v3: Fundamental Agents for GUI Automation](https://mlsanity.com/paper/2508.15144/) (2025)
- [A Survey of Reinforcement Learning for Large Reasoning Models](https://mlsanity.com/paper/2509.08827/) (2025)
- [A Survey of Self-Evolving Agents: What, When, How, and Where to Evolve on the Path to Artificial Super Intelligence](https://mlsanity.com/paper/2507.21046/) (2025)
- [GTA1: GUI Test-time Scaling Agent](https://mlsanity.com/paper/2507.05791/) (2025)
- [UI-Venus Technical Report: Building High-performance UI Agents with RFT](https://mlsanity.com/paper/2508.10833/) (2025)
- [MAI-UI Technical Report: Real-World Centric Foundation GUI Agents](https://mlsanity.com/paper/2512.22047/) (2025)
- [Perception, Reason, Think, and Plan: A Survey on Large Multimodal Reasoning Models](https://mlsanity.com/paper/2505.04921/) (2025)
- [Mobile-Agent-v3.5: Multi-platform Fundamental GUI Agents](https://mlsanity.com/paper/2602.16855/) (2026)
- [GUI-G1: Understanding R1-Zero-Like Training for Visual Grounding in GUI Agents](https://mlsanity.com/paper/2505.15810/) (2025)
- [GUI-G2: Gaussian Reward Modeling for GUI Grounding](https://mlsanity.com/paper/2507.15846/) (2025)
- [ScaleCUA: Scaling Open-Source Computer Use Agents with Cross-Platform Data](https://mlsanity.com/paper/2509.15221/) (2025)
- [MobileGUI-RL: Advancing Mobile GUI Agent through Reinforcement Learning in Online Environment](https://mlsanity.com/paper/2507.05720/) (2025)
- [A Survey of Agentic Reasoning for Large Language Models: Towards Recursively Self-Improving and Collective Agents](https://mlsanity.com/paper/2601.12538/) (2026)
- [An Empirical Study on Reinforcement Learning for Reasoning-Search Interleaved LLM Agents](https://mlsanity.com/paper/2505.15117/) (2025)
- [DeepShop: A Benchmark for Deep Research Shopping Agents](https://mlsanity.com/paper/2506.02839/) (2025)
- [ZeroGUI: Automating Online GUI Learning at Zero Human Cost](https://mlsanity.com/paper/2505.23762/) (2025)
- [AgentCPM-GUI: Building Mobile-Use Agents with Reinforcement Fine-Tuning](https://mlsanity.com/paper/2506.01391/) (2025)
- [UI-Venus-1.5 Technical Report](https://mlsanity.com/paper/2602.09082/) (2026)
- [Large Language Models for Planning: A Comprehensive and Systematic Survey](https://mlsanity.com/paper/2505.19683/) (2025)
- [OSWorld 2.0: Benchmarking Computer Use Agents on Long-Horizon Real-World Tasks](https://mlsanity.com/paper/2606.29537/) (2026)
- [UI-Genie: A Self-Improving Approach for Iteratively Boosting MLLM-based Mobile GUI Agents](https://mlsanity.com/paper/2505.21496/) (2025)
- [UI-AGILE: Advancing GUI Agents with Effective Reinforcement Learning and Precise Inference-Time Grounding](https://mlsanity.com/paper/2507.22025/) (2025)
- [UI-S1: Advancing GUI Automation via Semi-online Reinforcement Learning](https://mlsanity.com/paper/2509.11543/) (2025)
- [Code2World: A GUI World Model via Renderable Code Generation](https://mlsanity.com/paper/2602.09856/) (2026)
- [LocateAnything: Fast and High-Quality Vision-Language Grounding with Parallel Box Decoding](https://mlsanity.com/paper/2605.27365/) (2026)
- [InfiGUI-G1: Advancing GUI Grounding with Adaptive Exploration Policy Optimization](https://mlsanity.com/paper/2508.05731/) (2025)
- [Efficient Multi-turn RL for GUI Agents via Decoupled Training and Adaptive Data Curation](https://mlsanity.com/paper/2509.23866/) (2025)
- [Phi-Ground Tech Report: Advancing Perception in GUI Grounding](https://mlsanity.com/paper/2507.23779/) (2025)
- [A Survey on Agentic Multimodal Large Language Models](https://mlsanity.com/paper/2510.10991/) (2025)

## Abstract (arXiv)

Multimodal Large Language Models (MLLMs) have powered Graphical User Interface (GUI) Agents, showing promise in automating tasks on computing devices. Recent works have begun exploring reasoning in GUI tasks with encouraging results. However, many current approaches rely on manually designed reasoning templates, which may result in reasoning that is not sufficiently robust and adaptive for complex GUI environments. Meanwhile, some existing agents continue to operate as Reactive Actors, relying primarily on implicit reasoning that may lack sufficient depth for GUI tasks demanding planning and error recovery. We argue that advancing these agents requires a shift from reactive acting towards acting based on deliberate reasoning. To facilitate this transformation, we introduce InfiGUI-R1, an MLLM-based GUI agent developed through our Actor2Reasoner framework, a reasoning-centric, two-stage training approach designed to progressively evolve agents from Reactive Actors to Deliberative Reasoners. The first stage, Reasoning Injection, focuses on establishing a basic reasoner. We employ Spatial Reasoning Distillation to transfer cross-modal spatial reasoning capabilities from teacher models to MLLMs through trajectories with explicit reasoning steps, enabling models to integrate GUI visual-spatial information with logical reasoning before action generation. The second stage, Deliberation Enhancement, refines the basic reasoner into a deliberative one using Reinforcement Learning. This stage introduces two approaches: Sub-goal Guidance, which rewards models for generating accurate intermediate sub-goals, and Error Recovery Scenario Construction, which creates failure-and-recovery training scenarios from identified prone-to-error steps. Experimental results show InfiGUI-R1 achieves strong performance in GUI grounding and trajectory tasks. Resources at https://github.com/Reallm-Labs/InfiGUI-R1.