mlsanityData to 7 Oct 2026

InfiGUI-R1: Advancing Multimodal GUI Agents from Reactive Actors to Deliberative Reasoners

arXiv 2504.14239 ↗First posted 19 Apr 2025cs.AI cs.CLcited by 140

On the map: LLM reasoning → Multimodal reasoning

also in 2 more
Abstract (arXiv)

Multimodal Large Language Models (MLLMs) have powered Graphical User Interface (GUI) Agents, showing promise in automating tasks on computing devices. Recent works have begun exploring reasoning in GUI tasks with encouraging results. However, many current approaches rely on manually designed reasoning templates, which may result in reasoning that is not sufficiently robust and adaptive for complex GUI environments. Meanwhile, some existing agents continue to operate as Reactive Actors, relying primarily on implicit reasoning that may lack sufficient depth for GUI tasks demanding planning and error recovery. We argue that advancing these agents requires a shift from reactive acting towards acting based on deliberate reasoning. To facilitate this transformation, we introduce InfiGUI-R1, an MLLM-based GUI agent developed through our Actor2Reasoner framework, a reasoning-centric, two-stage training approach designed to progressively evolve agents from Reactive Actors to Deliberative Reasoners. The first stage, Reasoning Injection, focuses on establishing a basic reasoner. We employ Spatial Reasoning Distillation to transfer cross-modal spatial reasoning capabilities from teacher models to MLLMs through trajectories with explicit reasoning steps, enabling models to integrate GUI visual-spatial information with logical reasoning before action generation. The second stage, Deliberation Enhancement, refines the basic reasoner into a deliberative one using Reinforcement Learning. This stage introduces two approaches: Sub-goal Guidance, which rewards models for generating accurate intermediate sub-goals, and Error Recovery Scenario Construction, which creates failure-and-recovery training scenarios from identified prone-to-error steps. Experimental results show InfiGUI-R1 achieves strong performance in GUI grounding and trajectory tasks. Resources at https://github.com/Reallm-Labs/InfiGUI-R1.

Key facts. Cited by 140 papers in the map (data through 7 Oct 2026). Placed in Multimodal reasoning, one of 17 directions in LLM reasoning. Builds on 27 of its 52 references that are in the map. First posted on arXiv 19 Apr 2025.

Builds on

27 of 52 references.

Closest ancestors · 2

Same direction, newest first

  1. 2025UI-R1: Enhancing Efficient Action Prediction of GUI Agents by Reinforcement Learning27 Mar
  2. 2025InfiGUIAgent: A Multimodal Generalist GUI Agent with Native Reasoning and Reflection8 Jan

Other references · 25

  1. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2025Qwen2.5-VL Technical Reportcited by 5,418
  5. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 1,980
  6. 2023Kosmos-2: Grounding Multimodal Large Language Models to the Worldcited by 1,075
  7. 2023CogAgent: A Visual Language Model for GUI Agentscited by 692
  8. 2025UI-TARS: Pioneering Automated GUI Interaction with Native Agentscited by 584
  9. 2024SeeClick: Harnessing GUI Grounding for Advanced Visual GUI Agentscited by 553
  10. 2024AndroidWorld: A Dynamic Benchmarking Environment for Autonomous Agentscited by 450
  11. 2024Understanding the planning of LLM agents: A surveycited by 414
  12. 2023AppAgent: Multimodal Agents as Smartphone Userscited by 407
13 more
  1. 2024Navigating the Digital World as Humans Do: Universal Visual Grounding for GUI Agentscited by 373
  2. 2024OS-ATLAS: A Foundation Action Model for Generalist GUI Agentscited by 370
  3. 2025ScreenSpot-Pro: GUI Grounding for Professional High-Resolution Computer Usecited by 246
  4. 2025GUI-R1 : A Generalist R1-Style Vision-Language Action Model For GUI Agentscited by 240
  5. 2024Windows Agent Arena: Evaluating Multi-Modal OS Agents at Scalecited by 212
  6. 2023AutoDroid: LLM-powered Task Automation in Androidcited by 178
  7. 2024Ferret-UI: Grounded Mobile UI Understanding with Multimodal LLMscited by 173
  8. 2025Agent S2: A Compositional Generalist-Specialist Framework for Computer Use Agentscited by 156
  9. 2024Aria-UI: Visual Grounding for GUI Instructionscited by 136
  10. 2024InfiAgent-DABench: Evaluating Agents on Data Analysis Taskscited by 134
  11. 2021Lawformer: A Pre-trained Language Model for Chinese Legal Long Documentscited by 115
  12. 2020Widget Captioning: Generating Natural Language Description for Mobile User Interface Elementscited by 94
  13. 2024Chain-of-Action: Faithful and Multimodal Question Answering through Large Language Modelscited by 27

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 71; 2026: 69.

140 papers.

Sort:

By area:

  1. 2025The Landscape of Agentic Reinforcement Learning for LLMs: A Surveycited by 195
  2. 2025Mobile-Agent-v3: Fundamental Agents for GUI Automationcited by 183
  3. 2025A Survey of Reinforcement Learning for Large Reasoning Modelscited by 175
  4. 2025A Survey of Self-Evolving Agents: What, When, How, and Where to Evolve on the Path to Artificial Super Intelligencecited by 155
  5. 2025GTA1: GUI Test-time Scaling Agentcited by 102
  6. 2025UI-Venus Technical Report: Building High-performance UI Agents with RFTcited by 90
  7. 2025Perception, Reason, Think, and Plan: A Survey on Large Multimodal Reasoning Modelscited by 89
  8. 2025MAI-UI Technical Report: Real-World Centric Foundation GUI Agentscited by 88
  9. 2025GUI-G1: Understanding R1-Zero-Like Training for Visual Grounding in GUI Agentscited by 73
  10. 2026Mobile-Agent-v3.5: Multi-platform Fundamental GUI Agentscited by 72
  11. 2025GUI-G2: Gaussian Reward Modeling for GUI Groundingcited by 60
  12. 2025ScaleCUA: Scaling Open-Source Computer Use Agents with Cross-Platform Datacited by 58
38 more
  1. 2025MobileGUI-RL: Advancing Mobile GUI Agent through Reinforcement Learning in Online Environmentcited by 51
  2. 2026A Survey of Agentic Reasoning for Large Language Models: Towards Recursively Self-Improving and Collective Agentscited by 47
  3. 2025ZeroGUI: Automating Online GUI Learning at Zero Human Costcited by 44
  4. 2025AgentCPM-GUI: Building Mobile-Use Agents with Reinforcement Fine-Tuningcited by 44
  5. 2025An Empirical Study on Reinforcement Learning for Reasoning-Search Interleaved LLM Agentscited by 43
  6. 2025DeepShop: A Benchmark for Deep Research Shopping Agentscited by 42
  7. 2026UI-Venus-1.5 Technical Reportcited by 39
  8. 2025Large Language Models for Planning: A Comprehensive and Systematic Surveycited by 38
  9. 2025UI-Genie: A Self-Improving Approach for Iteratively Boosting MLLM-based Mobile GUI Agentscited by 33
  10. 2025UI-AGILE: Advancing GUI Agents with Effective Reinforcement Learning and Precise Inference-Time Groundingcited by 32
  11. 2025UI-S1: Advancing GUI Automation via Semi-online Reinforcement Learningcited by 32
  12. 2026OSWorld 2.0: Benchmarking Computer Use Agents on Long-Horizon Real-World Taskscited by 29
  13. 2026Code2World: A GUI World Model via Renderable Code Generationcited by 26
  14. 2025Phi-Ground Tech Report: Advancing Perception in GUI Groundingcited by 24
  15. 2025InfiGUI-G1: Advancing GUI Grounding with Adaptive Exploration Policy Optimizationcited by 24
  16. 2025Efficient Multi-turn RL for GUI Agents via Decoupled Training and Adaptive Data Curationcited by 24
  17. 2025UItron: Foundational GUI Agent with Advanced Perception and Planningcited by 22
  18. 2026MobileDreamer: Generative Sketch World Model for GUI Agentcited by 21
  19. 2025UI-Ins: Enhancing GUI Grounding with Multi-Perspective Instruction-as-Reasoningcited by 20
  20. 2026LocateAnything: Fast and High-Quality Vision-Language Grounding with Parallel Box Decodingcited by 20
  21. 2025MagicGUI: A Foundational Mobile GUI Agent with Scalable Data Pipeline and Reinforcement Fine-tuningcited by 19
  22. 2025A Survey on Agentic Multimodal Large Language Modelscited by 19
  23. 2025Mobile-R1: Towards Interactive Capability for VLM-Based Mobile Agent via Systematic Trainingcited by 17
  24. 2025LPO: Towards Accurate GUI Agent Interaction via Location Preference Optimizationcited by 16
  25. 2025Grounding Computer Use Agents on Human Demonstrationscited by 16
  26. 2025WebCoach: Self-Evolving Web Agents with Cross-Session Memory Guidancecited by 15
  27. 2025Sailing by the Stars: A Survey on Reward Models and Learning Strategies for Learning from Rewardscited by 14
  28. 2025WebCoT: Enhancing Web Agent Reasoning by Reconstructing Chain-of-Thought in Reflection, Branching, and Rollbackcited by 14
  29. 2025Q-Ponder: A Unified Training Pipeline for Reasoning-based Visual Quality Assessmentcited by 14
  30. 2025Ferret-UI Lite: Lessons from Building Small On-Device GUI Agentscited by 14
  31. 2025Reinforcement Fine-Tuning Powers Reasoning Capability of Multimodal Large Language Modelscited by 13
  32. 2025HiconAgent: History Context-aware Policy Optimization for GUI Agentscited by 13
  33. 2026OmegaUse: Building a General-Purpose GUI Agent for Autonomous Task Executioncited by 13
  34. 2026ClawGUI: A Unified Framework for Training, Evaluating, and Deploying GUI Agentscited by 13
  35. 2025Learning GUI Grounding with Spatial Reasoning from Visual Feedbackcited by 12
  36. 2025GhostEI-Bench: Do Mobile Agents Resilience to Environmental Injection in Dynamic On-Device Environments?cited by 12
  37. 2025GUI-Shift: Enhancing VLM-Based GUI Agents through Self-supervised Reinforcement Learningcited by 11
  38. 2025Stop Wasting Your Tokens: Towards Efficient Runtime Multi-Agent Systemscited by 11

50 most cited of 140 All 140 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025A Survey on (M)LLM-Based GUI Agents
  2. 2026GUI-Libra: Training Native GUI Agents to Reason and Act with Action-aware Supervision and Partially Verifiable RL
  3. 2025Breaking the Data Barrier -- Building GUI Agents Through Task Generalization
  4. 2025Mirage-1: Augmenting and Updating GUI Agent with Hierarchical Multimodal Skills
  5. 2025A Survey on GUI Agents with Foundation Models Enhanced by Reinforcement Learning
  6. 2025Learning, Reasoning, Refinement: A Framework for Kahneman's Dual-System Intelligence in GUI Agents
  7. 2024Ponder & Press: Advancing Visual GUI Agent towards General Computer Control
  8. 2025Does Chain-of-Thought Reasoning Help Mobile GUI Agent? An Empirical Study