GUI-Rise: Structured Reasoning and History Summarization for GUI Navigation
On the map: LLM reasoning → Multimodal reasoning
Abstract (arXiv)
While Multimodal Large Language Models (MLLMs) have advanced GUI navigation agents, current approaches face limitations in cross-domain generalization and effective history utilization. We present a reasoning-enhanced framework that systematically integrates structured reasoning, action prediction, and history summarization. The structured reasoning component generates coherent Chain-of-Thought analyses combining progress estimation and decision reasoning, which inform both immediate action predictions and compact history summaries for future steps. Based on this framework, we train a GUI agent, GUI-Rise, through supervised fine-tuning on pseudo-labeled trajectories and reinforcement learning with Group Relative Policy Optimization (GRPO). This framework employs specialized rewards, including a history-aware objective, directly linking summary quality to subsequent action performance. Comprehensive evaluations on standard benchmarks demonstrate state-of-the-art results under identical training data conditions, with particularly strong performance in out-of-domain scenarios. These findings validate our framework's ability to maintain robust reasoning and generalization across diverse GUI navigation tasks. Code is available at https://leon022.github.io/GUI-Rise.
Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Placed in Multimodal reasoning, one of 17 directions in LLM reasoning. Builds on 42 of its 53 references that are in the map. First posted on arXiv 31 Oct 2025.
Builds on
42 of 53 references.
Closest ancestors · 5
Same direction, newest first
- 2025VLM-R1: A Stable and Generalizable R1-style Large Vision-Language Model10 Apr
- 2025InfiGUIAgent: A Multimodal Generalist GUI Agent with Native Reasoning and Reflection8 Jan
- 2024Aguvis: Unified Pure Vision Agents for Autonomous GUI Interaction5 Dec
- 2024OSCAR: Operating System Control via State-Aware Reasoning and Re-Planning24 Oct
- 2024Android in the Zoo: Chain-of-Action-Thought for GUI Agents5 Mar
Other references · 37
- 2023GPT-4 Technical Reportcited by 18,132
- 2017Proximal Policy Optimization Algorithmscited by 14,441
- 2024The Llama 3 Herd of Modelscited by 14,192
- 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
- 2025Qwen2.5-VL Technical Reportcited by 5,418
- 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,151
- 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scalecited by 2,678
- 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,669
- 2023InternVL: Scaling up Vision Foundation Models and Aligning for Generic Visual-Linguistic Taskscited by 2,534
- 2023Qwen-VL: A Versatile Vision-Language Model for Understanding, Localization, Text Reading, and Beyondcited by 2,179
- 2023Mind2Web: Towards a Generalist Agent for the Webcited by 1,210
- 2023CogAgent: A Visual Language Model for GUI Agentscited by 692
25 more
- 2025UI-TARS: Pioneering Automated GUI Interaction with Native Agentscited by 584
- 2024SeeClick: Harnessing GUI Grounding for Advanced Visual GUI Agentscited by 553
- 2024GPT-4V(ision) is a Generalist Web Agent, if Groundedcited by 547
- 2023AppAgent: Multimodal Agents as Smartphone Userscited by 407
- 2024OS-ATLAS: A Foundation Action Model for Generalist GUI Agentscited by 370
- 2024Agent Workflow Memorycited by 342
- 2023Android in the Wild: A Large-Scale Dataset for Android Device Controlcited by 314
- 2024Mobile-Agent: Autonomous Multi-Modal Mobile Device Agent with Visual Perceptioncited by 283
- 2024MM-LLMs: Recent Advances in MultiModal Large Language Modelscited by 277
- 2025Logic-RL: Unleashing LLM Reasoning with Rule-Based Reinforcement Learningcited by 260
- 2024ShowUI: One Vision-Language-Action Model for GUI Visual Agentcited by 226
- 2024OmniParser for Pure Vision Based GUI Agentcited by 193
- 2024GUIOdyssey: A Comprehensive Dataset for Cross-App GUI Navigation on Mobile Devicescited by 190
- 2023You Only Look at Screens: Multimodal Chain-of-Action Agentscited by 185
- 2024Tree Search for Language Model Agentscited by 148
- 2024The Revolution of Multimodal Large Language Models: A Surveycited by 134
- 2024GUICourse: From General Vision Language Models to Versatile GUI Agentscited by 112
- 2024GUI Agents with Foundation Models: A Comprehensive Surveycited by 110
- 2024ScreenAgent: A Vision Language Model-driven Computer Control Agentcited by 101
- 2023BOLAA: Benchmarking and Orchestrating LLM-augmented Autonomous Agentscited by 83
- 2024CoCo-Agent: A Comprehensive Cognitive MLLM Agent for Smartphone GUI Automationcited by 70
- 2023Explore, Select, Derive, and Recall: Augmenting LLM with Human-like Memory for Mobile Task Automationcited by 57
- 2024Generative Artificial Intelligence: A Systematic Review and Applicationscited by 37
- 2025OS Agents: A Survey on MLLM-based Agents for General Computing Devices Usecited by 35
- 2025Token-Hungry, Yet Precise: DeepSeek R1 Highlights the Need for Multi-Step Reasoning Over Speed in MATHcited by 10
Built on by
Citing papers per year: 2026: 3.
3 papers.
Similar, not cited
Close in wording, no citation either way
- 2025History-Aware Reasoning for GUI Agents
- 2025PAL-UI: Planning with Active Look-back for Vision-Based GUI Agents
- 2026GUI-Libra: Training Native GUI Agents to Reason and Act with Action-aware Supervision and Partially Verifiable RL
- 2026Scaling, Benchmarking, and Reasoning of Vision-Language Agents for Mobile GUI Navigation
- 2025HiconAgent: History Context-aware Policy Optimization for GUI Agents
- 2025Chain-of-Memory: Enhancing GUI Agents for Cross-Application Navigation
- 2026AnchorGUI: Asymmetric Memory for Dual-Scale Learning in GUI Navigation
- 2026Learning Reliable GUI Agents under Imperfect Priors