# Step-GUI Technical Report

arXiv: https://arxiv.org/abs/2512.15431 · HTML: https://mlsanity.com/paper/2512.15431/

## Key facts

- Cited by 39 papers in the map (data through 14 Oct 2026).
- Placed in GUI and mobile agents, one of 16 directions in LLM agents and tool use.
- Builds on 53 of its 61 references that are in the map.
- First posted on arXiv 17 Dec 2025.

## Placed in

- LLM agents and tool use / [GUI and mobile agents](https://mlsanity.com/agents/gui-and-mobile-agents/)

## Builds on (most cited first, 30 of 53)

- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [ReAct: Synergizing Reasoning and Acting in Language Models](https://mlsanity.com/paper/2210.03629/) (2022)
- [Self-Consistency Improves Chain of Thought Reasoning in Language Models](https://mlsanity.com/paper/2203.11171/) (2022)
- [Qwen2.5-VL Technical Report](https://mlsanity.com/paper/2502.13923/) (2025)
- [Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilities](https://mlsanity.com/paper/2507.06261/) (2025)
- [Self-Refine: Iterative Refinement with Self-Feedback](https://mlsanity.com/paper/2303.17651/) (2023)
- [Tree of Thoughts: Deliberate Problem Solving with Large Language Models](https://mlsanity.com/paper/2305.10601/) (2023)
- [Qwen3-VL Technical Report](https://mlsanity.com/paper/2511.21631/) (2025)
- [MMBench: Is Your Multi-modal Model an All-around Player?](https://mlsanity.com/paper/2307.06281/) (2023)
- [MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contexts](https://mlsanity.com/paper/2310.02255/) (2023)
- [InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiency](https://mlsanity.com/paper/2508.18265/) (2025)
- [OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments](https://mlsanity.com/paper/2404.07972/) (2024)
- [STaR: Bootstrapping Reasoning With Reasoning](https://mlsanity.com/paper/2203.14465/) (2022)
- [Are We on the Right Way for Evaluating Large Vision-Language Models?](https://mlsanity.com/paper/2403.20330/) (2024)
- [UI-TARS: Pioneering Automated GUI Interaction with Native Agents](https://mlsanity.com/paper/2501.12326/) (2025)
- [SeeClick: Harnessing GUI Grounding for Advanced Visual GUI Agents](https://mlsanity.com/paper/2401.10935/) (2024)
- [Reinforced Self-Training (ReST) for Language Modeling](https://mlsanity.com/paper/2308.08998/) (2023)
- [V*: Guided Visual Search as a Core Mechanism in Multimodal LLMs](https://mlsanity.com/paper/2312.14135/) (2023)
- [OCRBench: On the Hidden Mystery of OCR in Large Multimodal Models](https://mlsanity.com/paper/2305.07895/) (2023)
- [AndroidWorld: A Dynamic Benchmarking Environment for Autonomous Agents](https://mlsanity.com/paper/2405.14573/) (2024)
- [OS-ATLAS: A Foundation Action Model for Generalist GUI Agents](https://mlsanity.com/paper/2410.23218/) (2024)
- [Seed1.5-VL Technical Report](https://mlsanity.com/paper/2505.07062/) (2025)
- [Android in the Wild: A Large-Scale Dataset for Android Device Control](https://mlsanity.com/paper/2307.10088/) (2023)
- [Large Language Models are Better Reasoners with Self-Verification](https://mlsanity.com/paper/2212.09561/) (2022)
- [Quiet-STaR: Language Models Can Teach Themselves to Think Before Speaking](https://mlsanity.com/paper/2403.09629/) (2024)
- [ScreenSpot-Pro: GUI Grounding for Professional High-Resolution Computer Use](https://mlsanity.com/paper/2504.07981/) (2025)
- [LogicVista: Multimodal LLM Logical Reasoning Benchmark in Visual Contexts](https://mlsanity.com/paper/2407.04973/) (2024)
- [Tool Learning with Large Language Models: A Survey](https://mlsanity.com/paper/2405.17935/) (2024)
- [UI-TARS-2 Technical Report: Advancing GUI Agent with Multi-Turn Reinforcement Learning](https://mlsanity.com/paper/2509.02544/) (2025)
- [Mobile-Agent-v3: Fundamental Agents for GUI Automation](https://mlsanity.com/paper/2508.15144/) (2025)

## Built on by (most cited first, 30 of 39)

- [MAI-UI Technical Report: Real-World Centric Foundation GUI Agents](https://mlsanity.com/paper/2512.22047/) (2025)
- [Mobile-Agent-v3.5: Multi-platform Fundamental GUI Agents](https://mlsanity.com/paper/2602.16855/) (2026)
- [EvoCUA: Evolving Computer Use Agents via Learning from Scalable Synthetic Experience](https://mlsanity.com/paper/2601.15876/) (2026)
- [UI-Venus-1.5 Technical Report](https://mlsanity.com/paper/2602.09082/) (2026)
- [STEP3-VL-10B Technical Report](https://mlsanity.com/paper/2601.09668/) (2026)
- [KnowU-Bench: Towards Interactive, Proactive, and Personalized Mobile Agent Evaluation](https://mlsanity.com/paper/2604.08455/) (2026)
- [OmegaUse: Building a General-Purpose GUI Agent for Autonomous Task Execution](https://mlsanity.com/paper/2601.20380/) (2026)
- [UI-Mem: Self-Evolving Experience Memory for Online Reinforcement Learning in Mobile GUI Agents](https://mlsanity.com/paper/2602.05832/) (2026)
- [DM0: An Embodied-Native Vision-Language-Action Model towards Physical AI](https://mlsanity.com/paper/2602.14974/) (2026)
- [UI-Voyager: A Self-Evolving GUI Agent Learning via Failed Experience](https://mlsanity.com/paper/2603.24533/) (2026)
- [OpenMobile: Building Open Mobile Agents with Task and Trajectory Synthesis](https://mlsanity.com/paper/2604.15093/) (2026)
- [ToolCUA: Towards Optimal GUI-Tool Path Orchestration for Computer Use Agents](https://mlsanity.com/paper/2605.12481/) (2026)
- [MementoGUI: Learning Agentic Multimodal Memory Control for Long-Horizon GUI Agents](https://mlsanity.com/paper/2605.18652/) (2026)
- [Qwen-UI-Agent Technical Report: Toward Next-Generation Real-World Centric Foundation GUI Agents](https://mlsanity.com/paper/2607.28227/) (2026)
- [AndroidDaily: A Verifiable Benchmark for Mobile GUI Agents on Real-World Closed-Source Applications](https://mlsanity.com/paper/2605.27761/) (2026)
- [UI-MOPD: Multi-Platform On-Policy Distillation for Unified GUI Agents](https://mlsanity.com/paper/2607.04425/) (2026)
- [Video2GUI: Synthesizing Large-Scale Interaction Trajectories for Generalized GUI Agent Pretraining](https://mlsanity.com/paper/2605.14747/) (2026)
- [MobileForge: Annotation-Free Adaptation for Mobile GUI Agents with Hierarchical Feedback-Guided Policy Optimization](https://mlsanity.com/paper/2606.19930/) (2026)
- [VenusBench-Mobile: A Challenging and User-Centric Benchmark for Mobile GUI Agents with Capability Diagnostics](https://mlsanity.com/paper/2604.06182/) (2026)
- [Xiaomi-GUI-0 Technical Report](https://mlsanity.com/paper/2606.31410/) (2026)
- [What Memory Do GUI Agents Really Need? From Passive Records to Active Task-Driving States](https://mlsanity.com/paper/2606.31612/) (2026)
- [Agentic Reward Modeling: Verifying GUI Agent via Progressive Trajectory-Grounded Interaction](https://mlsanity.com/paper/2602.00575/) (2026)
- [POINTS-GUI-G: GUI-Grounding Journey](https://mlsanity.com/paper/2602.06391/) (2026)
- [PhoneBuddy: Training Open Models for Agentic Phone Use](https://mlsanity.com/paper/2606.23049/) (2026)
- [GUI Agents with Reinforcement Learning: Toward Digital Inhabitants](https://mlsanity.com/paper/2604.27955/) (2026)
- [EvoCUA-1.5: Online Reinforcement Learning for Multi-turn Computer-Use Agents](https://mlsanity.com/paper/2607.09773/) (2026)
- [The Next Screenshot Knows: Gated Hindsight Distillation for Mobile GUI Agents](https://mlsanity.com/paper/2608.06065/) (2026)
- [UI-Venus-2 Technical Report](https://mlsanity.com/paper/2609.00028/) (2026)
- [DocOS: Towards Proactive Document-Guided Actions in GUI Agents](https://mlsanity.com/paper/2605.18048/) (2026)
- [Securing Computer-Use Agents: A Unified Architecture-Lifecycle Framework for Deployment-Grounded Reliability](https://mlsanity.com/paper/2605.07110/) (2026)

## Abstract (arXiv)

Recent advances in multimodal large language models unlock unprecedented opportunities for GUI automation. However, a fundamental challenge remains: how to efficiently acquire high-quality training data while maintaining annotation reliability? We introduce a self-evolving training pipeline powered by the Calibrated Step Reward System, which converts model-generated trajectories into reliable training signals through trajectory-level calibration, achieving >90% annotation accuracy with 10-100x lower cost. Leveraging this pipeline, we introduce Step-GUI, a family of models (4B/8B) that achieves state-of-the-art GUI performance (8B: 80.2% AndroidWorld, 48.5% OSWorld, 62.6% ScreenShot-Pro) while maintaining robust general capabilities. As GUI agent capabilities improve, practical deployment demands standardized interfaces across heterogeneous devices while protecting user privacy. To this end, we propose GUI-MCP, the first Model Context Protocol for GUI automation with hierarchical architecture that combines low-level atomic operations and high-level task delegation to local specialist models, enabling high-privacy execution where sensitive data stays on-device. Finally, to assess whether agents can handle authentic everyday usage, we introduce AndroidDaily, a benchmark grounded in real-world mobile usage patterns with 3146 static actions and 235 end-to-end tasks across high-frequency daily scenarios (8B: static 89.91%, end-to-end 52.50%). Our work advances the development of practical GUI agents and demonstrates strong potential for real-world deployment in everyday digital interactions.