StepReflect: Structured UI Transition Reflection for Mobile GUI Agents
On the map: LLM agents and tool use → GUI and mobile agents
Abstract (arXiv)
Autonomous mobile GUI agents require accurate action reflection for reliable long-horizon execution. Existing approaches rely on open-ended multimodal reasoning after each action, which is costly and poorly matched to the structured nature of GUI state transitions. We propose StepReflect, which formulates per-step GUI reflection as supervised structured prediction conditioned on explicit transition specifications and paired visual evidence. StepReflect is trained through a staged pipeline combining supervised fine-tuning, teacher-student distillation, and preference- and reward-based refinement. Offline, the resulting 8B model achieves 82.16% transition-level accuracy on AndroidWorld, exceeding zero-shot GPT-5.2 by 11.83 percentage points under the same structured input. Online, across M3A, Agent-SAMA, MAI-UI-8B, and Seed-2.0-Pro, StepReflect achieves higher task success in three of four agent configurations and remains within one successful task of the GPT-5.2 Reflection Agent in the fourth. It also reduces paid API charges relative to GPT-based reflection in all four configurations. These results establish StepReflect as a practical, locally deployable alternative to repeated frontier-model reflection for long-horizon mobile GUI agents.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in GUI and mobile agents, one of 16 directions in LLM agents and tool use. Builds on 34 of its 38 references that are in the map. First posted on arXiv 6 Aug 2026.
Builds on
34 of 38 references.
Closest ancestors · 14
Same direction, newest first
- 2026A Task-State Representation for Long-Horizon Mobile GUI Agents1 Jul
- 2025MAI-UI Technical Report: Real-World Centric Foundation GUI Agents26 Dec
- 2025MobileWorld: Benchmarking Autonomous Mobile Agents in Agent-User Interactive and MCP-Augmented Environments22 Dec
- 2025GUI-KV: Efficient GUI Agents via KV Cache with Spatio-Temporal Awareness1 Oct
- 2025Mobile-Agent-v3: Fundamental Agents for GUI Automation21 Aug
- 2025Test-Time Reinforcement Learning for GUI Grounding via Region Consistency7 Aug
- 2025Agent-SAMA: State-Aware Mobile Assistant29 May
- 2025UI-R1: Enhancing Efficient Action Prediction of GUI Agents by Reinforcement Learning27 Mar
- 2025Advancing Mobile GUI Agents: A Verifier-Driven Approach to Practical Deployment20 Mar
- 2025Mobile-Agent-E: Self-Evolving Mobile Assistant for Complex Tasks20 Jan
- 2024SPA-Bench: A Comprehensive Benchmark for SmartPhone Agent Evaluation19 Oct
- 2024SeeClick: Harnessing GUI Grounding for Advanced Visual GUI Agents17 Jan
2 more
Other references · 20
- 2015Distilling the Knowledge in a Neural Networkcited by 10,324
- 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
- 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
- 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,276
- 2023Let's Verify Step by Stepcited by 3,789
- 2023Distilling Step-by-Step! Outperforming Larger Language Models with Less Training Data and Smaller Model Sizescited by 737
- 2024GPT-4V(ision) is a Generalist Web Agent, if Groundedcited by 563
- 2024AndroidWorld: A Dynamic Benchmarking Environment for Autonomous Agentscited by 468
- 2022Large Language Models Are Reasoning Teacherscited by 329
- 2024Recursive Introspection: Teaching Language Model Agents How to Self-Improvecited by 165
- 2024Autonomous Evaluation and Refinement of Digital Agentscited by 137
- 2025Multi-Agent Design: Optimizing Agents with Better Prompts and Topologiescited by 113
8 more
- 2025KVFlow: Efficient Prefix Caching for Accelerating LLM-Based Multi-Agent Workflowscited by 56
- 2025Web-Shepherd: Advancing PRMs for Reinforcing Web Agentscited by 41
- 2025AutoPDL: Automatic Prompt Optimization for LLM Agentscited by 27
- 2025Look Before You Leap: A GUI-Critic-R1 Model for Pre-Operative Error Diagnosis in GUI Automationcited by 27
- 2025GUI-Shepherd: Reliable Process Reward and Verification for Long-Sequence GUI Taskscited by 26
- 2025GUI-Reflection: Empowering Multimodal GUI Models with Self-Reflection Behaviorcited by 23
- 2022Task-Relevant Failure Detection for Trajectory Predictors in Autonomous Vehiclescited by 11
- 2026Towards Structured, State-Aware, and Execution-Grounded Reasoning for Software Engineering Agentscited by 2
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2026SEE: Structure-aware Exploring & Exploiting for Long-horizon GUI Agent Trajectory Synthesis
- 2026GUI-CC: Benchmarking Contextual Consistency of GUI World Models as Agent Environments
- 2026UI-Voyager: A Self-Evolving GUI Agent Learning via Failed Experience
- 2026APPSim-Bench: Bridging Real-world Apps and Reproducible Evaluation for Mobile GUI Agents
- 2026MemGUI-Agent: An End-to-End Long-Horizon Mobile GUI Agent with Proactive Context Management
- 2025MobiBench: Multi-Branch, Modular Benchmark for Mobile GUI Agents
- 2025MobileUse: A GUI Agent with Hierarchical Reflection for Autonomous Mobile Operation
- 2026AppDeltaWorld: Transition-Grounded Delta Code World Model for Mobile GUI Agents