HyMobileAgent: Data-Environment Co-Scaling for Efficient GUI Agents
On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models
also in 1 more
Abstract (arXiv)
As large multimodal models move from understanding content to operating on digital environments, mobile GUI has emerged as a challenging and consequential testbed for digital embodied intelligence. Mobile agents operate under three coupled constraints: precise perception of complex interfaces, scalable acquisition of high-quality interaction data, and robust long-horizon decision making under compounding execution errors. This report presents HyMobileAgent, a mobile GUI agent built on Hy3.0-VL-A3B, a vision-native foundation model featuring native any-resolution input, an A3B-scale deployment budget, and a 32K context window to model extended interaction histories. Rather than relying solely on model scaling, we develop a joint data and environment centric scaling framework to address the key bottlenecks of mobile interaction. Our framework integrates a GUI perception flywheel combining mock-interface synthesis, rejection sampling, and icon-specific augmentation; a knowledge pipeline that transforms tutorial videos into structured interaction data; a million-scale action data pipeline deployed across more than 2000 sandbox and real-device instances with automated failure attribution; the PhoneWorld Mock App Factory, providing a resettable training environment with 34 mock applications and over 34000 tasks; and a structured Planning-and-Reflection mechanism with explicit dead-loop detection for reliable long-horizon execution. We also introduce a progressive training recipe consisting of mid-training, supervised fine-tuning, and reinforcement learning with task-specific reward designs.
Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 42 of its 55 references that are in the map. First posted on arXiv 16 Jul 2026.
Builds on
42 of 55 references.
Closest ancestors · 7
Same direction, newest first
- 2025InfiGUI-R1: Advancing Multimodal GUI Agents from Reactive Actors to Deliberative Reasoners19 Apr
- 2025GUI-R1 : A Generalist R1-Style Vision-Language Action Model For GUI Agents14 Apr
- 2025ScreenSpot-Pro: GUI Grounding for Professional High-Resolution Computer Use4 Apr
- 2024OS-ATLAS: A Foundation Action Model for Generalist GUI Agents30 Oct
- 2024Navigating the Digital World as Humans Do: Universal Visual Grounding for GUI Agents7 Oct
- 2024WebVoyager: Building an End-to-End Web Agent with Large Multimodal Models25 Jan
- 2023CogAgent: A Visual Language Model for GUI Agents14 Dec
Other references · 35
- 2022Training language models to follow instructions with human feedbackcited by 15,712
- 2017Proximal Policy Optimization Algorithmscited by 14,441
- 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
- 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
- 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
- 2023Reflexion: Language Agents with Verbal Reinforcement Learningcited by 4,064
- 2023WebArena: A Realistic Web Environment for Building Autonomous Agentscited by 1,908
- 2023Mind2Web: Towards a Generalist Agent for the Webcited by 1,210
- 2024OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environmentscited by 1,134
- 2025UI-TARS: Pioneering Automated GUI Interaction with Native Agentscited by 584
- 2024SeeClick: Harnessing GUI Grounding for Advanced Visual GUI Agentscited by 553
- 2024GPT-4V(ision) is a Generalist Web Agent, if Groundedcited by 547
23 more
- 2024AndroidWorld: A Dynamic Benchmarking Environment for Autonomous Agentscited by 450
- 2023AppAgent: Multimodal Agents as Smartphone Userscited by 407
- 2023Android in the Wild: A Large-Scale Dataset for Android Device Controlcited by 314
- 2024Mobile-Agent: Autonomous Multi-Modal Mobile Device Agent with Visual Perceptioncited by 283
- 2025UI-TARS-2 Technical Report: Advancing GUI Agent with Multi-Turn Reinforcement Learningcited by 205
- 2024GUIOdyssey: A Comprehensive Dataset for Cross-App GUI Navigation on Mobile Devicescited by 190
- 2024On the Effects of Data Scale on UI Control Agentscited by 187
- 2025Mobile-Agent-v3: Fundamental Agents for GUI Automationcited by 183
- 2020Mapping Natural Language Instructions to Mobile UI Action Sequencescited by 180
- 2024Aria-UI: Visual Grounding for GUI Instructionscited by 136
- 2025Scaling Computer-Use Grounding via User Interface Decomposition and Synthesiscited by 132
- 2024AndroidLab: Training and Systematic Benchmarking of Android Autonomous Agentscited by 105
- 2024AutoGLM: Autonomous Foundation Agents for GUIscited by 96
- 2025GUI-Actor: Coordinate-Free Visual Grounding for GUI Agentscited by 89
- 2026Mobile-Agent-v3.5: Multi-platform Fundamental GUI Agentscited by 72
- 2025MMBench-GUI: Hierarchical Multi-Platform Evaluation Framework for GUI Agentscited by 64
- 2026EvoCUA: Evolving Computer Use Agents via Learning from Scalable Synthetic Experiencecited by 51
- 2025MobileRL: Online Agentic Reinforcement Learning for Mobile GUI Agentscited by 43
- 2026UI-Venus-1.5 Technical Reportcited by 39
- 2025Fara-7B: An Efficient Agentic Model for Computer Usecited by 34
- 2025Phi-Ground Tech Report: Advancing Perception in GUI Groundingcited by 24
- 2026MobileDreamer: Generative Sketch World Model for GUI Agentcited by 21
- 2025AUTO-Explorer: Automated Data Collection for GUI Agentcited by 5
Built on by
Citing papers per year: 2026: 3.
3 papers.
Similar, not cited
Close in wording, no citation either way
- 2025MagicGUI: A Foundational Mobile GUI Agent with Scalable Data Pipeline and Reinforcement Fine-tuning
- 2026OmniGUI: Benchmarking GUI Agents in Omni-Modal Smartphone Environments
- 2025MobiBench: Multi-Branch, Modular Benchmark for Mobile GUI Agents
- 2024MobileViews: A Million-scale and Diverse Mobile GUI Dataset
- 2026VenusBench-Mobile: A Challenging and User-Centric Benchmark for Mobile GUI Agents with Capability Diagnostics
- 2026GUI-CEval: A Hierarchical and Comprehensive Chinese Benchmark for Mobile GUI Agents
- 2026GUI-CC: Benchmarking Contextual Consistency of GUI World Models as Agent Environments
- 2026Benchmarking and Improving GUI Agents in High-Dynamic Environments