Qwen-UI-Agent Technical Report: Toward Next-Generation Real-World Centric Foundation GUI Agents
On the map: LLM agents and tool use → GUI and mobile agents
Abstract (arXiv)
GUI agents have the potential to become a general purpose executor over existing digital devices. To advance them toward real-world use, we envision agents that operate reliably on real devices, execute workflows across platforms, combine GUI interaction with CLI execution, complete long-horizon tasks, proactively initiate useful services, and autonomously improve their capabilities with minimal human effort. Guided by this vision, we present Qwen-UI-Agent, a real-world centric foundation GUI agent spanning mobile, computer-use, web, and DeepSearch environments. Qwen-UI-Agent combines diverse sandbox environments with a large-scale real-device mobile runtime. Its unified action space interleaves GUI operations with CLI execution and generates batched actions in a single model turn. An AutoResearch-style data flywheel uses agents to construct tasks and environments, diagnose failures, and plan subsequent iterations. Online RL supports training on trajectories exceeding 100 turns, with over 10,000 concurrent environments accelerating rollout. A lightweight harness layer supports proactive service initiation and stateful workflows across mobile and computer. Across a broad suite of evaluations, Qwen-UI-Agent sets state-of-the-art performance on mobile-use benchmarks while delivering competitive performance on computer- and browser-use tasks against frontier models, including Opus 4.8, Gemini 3.1 Pro, and GPT-5.6 Sol. On mobile use, it achieves 82.1% on MobileWorld, 92.2% on MobileWorld-Real, and 97.5% on AndroidDaily. On computer use, it achieves 79.5% on OSWorld-Verified and a 40.0% partial-progress score on OSWorld-v2. On browser use and GUI grounding, it achieves 73.6% on WebArena and 81.5% on ScreenSpot-Pro, respectively.
Key facts. Cited by 8 papers in the map (data through 7 Oct 2026). Placed in GUI and mobile agents, one of 16 directions in LLM agents and tool use. Builds on 48 of its 79 references that are in the map. First posted on arXiv 30 Jul 2026.
Builds on
48 of 79 references.
Closest ancestors · 20
Same direction, newest first
- 2026HyMobileAgent: Data-Environment Co-Scaling for Efficient GUI Agents16 Jul
- 2026UI-MOPD: Multi-Platform On-Policy Distillation for Unified GUI Agents5 Jul
- 2026Xiaomi-GUI-0 Technical Report30 Jun
- 2026OSWorld 2.0: Benchmarking Computer Use Agents on Long-Horizon Real-World Tasks28 Jun
- 2026PhoneBuddy: Training Open Models for Agentic Phone Use22 Jun
- 2026AndroidDaily: A Verifiable Benchmark for Mobile GUI Agents on Real-World Closed-Source Applications26 May
- 2026ToolCUA: Towards Optimal GUI-Tool Path Orchestration for Computer Use Agents12 May
- 2026Mobile-Agent-v3.5: Multi-platform Fundamental GUI Agents15 Feb
- 2026UI-Venus-1.5 Technical Report9 Feb
- 2025MAI-UI Technical Report: Real-World Centric Foundation GUI Agents26 Dec
- 2025Step-GUI Technical Report17 Dec
- 2025UI-Ins: Enhancing GUI Grounding with Multi-Perspective Instruction-as-Reasoning23 Oct
8 more
- 2025UltraCUA: A Foundation Model for Computer Use Agents with Hybrid Action20 Oct
- 2025Efficient Multi-turn RL for GUI Agents via Decoupled Training and Adaptive Data Curation28 Sep
- 2025UI-TARS-2 Technical Report: Advancing GUI Agent with Multi-Turn Reinforcement Learning2 Sep
- 2025OpenCUA: Open Foundations for Computer-Use Agents12 Aug
- 2025MMBench-GUI: Hierarchical Multi-Platform Evaluation Framework for GUI Agents25 Jul
- 2025Scaling Computer-Use Grounding via User Interface Decomposition and Synthesis19 May
- 2025UI-TARS: Pioneering Automated GUI Interaction with Native Agents21 Jan
- 2024OS-ATLAS: A Foundation Action Model for Generalist GUI Agents30 Oct
Other references · 28
- 2023WebArena: A Realistic Web Environment for Building Autonomous Agentscited by 1,908
- 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,758
- 2023Instruction-Following Evaluation for Large Language Modelscited by 1,143
- 2016A Diagram Is Worth A Dozen Imagescited by 793
- 2025DeepSeek-V3.2: Pushing the Frontier of Open Large Language Modelscited by 755
- 2025BrowseComp: A Simple Yet Challenging Benchmark for Browsing Agentscited by 616
- 2025τ2-Bench: Evaluating Conversational Agents in a Dual-Control Environmentcited by 558
- 2024GPT-4V(ision) is a Generalist Web Agent, if Groundedcited by 547
- 2024Measuring Multimodal Mathematical Reasoning with MATH-Vision Datasetcited by 516
- 2026Terminal-Bench: Benchmarking Agents on Hard, Realistic Tasks in Command Line Interfacescited by 515
- 2024MMMU-Pro: A More Robust Multi-discipline Multimodal Understanding Benchmarkcited by 494
- 2024AndroidWorld: A Dynamic Benchmarking Environment for Autonomous Agentscited by 450
16 more
- 2024WebVoyager: Building an End-to-End Web Agent with Large Multimodal Modelscited by 424
- 2026SkillsBench: Benchmarking How Well Agent Skills Work Across Diverse Taskscited by 289
- 2024CharXiv: Charting Gaps in Realistic Chart Understanding in Multimodal LLMscited by 229
- 2025An Illusion of Progress? Assessing the Current State of Web Agentscited by 165
- 2025BrowseComp-ZH: Benchmarking Web Browsing Ability of Large Language Models in Chinesecited by 139
- 2025GTA1: GUI Test-time Scaling Agentcited by 102
- 2025UI-Vision: A Desktop-centric GUI Benchmark for Visual Perception and Interactioncited by 68
- 2026Claw-Eval: Towards Trustworthy Evaluation of Autonomous Agentscited by 57
- 2025Tongyi DeepResearch Technical Reportcited by 53
- 2026Single-Rollout Asynchronous Optimization for Agentic Reinforcement Learningcited by 26
- 2026WebWorld: A Large-Scale World Model for Web Agent Trainingcited by 23
- 2026CUA-Gym: Scaling Verifiable Training Environments and Tasks for Computer-Use Agentscited by 21
- 2026Odysseys: Benchmarking Web Agents on Realistic Long Horizon Taskscited by 19
- 2026PhoneHarness: Harnessing Phone-Use Agents through Mixed GUI, CLI, and Tool Actionscited by 6
- 2026EvoCUA-1.5: Online Reinforcement Learning for Multi-turn Computer-Use Agentscited by 2
- 2026One Forward Beats Two: InnerZoom for Accurate and Efficient GUI Groundingcited by 1
Built on by
Citing papers per year: 2026: 8.
8 papers.
- 2026UI-Mate: Advancing Open-Weight Foundation GUI Agents with In-Context Demonstrationscited by 1
- 2026Wuying-Browser-Agent: Real-World Centric Fundamental Long-Horizon Browser Agentscited by 1
- 2026UI-Venus-2 Technical Reportcited by 1
- 2026RecreationWorld: Scalable and Verifiable Environments for Hybrid Computer-Use Agentscited by 1
- 2026From Tapping to Hopping: Augmenting Mobile GUI Agents with App-Native Deeplinkscited by 0
- 2026CUA-Sandbox: Efficient Environments for Computer-Use Agent Reinforcement Learningcited by 0
- 2026Org-Agent: Beyond Personal Assistants Towards Organizational Agentscited by 0
- 2026HybridCUA: Learning to Orchestrate GUI and CLI for Computer-Use Agentscited by 0
- 2026HybridCUA: Learning to Orchestrate GUI and CLI for Computer-Use Agents29 Sep 2026
- 2026Org-Agent: Beyond Personal Assistants Towards Organizational Agents28 Sep 2026
- 2026CUA-Sandbox: Efficient Environments for Computer-Use Agent Reinforcement Learning26 Sep 2026
- 2026From Tapping to Hopping: Augmenting Mobile GUI Agents with App-Native Deeplinks25 Sep 2026
- 2026RecreationWorld: Scalable and Verifiable Environments for Hybrid Computer-Use Agents18 Sep 2026
- 2026UI-Venus-2 Technical Report27 Aug 2026
- 2026Wuying-Browser-Agent: Real-World Centric Fundamental Long-Horizon Browser Agents18 Aug 2026
- 2026UI-Mate: Advancing Open-Weight Foundation GUI Agents with In-Context Demonstrations16 Aug 2026
Similar, not cited
Close in wording, no citation either way
- 2025Mobile GUI Agents under Real-world Threats: Are We There Yet?
- 2026APPSim-Bench: Bridging Real-world Apps and Reproducible Evaluation for Mobile GUI Agents
- 2026ClawGUI: A Unified Framework for Training, Evaluating, and Deploying GUI Agents
- 2026CUA-Universe: A Scalable and Dynamic Environment for Hybrid GUI+CLI Agents
- 2026BlueLM-GUI Technical Report: A Real-Device-Centric Flywheel for Self-Improving Mobile GUI Agents
- 2026VenusBench-Mobile: A Challenging and User-Centric Benchmark for Mobile GUI Agents with Capability Diagnostics
- 2026WeaveBench: A Long-Horizon, Real-World Benchmark for Computer-Use Agents with Hybrid Interfaces
- 2026ScaleWoB: Guiding GUI Agents with Coding Agents via Large-Scale Environmental Synthesis