OmegaUse: Building a General-Purpose GUI Agent for Autonomous Task Execution
On the map: LLM agents and tool use → GUI and mobile agents
also in 1 more
Abstract (arXiv)
Graphical User Interface (GUI) agents show great potential for enabling foundation models to complete real-world tasks, revolutionizing human-computer interaction and improving human productivity. In this report, we present OmegaUse, a general-purpose GUI agent model for autonomous task execution on both mobile and desktop platforms, supporting computer-use and phone-use scenarios. Building an effective GUI agent model relies on two factors: (1) high-quality data and (2) effective training methods. To address these, we introduce a carefully engineered data-construction pipeline and a decoupled training paradigm. For data construction, we leverage rigorously curated open-source datasets and introduce a novel automated synthesis framework that integrates bottom-up autonomous exploration with top-down taxonomy-guided generation to create high-fidelity synthetic data. For training, to better leverage these data, we adopt a two-stage strategy: Supervised Fine-Tuning (SFT) to establish fundamental interaction syntax, followed by Group Relative Policy Optimization (GRPO) to improve spatial grounding and sequential planning. To balance computational efficiency with agentic reasoning capacity, OmegaUse is built on a Mixture-of-Experts (MoE) backbone. To evaluate cross-terminal capabilities in an offline setting, we introduce OS-Nav, a benchmark suite spanning multiple operating systems: ChiM-Nav, targeting Chinese Android mobile environments, and Ubu-Nav, focusing on routine desktop interactions on Ubuntu. Extensive experiments show that OmegaUse is highly competitive across established GUI benchmarks, achieving a state-of-the-art (SOTA) score of 96.3% on ScreenSpot-V2 and a leading 79.1% step success rate on AndroidControl. OmegaUse also performs strongly on OS-Nav, reaching 74.24% step success on ChiM-Nav and 55.9% average success on Ubu-Nav.
Key facts. Cited by 13 papers in the map (data through 7 Oct 2026). Placed in GUI and mobile agents, one of 16 directions in LLM agents and tool use. Builds on 56 of its 63 references that are in the map. First posted on arXiv 28 Jan 2026.
Builds on
56 of 63 references.
Closest ancestors · 31
Same direction, newest first
- 2025MAI-UI Technical Report: Real-World Centric Foundation GUI Agents26 Dec
- 2025Step-GUI Technical Report17 Dec
- 2025Scaling Agents for Computer Use2 Oct
- 2025UI-TARS-2 Technical Report: Advancing GUI Agent with Multi-Turn Reinforcement Learning2 Sep
- 2025Mobile-Agent-v3: Fundamental Agents for GUI Automation21 Aug
- 2025UI-Venus Technical Report: Building High-performance UI Agents with RFT14 Aug
- 2025OpenCUA: Open Foundations for Computer-Use Agents12 Aug
- 2025Phi-Ground Tech Report: Advancing Perception in GUI Grounding31 Jul
- 2025UI-AGILE: Advancing GUI Agents with Effective Reinforcement Learning and Precise Inference-Time Grounding29 Jul
- 2025LPO: Towards Accurate GUI Agent Interaction via Location Preference Optimization11 Jun
- 2025GUI-Actor: Coordinate-Free Visual Grounding for GUI Agents3 Jun
- 2025AgentCPM-GUI: Building Mobile-Use Agents with Reinforcement Fine-Tuning2 Jun
19 more
- 2025GUI-G1: Understanding R1-Zero-Like Training for Visual Grounding in GUI Agents21 May
- 2025Scaling Computer-Use Grounding via User Interface Decomposition and Synthesis19 May
- 2025Enhancing Visual Grounding for GUI Agents via Self-Evolutionary Reinforcement Learning18 May
- 2025ScaleTrack: Scaling and back-tracking Automated GUI Agents1 May
- 2025InfiGUI-R1: Advancing Multimodal GUI Agents from Reactive Actors to Deliberative Reasoners19 Apr
- 2025GUI-R1 : A Generalist R1-Style Vision-Language Action Model For GUI Agents14 Apr
- 2025ScreenSpot-Pro: GUI Grounding for Professional High-Resolution Computer Use4 Apr
- 2025UI-R1: Enhancing Efficient Action Prediction of GUI Agents by Reinforcement Learning27 Mar
- 2025UI-TARS: Pioneering Automated GUI Interaction with Native Agents21 Jan
- 2024OS-Genesis: Automating GUI Agent Trajectory Construction via Reverse Task Synthesis27 Dec
- 2024OS-ATLAS: A Foundation Action Model for Generalist GUI Agents30 Oct
- 2024AutoGLM: Autonomous Foundation Agents for GUIs28 Oct
- 2024Agent S: An Open Agentic Framework that Uses Computers Like a Human10 Oct
- 2024Navigating the Digital World as Humans Do: Universal Visual Grounding for GUI Agents7 Oct
- 2024On the Effects of Data Scale on UI Control Agents6 Jun
- 2024Mobile-Agent-v2: Mobile Device Operation Assistant with Effective Navigation via Multi-Agent Collaboration3 Jun
- 2024Mobile-Agent: Autonomous Multi-Modal Mobile Device Agent with Visual Perception29 Jan
- 2024SeeClick: Harnessing GUI Grounding for Advanced Visual GUI Agents17 Jan
- 2023CogAgent: A Visual Language Model for GUI Agents14 Dec
Other references · 25
- 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
- 2025Qwen2.5-VL Technical Reportcited by 5,418
- 2025Qwen3-VL Technical Reportcited by 2,534
- 2023Mind2Web: Towards a Generalist Agent for the Webcited by 1,210
- 2024AndroidWorld: A Dynamic Benchmarking Environment for Autonomous Agentscited by 450
- 2025Seed1.5-VL Technical Reportcited by 367
- 2023Android in the Wild: A Large-Scale Dataset for Android Device Controlcited by 314
- 2024Aguvis: Unified Pure Vision Agents for Autonomous GUI Interactioncited by 271
- 2024ShowUI: One Vision-Language-Action Model for GUI Visual Agentcited by 226
- 2024Aria-UI: Visual Grounding for GUI Instructionscited by 136
- 2024AutoWebGLM: A Large Language Model-based Web Navigating Agentcited by 130
- 2024Cradle: Empowering Foundation Agents Towards General Computer Controlcited by 114
13 more
- 2025GTA1: GUI Test-time Scaling Agentcited by 102
- 2020Widget Captioning: Generating Natural Language Description for Mobile User Interface Elementscited by 94
- 2021UIBert: Learning Generic Multimodal Representations for UI Understandingcited by 93
- 2024MM1.5: Methods, Analysis & Insights from Multimodal LLM Fine-tuningcited by 75
- 2024OmniParser: A Unified Framework for Text Spotting, Key Information Extraction and Table Recognitioncited by 74
- 2025GUI-G2: Gaussian Reward Modeling for GUI Groundingcited by 60
- 2025Look Before You Leap: A GUI-Critic-R1 Model for Pre-Operative Error Diagnosis in GUI Automationcited by 27
- 2025Think Twice, Click Once: Enhancing GUI Grounding via Fast and Slow Systemscited by 23
- 2026OS-Symphony: A Holistic Framework for Robust and Generalist Computer-Using Agentcited by 19
- 2024Improved GUI Grounding via Iterative Narrowingcited by 15
- 2025UGround: Towards Unified Visual Grounding with Unrolled Transformerscited by 14
- 2025Mano Technical Reportcited by 13
- 2025NaviMaster: Learning a Unified Policy for GUI and Embodied Navigation Taskscited by 10
Built on by
Citing papers per year: 2026: 13.
13 papers.
Sort:
By area:
- 2026UI-Venus-1.5 Technical Reportcited by 39
- 2026GUI Agents with Reinforcement Learning: Toward Digital Inhabitantscited by 3
- 2026AppDeltaWorld: Transition-Grounded Delta Code World Model for Mobile GUI Agentscited by 2
- 2026OmegaUse-OfficeVal: Benchmarking LLM Agents on Long-Horizon Office-Suite Tasks with Economic Groundingcited by 1
- 2026The Horizon Gap: Planning, Memory, Execution, Training, and Evaluation for Long-Horizon LLM Agentscited by 1
- 2026Software Engineering for and with GUI Agentcited by 1
- 2026UI-Venus-2 Technical Reportcited by 1
- 2026Unrewarded Exploration in Large Language Models Reveals Latent Learning from Psychologycited by 0
- 2026Securing Computer-Use Agents: A Unified Architecture-Lifecycle Framework for Deployment-Grounded Reliabilitycited by 0
- 2026ReGround: Restoring Visual Grounding in Multi-Step Reasoning through Self-Diagnosis and Visual Re-Examinationcited by 0
- 2026Clarify-Then-Search: A Clarification Benchmark for Deep Search with End-to-End Nugget Restorationcited by 0
- 2026OmegaUse-SOP: SOP Engineering for Professional Computer Use from Human Demonstrationscited by 0
1 more
- 2026Learning Reliable GUI Agents under Imperfect Priorscited by 0
- 2026Learning Reliable GUI Agents under Imperfect Priors30 Sep 2026
- 2026OmegaUse-SOP: SOP Engineering for Professional Computer Use from Human Demonstrations2 Sep 2026
- 2026UI-Venus-2 Technical Report27 Aug 2026
- 2026Software Engineering for and with GUI Agent10 Aug 2026
- 2026The Horizon Gap: Planning, Memory, Execution, Training, and Evaluation for Long-Horizon LLM Agents7 Aug 2026
- 2026AppDeltaWorld: Transition-Grounded Delta Code World Model for Mobile GUI Agents6 Aug 2026
- 2026ReGround: Restoring Visual Grounding in Multi-Step Reasoning through Self-Diagnosis and Visual Re-Examination5 Aug 2026
- 2026OmegaUse-OfficeVal: Benchmarking LLM Agents on Long-Horizon Office-Suite Tasks with Economic Grounding29 Jul 2026
- 2026Clarify-Then-Search: A Clarification Benchmark for Deep Search with End-to-End Nugget Restoration17 Jun 2026
- 2026Securing Computer-Use Agents: A Unified Architecture-Lifecycle Framework for Deployment-Grounded Reliability8 May 2026
- 2026GUI Agents with Reinforcement Learning: Toward Digital Inhabitants30 Apr 2026
- 2026UI-Venus-1.5 Technical Report9 Feb 2026
- 2026UI-Venus-1.5 Technical Reportcited by 39
- 2026GUI Agents with Reinforcement Learning: Toward Digital Inhabitantscited by 3
- 2026OmegaUse-OfficeVal: Benchmarking LLM Agents on Long-Horizon Office-Suite Tasks with Economic Groundingcited by 1
- 2026Software Engineering for and with GUI Agentcited by 1
- 2026UI-Venus-2 Technical Reportcited by 1
- 2026Securing Computer-Use Agents: A Unified Architecture-Lifecycle Framework for Deployment-Grounded Reliabilitycited by 0
- 2026OmegaUse-SOP: SOP Engineering for Professional Computer Use from Human Demonstrationscited by 0
- 2026OmegaUse-SOP: SOP Engineering for Professional Computer Use from Human Demonstrations2 Sep 2026
- 2026UI-Venus-2 Technical Report27 Aug 2026
- 2026Software Engineering for and with GUI Agent10 Aug 2026
- 2026OmegaUse-OfficeVal: Benchmarking LLM Agents on Long-Horizon Office-Suite Tasks with Economic Grounding29 Jul 2026
- 2026Securing Computer-Use Agents: A Unified Architecture-Lifecycle Framework for Deployment-Grounded Reliability8 May 2026
- 2026GUI Agents with Reinforcement Learning: Toward Digital Inhabitants30 Apr 2026
- 2026UI-Venus-1.5 Technical Report9 Feb 2026
- 2026Learning Reliable GUI Agents under Imperfect Priorscited by 0
- 2026Learning Reliable GUI Agents under Imperfect Priors30 Sep 2026
Similar, not cited
Close in wording, no citation either way
- 2025Agent S2: A Compositional Generalist-Specialist Framework for Computer Use Agents
- 2025GUI-Xplore: Empowering Generalizable GUI Agents with One Exploration
- 2025Breaking the Data Barrier -- Building GUI Agents Through Task Generalization
- 2026APPSim-Bench: Bridging Real-world Apps and Reproducible Evaluation for Mobile GUI Agents
- 2023ASSISTGUI: Task-Oriented Desktop Graphical User Interface Automation
- 2026DocOS: Towards Proactive Document-Guided Actions in GUI Agents
- 2025MMBench-GUI: Hierarchical Multi-Platform Evaluation Framework for GUI Agents
- 2025UIPro: Unleashing Superior Interaction Capability For GUI Agents