mlsanityData to 7 Oct 2026

OmegaUse: Building a General-Purpose GUI Agent for Autonomous Task Execution

arXiv 2601.20380 ↗First posted 28 Jan 2026cs.AIcited by 13

On the map: LLM agents and tool use → GUI and mobile agents

also in 1 more
Abstract (arXiv)

Graphical User Interface (GUI) agents show great potential for enabling foundation models to complete real-world tasks, revolutionizing human-computer interaction and improving human productivity. In this report, we present OmegaUse, a general-purpose GUI agent model for autonomous task execution on both mobile and desktop platforms, supporting computer-use and phone-use scenarios. Building an effective GUI agent model relies on two factors: (1) high-quality data and (2) effective training methods. To address these, we introduce a carefully engineered data-construction pipeline and a decoupled training paradigm. For data construction, we leverage rigorously curated open-source datasets and introduce a novel automated synthesis framework that integrates bottom-up autonomous exploration with top-down taxonomy-guided generation to create high-fidelity synthetic data. For training, to better leverage these data, we adopt a two-stage strategy: Supervised Fine-Tuning (SFT) to establish fundamental interaction syntax, followed by Group Relative Policy Optimization (GRPO) to improve spatial grounding and sequential planning. To balance computational efficiency with agentic reasoning capacity, OmegaUse is built on a Mixture-of-Experts (MoE) backbone. To evaluate cross-terminal capabilities in an offline setting, we introduce OS-Nav, a benchmark suite spanning multiple operating systems: ChiM-Nav, targeting Chinese Android mobile environments, and Ubu-Nav, focusing on routine desktop interactions on Ubuntu. Extensive experiments show that OmegaUse is highly competitive across established GUI benchmarks, achieving a state-of-the-art (SOTA) score of 96.3% on ScreenSpot-V2 and a leading 79.1% step success rate on AndroidControl. OmegaUse also performs strongly on OS-Nav, reaching 74.24% step success on ChiM-Nav and 55.9% average success on Ubu-Nav.

Key facts. Cited by 13 papers in the map (data through 7 Oct 2026). Placed in GUI and mobile agents, one of 16 directions in LLM agents and tool use. Builds on 56 of its 63 references that are in the map. First posted on arXiv 28 Jan 2026.

Builds on

56 of 63 references.

Closest ancestors · 31

Same direction, newest first

  1. 2025MAI-UI Technical Report: Real-World Centric Foundation GUI Agents26 Dec
  2. 2025Step-GUI Technical Report17 Dec
  3. 2025Scaling Agents for Computer Use2 Oct
  4. 2025UI-TARS-2 Technical Report: Advancing GUI Agent with Multi-Turn Reinforcement Learning2 Sep
  5. 2025Mobile-Agent-v3: Fundamental Agents for GUI Automation21 Aug
  6. 2025UI-Venus Technical Report: Building High-performance UI Agents with RFT14 Aug
  7. 2025OpenCUA: Open Foundations for Computer-Use Agents12 Aug
  8. 2025Phi-Ground Tech Report: Advancing Perception in GUI Grounding31 Jul
  9. 2025UI-AGILE: Advancing GUI Agents with Effective Reinforcement Learning and Precise Inference-Time Grounding29 Jul
  10. 2025LPO: Towards Accurate GUI Agent Interaction via Location Preference Optimization11 Jun
  11. 2025GUI-Actor: Coordinate-Free Visual Grounding for GUI Agents3 Jun
  12. 2025AgentCPM-GUI: Building Mobile-Use Agents with Reinforcement Fine-Tuning2 Jun
19 more
  1. 2025GUI-G1: Understanding R1-Zero-Like Training for Visual Grounding in GUI Agents21 May
  2. 2025Scaling Computer-Use Grounding via User Interface Decomposition and Synthesis19 May
  3. 2025Enhancing Visual Grounding for GUI Agents via Self-Evolutionary Reinforcement Learning18 May
  4. 2025ScaleTrack: Scaling and back-tracking Automated GUI Agents1 May
  5. 2025InfiGUI-R1: Advancing Multimodal GUI Agents from Reactive Actors to Deliberative Reasoners19 Apr
  6. 2025GUI-R1 : A Generalist R1-Style Vision-Language Action Model For GUI Agents14 Apr
  7. 2025ScreenSpot-Pro: GUI Grounding for Professional High-Resolution Computer Use4 Apr
  8. 2025UI-R1: Enhancing Efficient Action Prediction of GUI Agents by Reinforcement Learning27 Mar
  9. 2025UI-TARS: Pioneering Automated GUI Interaction with Native Agents21 Jan
  10. 2024OS-Genesis: Automating GUI Agent Trajectory Construction via Reverse Task Synthesis27 Dec
  11. 2024OS-ATLAS: A Foundation Action Model for Generalist GUI Agents30 Oct
  12. 2024AutoGLM: Autonomous Foundation Agents for GUIs28 Oct
  13. 2024Agent S: An Open Agentic Framework that Uses Computers Like a Human10 Oct
  14. 2024Navigating the Digital World as Humans Do: Universal Visual Grounding for GUI Agents7 Oct
  15. 2024On the Effects of Data Scale on UI Control Agents6 Jun
  16. 2024Mobile-Agent-v2: Mobile Device Operation Assistant with Effective Navigation via Multi-Agent Collaboration3 Jun
  17. 2024Mobile-Agent: Autonomous Multi-Modal Mobile Device Agent with Visual Perception29 Jan
  18. 2024SeeClick: Harnessing GUI Grounding for Advanced Visual GUI Agents17 Jan
  19. 2023CogAgent: A Visual Language Model for GUI Agents14 Dec

Other references · 25

  1. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  2. 2025Qwen2.5-VL Technical Reportcited by 5,418
  3. 2025Qwen3-VL Technical Reportcited by 2,534
  4. 2023Mind2Web: Towards a Generalist Agent for the Webcited by 1,210
  5. 2024AndroidWorld: A Dynamic Benchmarking Environment for Autonomous Agentscited by 450
  6. 2025Seed1.5-VL Technical Reportcited by 367
  7. 2023Android in the Wild: A Large-Scale Dataset for Android Device Controlcited by 314
  8. 2024Aguvis: Unified Pure Vision Agents for Autonomous GUI Interactioncited by 271
  9. 2024ShowUI: One Vision-Language-Action Model for GUI Visual Agentcited by 226
  10. 2024Aria-UI: Visual Grounding for GUI Instructionscited by 136
  11. 2024AutoWebGLM: A Large Language Model-based Web Navigating Agentcited by 130
  12. 2024Cradle: Empowering Foundation Agents Towards General Computer Controlcited by 114
13 more
  1. 2025GTA1: GUI Test-time Scaling Agentcited by 102
  2. 2020Widget Captioning: Generating Natural Language Description for Mobile User Interface Elementscited by 94
  3. 2021UIBert: Learning Generic Multimodal Representations for UI Understandingcited by 93
  4. 2024MM1.5: Methods, Analysis & Insights from Multimodal LLM Fine-tuningcited by 75
  5. 2024OmniParser: A Unified Framework for Text Spotting, Key Information Extraction and Table Recognitioncited by 74
  6. 2025GUI-G2: Gaussian Reward Modeling for GUI Groundingcited by 60
  7. 2025Look Before You Leap: A GUI-Critic-R1 Model for Pre-Operative Error Diagnosis in GUI Automationcited by 27
  8. 2025Think Twice, Click Once: Enhancing GUI Grounding via Fast and Slow Systemscited by 23
  9. 2026OS-Symphony: A Holistic Framework for Robust and Generalist Computer-Using Agentcited by 19
  10. 2024Improved GUI Grounding via Iterative Narrowingcited by 15
  11. 2025UGround: Towards Unified Visual Grounding with Unrolled Transformerscited by 14
  12. 2025Mano Technical Reportcited by 13
  13. 2025NaviMaster: Learning a Unified Policy for GUI and Embodied Navigation Taskscited by 10

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 13.

13 papers.

Sort:

By area:

Similar, not cited

Close in wording, no citation either way

  1. 2025Agent S2: A Compositional Generalist-Specialist Framework for Computer Use Agents
  2. 2025GUI-Xplore: Empowering Generalizable GUI Agents with One Exploration
  3. 2025Breaking the Data Barrier -- Building GUI Agents Through Task Generalization
  4. 2026APPSim-Bench: Bridging Real-world Apps and Reproducible Evaluation for Mobile GUI Agents
  5. 2023ASSISTGUI: Task-Oriented Desktop Graphical User Interface Automation
  6. 2026DocOS: Towards Proactive Document-Guided Actions in GUI Agents
  7. 2025MMBench-GUI: Hierarchical Multi-Platform Evaluation Framework for GUI Agents
  8. 2025UIPro: Unleashing Superior Interaction Capability For GUI Agents