mlsanityData to 7 Oct 2026

OpenCUA: Open Foundations for Computer-Use Agents

arXiv 2508.09123 ↗First posted 12 Aug 2025cs.AI cs.CVcited by 151

On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models

also in 1 more
Abstract (arXiv)

Vision-language models have demonstrated impressive capabilities as computer-use agents (CUAs) capable of automating diverse computer tasks. As their commercial potential grows, critical details of the most capable CUA systems remain closed. As these agents will increasingly mediate digital interactions and execute consequential decisions on our behalf, the research community needs access to open CUA frameworks to study their capabilities, limitations, and risks. To bridge this gap, we propose OpenCUA, a comprehensive open-source framework for scaling CUA data and foundation models. Our framework consists of: (1) an annotation infrastructure that seamlessly captures human computer-use demonstrations; (2) AgentNet, the first large-scale computer-use task dataset spanning 3 operating systems and 200+ applications and websites; (3) a scalable pipeline that transforms demonstrations into state-action pairs with reflective long Chain-of-Thought reasoning that sustain robust performance gains as data scales. Our end-to-end agent models demonstrate strong performance across CUA benchmarks. In particular, OpenCUA-72B achieves an average success rate of 45.0% on OSWorld-Verified, establishing a new state-of-the-art (SOTA) among open-source models. Further analysis confirms that our approach generalizes well across domains and benefits significantly from increased test-time computation. We release our annotation tool, datasets, code, and models to build open foundations for further CUA research.

Key facts. Cited by 151 papers in the map (data through 7 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 41 of its 58 references that are in the map. First posted on arXiv 12 Aug 2025.

Builds on

41 of 58 references.

Closest ancestors · 4

Same direction, newest first

  1. 2025ScreenSpot-Pro: GUI Grounding for Professional High-Resolution Computer Use4 Apr
  2. 2024OS-ATLAS: A Foundation Action Model for Generalist GUI Agents30 Oct
  3. 2024Navigating the Digital World as Humans Do: Universal Visual Grounding for GUI Agents7 Oct
  4. 2024GUICourse: From General Vision Language Models to Versatile GUI Agents17 Jun

Other references · 37

  1. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  2. 2025Qwen2.5-VL Technical Reportcited by 5,418
  3. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  4. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,151
  5. 2023WebArena: A Realistic Web Environment for Building Autonomous Agentscited by 1,908
  6. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,722
  7. 2023CAMEL: Communicative Agents for "Mind" Exploration of Large Language Model Societycited by 1,348
  8. 2023Mind2Web: Towards a Generalist Agent for the Webcited by 1,210
  9. 2024OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environmentscited by 1,134
  10. 2025UI-TARS: Pioneering Automated GUI Interaction with Native Agentscited by 584
  11. 2024SeeClick: Harnessing GUI Grounding for Advanced Visual GUI Agentscited by 553
  12. 2023Identifying the Risks of LM Agents with an LM-Emulated Sandboxcited by 514
25 more
  1. 2023V*: Guided Visual Search as a Core Mechanism in Multimodal LLMscited by 459
  2. 2025Kimi-VL Technical Reportcited by 381
  3. 2025Seed1.5-VL Technical Reportcited by 367
  4. 2024WorkArena: How Capable Are Web Agents at Solving Common Knowledge Work Tasks?cited by 348
  5. 2023Android in the Wild: A Large-Scale Dataset for Android Device Controlcited by 314
  6. 2024Aguvis: Unified Pure Vision Agents for Autonomous GUI Interactioncited by 271
  7. 2024ShowUI: One Vision-Language-Action Model for GUI Visual Agentcited by 226
  8. 2024Windows Agent Arena: Evaluating Multi-Modal OS Agents at Scalecited by 212
  9. 2024GUIOdyssey: A Comprehensive Dataset for Cross-App GUI Navigation on Mobile Devicescited by 190
  10. 2024Agent S: An Open Agentic Framework that Uses Computers Like a Humancited by 182
  11. 2025An Illusion of Progress? Assessing the Current State of Web Agentscited by 165
  12. 2024OmniACT: A Dataset and Benchmark for Enabling Multimodal Generalist Autonomous Agents for Desktop and Webcited by 156
  13. 2024Android in the Zoo: Chain-of-Action-Thought for GUI Agentscited by 156
  14. 2025Agent S2: A Compositional Generalist-Specialist Framework for Computer Use Agentscited by 156
  15. 2024OS-Genesis: Automating GUI Agent Trajectory Construction via Reverse Task Synthesiscited by 155
  16. 2025Scaling Computer-Use Grounding via User Interface Decomposition and Synthesiscited by 132
  17. 2024AgentTrek: Agent Trajectory Synthesis via Guiding Replay with Web Tutorialscited by 123
  18. 2024Dissecting Adversarial Robustness of Multimodal LM Agentscited by 118
  19. 2024AMEX: Android Multi-annotation Expo Dataset for Mobile GUI Agentscited by 110
  20. 2023Lemur: Harmonizing Natural Language and Code for Language Agentscited by 71
  21. 2025UI-Vision: A Desktop-centric GUI Benchmark for Visual Perception and Interactioncited by 68
  22. 2024AgentStudio: A Toolkit for Building General Virtual Agentscited by 45
  23. 2025TongUI: Internet-Scale Trajectories from Multimodal Web Tutorials for Generalized GUI Agentscited by 40
  24. 2024ReAct Meets ActRe: When Language Agents Enjoy Training Data Autonomycited by 22
  25. 2024WONDERBREAD: A Benchmark for Evaluating Multimodal Foundation Models on Business Process Management Taskscited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 31; 2026: 120.

151 papers.

Sort:

By area:

  1. 2025Qwen3-VL Technical Reportcited by 2,534
  2. 2026Kimi K2.5: Visual Agentic Intelligencecited by 449
  3. 2026Agent Skills for Large Language Models: Architecture, Acquisition, Security, and the Path Forwardcited by 105
  4. 2026OpenClaw-RL: Train Any Agent Simply by Talkingcited by 83
  5. 2025ScaleCUA: Scaling Open-Source Computer Use Agents with Cross-Platform Datacited by 58
  6. 2026EvoCUA: Evolving Computer Use Agents via Learning from Scalable Synthetic Experiencecited by 51
  7. 2025Step-GUI Technical Reportcited by 38
  8. 2025Fara-7B: An Efficient Agentic Model for Computer Usecited by 34
  9. 2025UI-S1: Advancing GUI Automation via Semi-online Reinforcement Learningcited by 32
  10. 2026CUA-Skill: Develop Skills for Computer Using Agentcited by 32
  11. 2025UltraCUA: A Foundation Model for Computer Use Agents with Hybrid Actioncited by 30
  12. 2026LongCat-Next: Lexicalizing Modalities as Discrete Tokenscited by 30
38 more
  1. 2026OSWorld 2.0: Benchmarking Computer Use Agents on Long-Horizon Real-World Taskscited by 29
  2. 2026Unsafer in Many Turns: Benchmarking and Defending Multi-Turn Safety Risks in Tool-Using Agentscited by 28
  3. 2025Efficient Multi-turn RL for GUI Agents via Decoupled Training and Adaptive Data Curationcited by 24
  4. 2026Benchmark Test-Time Scaling of General LLM Agentscited by 24
  5. 2026Gym-Anything: Turn any Software into an Agent Environmentcited by 24
  6. 2025Game-TARS: Pretrained Foundation Models for Scalable Generalist Multimodal Game Agentscited by 23
  7. 2025UItron: Foundational GUI Agent with Advanced Perception and Planningcited by 22
  8. 2026CUA-Gym: Scaling Verifiable Training Environments and Tasks for Computer-Use Agentscited by 21
  9. 2026AutoWebWorld: Synthesizing Infinite Verifiable Web Environments via Finite State Machinescited by 20
  10. 2026RLAnything: Forge Environment, Policy, and Reward Model in Completely Dynamic RL Systemcited by 19
  11. 2025MGA: Memory-Driven GUI Agent for Observation-Centric Interactioncited by 18
  12. 2025OS-Sentinel: Towards Safety-Enhanced Mobile GUI Agents via Hybrid Validation in Realistic Workflowscited by 18
  13. 2025Don't Just Fine-tune the Agent, Tune the Environmentcited by 17
  14. 2025How Do AI Agents Do Human Work? Comparing AI and Human Workflows Across Diverse Occupationscited by 17
  15. 2025CUARewardBench: A Benchmark for Evaluating Reward Models on Computer-using Agentcited by 16
  16. 2025Ferret-UI Lite: Lessons from Building Small On-Device GUI Agentscited by 14
  17. 2025Environment Scaling for Interactive Agentic Experience Collection: A Surveycited by 13
  18. 2026OmegaUse: Building a General-Purpose GUI Agent for Autonomous Task Executioncited by 13
  19. 2026The Blind Spot of Agent Safety: How Benign User Instructions Expose Critical Vulnerabilities in Computer-Use Agentscited by 12
  20. 2026MolmoPoint: Better Pointing for VLMs with Grounding Tokenscited by 11
  21. 2025R-WoM: Retrieval-augmented World Model For Computer-use Agentscited by 10
  22. 2026ToolCUA: Towards Optimal GUI-Tool Path Orchestration for Computer Use Agentscited by 10
  23. 2025Watch and Learn: Learning to Use Computers from Online Videoscited by 9
  24. 2025Just Do It!? Computer-Use Agents Exhibit Blind Goal-Directednesscited by 8
  25. 2025SusBench: An Online Benchmark for Evaluating Dark Pattern Susceptibility of Computer-Use Agentscited by 8
  26. 2025Beyond Pipelines: A Survey of the Paradigm Shift toward Model-Native Agentic AIcited by 8
  27. 2026When Actions Go Off-Task: Detecting and Correcting Misaligned Actions in Computer-Use Agentscited by 8
  28. 2026Qwen-UI-Agent Technical Report: Toward Next-Generation Real-World Centric Foundation GUI Agentscited by 8
  29. 2025SCUBA: Salesforce Computer Use Benchmarkcited by 7
  30. 2026Learning Next Action Predictors from Human-Computer Interactioncited by 7
  31. 2026UI-MOPD: Multi-Platform On-Policy Distillation for Unified GUI Agentscited by 7
  32. 2026SCALECUA: Scaling Computer Use Agents with Verifiable Task Synthesis and Efficient Online RLcited by 7
  33. 2025ColorAgent: Building A Robust, Personalized, and Interactive OS Agentcited by 6
  34. 2026CocoaBench: Evaluating Unified Digital Agents in the Wildcited by 6
  35. 2026SpatialWorld: Benchmarking Interactive Spatial Reasoning of Multimodal Agents in Real-World Taskscited by 6
  36. 2026PPT-Eval: A Benchmark for Computer-Use Agents on PowerPoint Taskscited by 6
  37. 2025Surfer 2: The Next Generation of Cross-Platform Computer Use Agentscited by 5
  38. 2025Chain-of-Ground: Improving GUI Grounding via Iterative Reasoning and Reference Feedbackcited by 5

50 most cited of 151 All 151 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026CUA-Suite: Massive Human-annotated Video Demonstrations for Computer-Use Agents
  2. 2026CUAAudit: Meta-Evaluation of Vision-Language Models as Auditors of Autonomous Computer-Use Agents
  3. 2025GUI-360^circ: A Comprehensive Dataset and Benchmark for Computer-Using Agents
  4. 2026OSReward: Instituting Standardized Evaluation for Cross-Platform Computer-Use Reward Models
  5. 2026cua-speedrun: Standardized Benchmarking of the Speed of Computer-Use Agents
  6. 2025Computer-Use Agents as Judges for Generative User Interface
  7. 2026Qwen-CUA: Native Computer Use for (almost) Everything
  8. 2026Multi-Agent Computer Use