# UFO3: Weaving the Digital Agent Galaxy

arXiv: https://arxiv.org/abs/2511.11332 · HTML: https://mlsanity.com/paper/2511.11332/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Placed in General LLM agent frameworks and benchmarks, one of 16 directions in LLM agents and tool use.
- Builds on 32 of its 63 references that are in the map.
- First posted on arXiv 14 Nov 2025.

## Placed in

- LLM agents and tool use / [General LLM agent frameworks and benchmarks](https://mlsanity.com/agents/general-llm-agent-frameworks-and-benchmarks/)

## Builds on (most cited first, 30 of 32)

- [Large Language Model based Multi-Agents: A Survey of Progress and Challenges](https://mlsanity.com/paper/2402.01680/) (2024)
- [A Survey on In-context Learning](https://mlsanity.com/paper/2301.00234/) (2022)
- [The Dawn of LMMs: Preliminary Explorations with GPT-4V(ision)](https://mlsanity.com/paper/2309.17421/) (2023)
- [Why Do Multi-Agent LLM Systems Fail?](https://mlsanity.com/paper/2503.13657/) (2025)
- [GPT-4V(ision) is a Generalist Web Agent, if Grounded](https://mlsanity.com/paper/2401.01614/) (2024)
- [Multi-Agent Collaboration Mechanisms: A Survey of LLMs](https://mlsanity.com/paper/2501.06322/) (2025)
- [Understanding the planning of LLM agents: A survey](https://mlsanity.com/paper/2402.02716/) (2024)
- [Mobile-Agent: Autonomous Multi-Modal Mobile Device Agent with Visual Perception](https://mlsanity.com/paper/2401.16158/) (2024)
- [Model Context Protocol (MCP): Landscape, Security Threats, and Future Research Directions](https://mlsanity.com/paper/2503.23278/) (2025)
- [GUI-R1 : A Generalist R1-Style Vision-Language Action Model For GUI Agents](https://mlsanity.com/paper/2504.10458/) (2025)
- [UI-TARS-2 Technical Report: Advancing GUI Agent with Multi-Turn Reinforcement Learning](https://mlsanity.com/paper/2509.02544/) (2025)
- [Large Language Model-Brained GUI Agents: A Survey](https://mlsanity.com/paper/2411.18279/) (2024)
- [Deep Research Agents: A Systematic Examination And Roadmap](https://mlsanity.com/paper/2506.18096/) (2025)
- [Multi-Agent Collaboration via Evolving Orchestration](https://mlsanity.com/paper/2505.19591/) (2025)
- [A survey of agent interoperability protocols: Model Context Protocol (MCP), Agent Communication Protocol (ACP), Agent-to-Agent Protocol (A2A), and Agent Network Protocol (ANP)](https://mlsanity.com/paper/2505.02279/) (2025)
- [GUI-Actor: Coordinate-Free Visual Grounding for GUI Agents](https://mlsanity.com/paper/2506.03143/) (2025)
- [TaskWeaver: A Code-First Agent Framework](https://mlsanity.com/paper/2311.17541/) (2023)
- [A Survey of AI Agent Protocols](https://mlsanity.com/paper/2504.16736/) (2025)
- [SWE-bench Goes Live!](https://mlsanity.com/paper/2505.23419/) (2025)
- [A Survey of LLM-Driven AI Agent Communication: Protocols, Security Risks, and Defense Countermeasures](https://mlsanity.com/paper/2506.19676/) (2025)
- [Agentic Web: Weaving the Next Web with AI Agents](https://mlsanity.com/paper/2507.21206/) (2025)
- [UFO2: The Desktop AgentOS](https://mlsanity.com/paper/2504.14603/) (2025)
- [Graphs Meet AI Agents: Taxonomy, Progress, and Future Opportunities](https://mlsanity.com/paper/2506.18019/) (2025)
- [DynTaskMAS: A Dynamic Task Graph-driven Framework for Asynchronous and Parallel LLM-based Multi-Agent Systems](https://mlsanity.com/paper/2503.07675/) (2025)
- [VEM: Environment-Free Exploration for Training GUI Agent with Value Environment Model](https://mlsanity.com/paper/2502.18906/) (2025)
- [Large Action Models: From Inception to Implementation](https://mlsanity.com/paper/2412.10047/) (2024)
- [AgentKit: Structured LLM Reasoning with Dynamic Graphs](https://mlsanity.com/paper/2404.11483/) (2024)
- [Learning GUI Grounding with Spatial Reasoning from Visual Feedback](https://mlsanity.com/paper/2509.21552/) (2025)
- [GUI-360^circ: A Comprehensive Dataset and Benchmark for Computer-Using Agents](https://mlsanity.com/paper/2511.04307/) (2025)
- [When Should We Orchestrate Multiple Agents?](https://mlsanity.com/paper/2503.13577/) (2025)

## Built on by (most cited first, 6 of 6)

- [ToolCUA: Towards Optimal GUI-Tool Path Orchestration for Computer Use Agents](https://mlsanity.com/paper/2605.12481/) (2026)
- [GameplayQA: A Benchmarking Framework for Decision-Dense POV-Synced Multi-Video Understanding of 3D Virtual Agents](https://mlsanity.com/paper/2603.24329/) (2026)
- [Exploring LLM Agent Designs and Interaction Modalities for Scientific Visualization](https://mlsanity.com/paper/2604.27996/) (2026)
- [ComAct: Reframing Professional Software Manipulation via COM-as-Action Paradigm](https://mlsanity.com/paper/2606.13239/) (2026)
- [OctoNest: Adaptive Cross-Device Execution through Stateful Control](https://mlsanity.com/paper/2606.20487/) (2026)
- [Beyond Sequential Interaction: Benchmarking Parallel Execution and Coordination for GUI Agents](https://mlsanity.com/paper/2607.22689/) (2026)

## Abstract (arXiv)

Large language model (LLM)-powered agents are transforming digital devices from passive tools into proactive intelligent collaborators. However, most existing frameworks remain confined to a single OS or device, making cross-device workflows brittle and largely manual. We present UFO3, a system that unifies heterogeneous endpoints, desktops, servers, mobile devices, and edge, into a single orchestration fabric. UFO3 models each user request as a mutable TaskConstellation: a distributed DAG of atomic subtasks (TaskStars) with explicit control and data dependencies (TaskStarLines). The TaskConstellation continuously evolves as results stream in from distributed devices, enabling asynchronous execution, adaptive recovery, and dynamic optimization. A Constellation Orchestrator executes tasks safely and asynchronously while applying dynamic DAG updates, and the Agent Interaction Protocol (AIP) provides persistent, low-latency channels for reliable task dispatch and result streaming. These designs dissolve the traditional boundaries between devices and platforms, allowing agents to collaborate seamlessly and amplify their collective intelligence. We evaluate UFO3 on NebulaBench, a benchmark of 55 cross-device tasks across 5 machines and 10 categories. UFO3 achieves 83.3% subtask completion, 70.9% task success, exposes parallelism with an average width of 1.72, and reduces end-to-end latency by 31% relative to a sequential baseline. Fault-injection experiments demonstrate graceful degradation and recovery under transient and permanent agent failures. These results show that UFO3 achieves accurate, efficient, and resilient task orchestration across heterogeneous devices, uniting isolated agents into a coherent, adaptive computing fabric that extends across the landscape of ubiquitous computing.