# OS-Symphony: A Holistic Framework for Robust and Generalist Computer-Using Agent

arXiv: https://arxiv.org/abs/2601.07779 · HTML: https://mlsanity.com/paper/2601.07779/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 39 of its 58 references that are in the map.
- First posted on arXiv 12 Jan 2026.

## Placed in

- Multimodal LLMs / vision-language models / [Embodied and 3D vision-language models](https://mlsanity.com/mllm/embodied-and-3d-vision-language-models/)
- LLM agents and tool use / [Multimodal and embodied agents](https://mlsanity.com/agents/multimodal-and-embodied-agents/)

## Builds on (most cited first, 30 of 39)

- [ReAct: Synergizing Reasoning and Acting in Language Models](https://mlsanity.com/paper/2210.03629/) (2022)
- [InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiency](https://mlsanity.com/paper/2508.18265/) (2025)
- [OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments](https://mlsanity.com/paper/2404.07972/) (2024)
- [A Survey on the Memory Mechanism of Large Language Model based Agents](https://mlsanity.com/paper/2404.13501/) (2024)
- [SeeClick: Harnessing GUI Grounding for Advanced Visual GUI Agents](https://mlsanity.com/paper/2401.10935/) (2024)
- [GPT-4V(ision) is a Generalist Web Agent, if Grounded](https://mlsanity.com/paper/2401.01614/) (2024)
- [OS-ATLAS: A Foundation Action Model for Generalist GUI Agents](https://mlsanity.com/paper/2410.23218/) (2024)
- [Aguvis: Unified Pure Vision Agents for Autonomous GUI Interaction](https://mlsanity.com/paper/2412.04454/) (2024)
- [Windows Agent Arena: Evaluating Multi-Modal OS Agents at Scale](https://mlsanity.com/paper/2409.08264/) (2024)
- [UI-TARS-2 Technical Report: Advancing GUI Agent with Multi-Turn Reinforcement Learning](https://mlsanity.com/paper/2509.02544/) (2025)
- [Agent S: An Open Agentic Framework that Uses Computers Like a Human](https://mlsanity.com/paper/2410.08164/) (2024)
- [OS-Genesis: Automating GUI Agent Trajectory Construction via Reverse Task Synthesis](https://mlsanity.com/paper/2412.19723/) (2024)
- [Agent S2: A Compositional Generalist-Specialist Framework for Computer Use Agents](https://mlsanity.com/paper/2504.00906/) (2025)
- [Scaling Long-Horizon LLM Agent via Context-Folding](https://mlsanity.com/paper/2510.11967/) (2025)
- [Scaling Computer-Use Grounding via User Interface Decomposition and Synthesis](https://mlsanity.com/paper/2505.13227/) (2025)
- [AgentTrek: Agent Trajectory Synthesis via Guiding Replay with Web Tutorials](https://mlsanity.com/paper/2412.09605/) (2024)
- [GTA1: GUI Test-time Scaling Agent](https://mlsanity.com/paper/2507.05791/) (2025)
- [DeepAgent: A General Reasoning Agent with Scalable Toolsets](https://mlsanity.com/paper/2510.21618/) (2025)
- [Corex: Pushing the Boundaries of Complex Reasoning through Multi-Model Collaboration](https://mlsanity.com/paper/2310.00280/) (2023)
- [ScaleCUA: Scaling Open-Source Computer Use Agents with Cross-Platform Data](https://mlsanity.com/paper/2509.15221/) (2025)
- [ComputerRL: Scaling End-to-End Online Reinforcement Learning for Computer Use Agents](https://mlsanity.com/paper/2508.14040/) (2025)
- [PC-Agent: A Hierarchical Multi-Agent Collaboration Framework for Complex Task Automation on PC](https://mlsanity.com/paper/2502.14282/) (2025)
- [OS Agents: A Survey on MLLM-based Agents for General Computing Devices Use](https://mlsanity.com/paper/2508.04482/) (2025)
- [UltraCUA: A Foundation Model for Computer Use Agents with Hybrid Action](https://mlsanity.com/paper/2510.17790/) (2025)
- [API Agents vs. GUI Agents: Divergence and Convergence](https://mlsanity.com/paper/2503.11069/) (2025)
- [AgentStore: Scalable Integration of Heterogeneous Agents As Specialized Generalist Computer Assistant](https://mlsanity.com/paper/2410.18603/) (2024)
- [BacktrackAgent: Enhancing GUI Agent with Error Detection and Backtracking Mechanism](https://mlsanity.com/paper/2505.20660/) (2025)
- [Game-TARS: Pretrained Foundation Models for Scalable Generalist Multimodal Game Agents](https://mlsanity.com/paper/2510.23691/) (2025)
- [Grounding Computer Use Agents on Human Demonstrations](https://mlsanity.com/paper/2511.07332/) (2025)
- [WorldGUI: An Interactive Benchmark for Desktop GUI Automation from Any Starting Point](https://mlsanity.com/paper/2502.08047/) (2025)

## Built on by (most cited first, 21 of 21)

- [Natural-Language Agent Harnesses](https://mlsanity.com/paper/2603.25723/) (2026)
- [OmegaUse: Building a General-Purpose GUI Agent for Autonomous Task Execution](https://mlsanity.com/paper/2601.20380/) (2026)
- [Agent Alpha: Tree Search Unifying Generation, Exploration and Evaluation for Computer-Use Agents](https://mlsanity.com/paper/2602.02995/) (2026)
- [OpenMobile: Building Open Mobile Agents with Task and Trajectory Synthesis](https://mlsanity.com/paper/2604.15093/) (2026)
- [ToolCUA: Towards Optimal GUI-Tool Path Orchestration for Computer Use Agents](https://mlsanity.com/paper/2605.12481/) (2026)
- [OS-Themis: A Scalable Critic Framework for Generalist GUI Rewards](https://mlsanity.com/paper/2603.19191/) (2026)
- [StructAgent: Harness Long-horizon Digital Agents with Unified Causal Structure](https://mlsanity.com/paper/2607.11388/) (2026)
- [From Chatbot to Digital Colleague: The Paradigm Shift Toward Persistent Autonomous AI](https://mlsanity.com/paper/2606.14502/) (2026)
- [VLAA-GUI: Knowing When to Stop, Recover, and Search, A Modular Framework for GUI Automation](https://mlsanity.com/paper/2604.21375/) (2026)
- [GUIDE: Resolving Domain Bias in GUI Agents through Real-Time Web Video Retrieval and Plug-and-Play Annotation](https://mlsanity.com/paper/2603.26266/) (2026)
- [MAGA: Multi-Platform Self-Fusion of GUI Agents via Structured Action Distillation](https://mlsanity.com/paper/2607.29320/) (2026)
- [MacAgentBench: Benchmarking AI Agents on Real-World macOS Desktop](https://mlsanity.com/paper/2606.22557/) (2026)
- [Retrieval, Reward, and Training Protocols: What Matters in Training Search Agents?](https://mlsanity.com/paper/2605.27881/) (2026)
- [A History-Aware Visually Grounded Critic for Computer Use Agents](https://mlsanity.com/paper/2606.11078/) (2026)
- [Global Commander and Local Operative: A Dual-Agent Framework for Scene Navigation](https://mlsanity.com/paper/2602.18941/) (2026)
- [SynChain: Inducing Computer-Use Agent Systems to Construct Their Own Attack Chains](https://mlsanity.com/paper/2608.06862/) (2026)
- [LocalLSTC: A Long Short-Term Control Architecture for Locally Deployed GUI Agents](https://mlsanity.com/paper/2608.25777/) (2026)
- [Efficient GUI Agents: A Systems Survey of Observation, Memory, Action, and Runtime Optimization](https://mlsanity.com/paper/2609.02309/) (2026)
- [AnyAct: Universal Action for Self-Evolving Agents](https://mlsanity.com/paper/2609.37025/) (2026)
- [GUI-HARVEST: Self-Improving GUI Agents through Evidence-Driven Harness Evolution](https://mlsanity.com/paper/2610.00948/) (2026)
- [OpenMobile-2: Building Versatile Mobile Agents with Scalable Environments and App-Native Tools](https://mlsanity.com/paper/2610.14157/) (2026)

## Abstract (arXiv)

While Vision-Language Models (VLMs) have significantly advanced Computer-Using Agents (CUAs), current frameworks struggle with robustness in long-horizon workflows and generalization in novel domains. These limitations stem from a lack of granular control over historical visual context curation and the absence of visual-aware tutorial retrieval. To bridge these gaps, we introduce OS-Symphony, a holistic framework that comprises an Orchestrator coordinating two key innovations for robust automation: (1) a Reflection-Memory Agent that utilizes milestone-driven long-term memory to enable trajectory-level self-correction, effectively mitigating visual context loss in long-horizon tasks; (2) Versatile Tool Agents featuring a Multimodal Searcher that adopts a SeeAct paradigm to navigate a browser-based sandbox to synthesize live, visually aligned tutorials, thereby resolving fidelity issues in unseen scenarios. Experimental results demonstrate that OS-Symphony delivers substantial performance gains across varying model scales, establishing new state-of-the-art results on three online benchmarks, notably achieving 65.84% on OSWorld.