OS-Symphony: A Holistic Framework for Robust and Generalist Computer-Using Agent
On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models
also in 1 more
Abstract (arXiv)
While Vision-Language Models (VLMs) have significantly advanced Computer-Using Agents (CUAs), current frameworks struggle with robustness in long-horizon workflows and generalization in novel domains. These limitations stem from a lack of granular control over historical visual context curation and the absence of visual-aware tutorial retrieval. To bridge these gaps, we introduce OS-Symphony, a holistic framework that comprises an Orchestrator coordinating two key innovations for robust automation: (1) a Reflection-Memory Agent that utilizes milestone-driven long-term memory to enable trajectory-level self-correction, effectively mitigating visual context loss in long-horizon tasks; (2) Versatile Tool Agents featuring a Multimodal Searcher that adopts a SeeAct paradigm to navigate a browser-based sandbox to synthesize live, visually aligned tutorials, thereby resolving fidelity issues in unseen scenarios. Experimental results demonstrate that OS-Symphony delivers substantial performance gains across varying model scales, establishing new state-of-the-art results on three online benchmarks, notably achieving 65.84% on OSWorld.
Key facts. Cited by 19 papers in the map (data through 7 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 39 of its 58 references that are in the map. First posted on arXiv 12 Jan 2026.
Builds on
39 of 58 references.
Closest ancestors · 4
Same direction, newest first
- 2025MGA: Memory-Driven GUI Agent for Observation-Centric Interaction28 Oct
- 2025Retrieval-augmented GUI Agents with Generative Guidelines29 Sep
- 2025ScaleCUA: Scaling Open-Source Computer Use Agents with Cross-Platform Data18 Sep
- 2024OS-ATLAS: A Foundation Action Model for Generalist GUI Agents30 Oct
Other references · 35
- 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
- 2025InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiencycited by 1,419
- 2024OSWorld: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environmentscited by 1,134
- 2024A Survey on the Memory Mechanism of Large Language Model based Agentscited by 622
- 2024SeeClick: Harnessing GUI Grounding for Advanced Visual GUI Agentscited by 553
- 2024GPT-4V(ision) is a Generalist Web Agent, if Groundedcited by 547
- 2024Aguvis: Unified Pure Vision Agents for Autonomous GUI Interactioncited by 271
- 2024Windows Agent Arena: Evaluating Multi-Modal OS Agents at Scalecited by 212
- 2025UI-TARS-2 Technical Report: Advancing GUI Agent with Multi-Turn Reinforcement Learningcited by 205
- 2024Agent S: An Open Agentic Framework that Uses Computers Like a Humancited by 182
- 2025Agent S2: A Compositional Generalist-Specialist Framework for Computer Use Agentscited by 156
- 2024OS-Genesis: Automating GUI Agent Trajectory Construction via Reverse Task Synthesiscited by 155
23 more
- 2025Scaling Long-Horizon LLM Agent via Context-Foldingcited by 136
- 2025Scaling Computer-Use Grounding via User Interface Decomposition and Synthesiscited by 132
- 2024AgentTrek: Agent Trajectory Synthesis via Guiding Replay with Web Tutorialscited by 123
- 2025GTA1: GUI Test-time Scaling Agentcited by 102
- 2023Corex: Pushing the Boundaries of Complex Reasoning through Multi-Model Collaborationcited by 70
- 2025DeepAgent: A General Reasoning Agent with Scalable Toolsetscited by 69
- 2025ComputerRL: Scaling End-to-End Online Reinforcement Learning for Computer Use Agentscited by 49
- 2025PC-Agent: A Hierarchical Multi-Agent Collaboration Framework for Complex Task Automation on PCcited by 39
- 2025OS Agents: A Survey on MLLM-based Agents for General Computing Devices Usecited by 35
- 2025UltraCUA: A Foundation Model for Computer Use Agents with Hybrid Actioncited by 30
- 2025API Agents vs. GUI Agents: Divergence and Convergencecited by 28
- 2024AgentStore: Scalable Integration of Heterogeneous Agents As Specialized Generalist Computer Assistantcited by 27
- 2025BacktrackAgent: Enhancing GUI Agent with Error Detection and Backtracking Mechanismcited by 25
- 2025Game-TARS: Pretrained Foundation Models for Scalable Generalist Multimodal Game Agentscited by 23
- 2025WorldGUI: An Interactive Benchmark for Desktop GUI Automation from Any Starting Pointcited by 19
- 2025Grounding Computer Use Agents on Human Demonstrationscited by 16
- 2025Breaking the Data Barrier -- Building GUI Agents Through Task Generalizationcited by 13
- 2025R-WoM: Retrieval-augmented World Model For Computer-use Agentscited by 10
- 2025AgentProg: Empowering Long-Horizon GUI Agents with Program-Guided Context Managementcited by 10
- 2025Agentic Lybic: Multi-Agent Execution System with Tiered Reasoning and Orchestrationcited by 8
- 2025OS-Oracle: A Comprehensive Framework for Cross-Platform GUI Critic Modelscited by 6
- 2025Nested Browser-Use Learning for Agentic Information Seekingcited by 5
- 2025UI-CUBE: Enterprise-Grade Computer Use Agent Benchmarking Beyond Task Accuracy to Operational Reliabilitycited by 3
Built on by
Citing papers per year: 2026: 19.
19 papers.
Sort:
By area:
- 2026Natural-Language Agent Harnessescited by 54
- 2026OmegaUse: Building a General-Purpose GUI Agent for Autonomous Task Executioncited by 13
- 2026Agent Alpha: Tree Search Unifying Generation, Exploration and Evaluation for Computer-Use Agentscited by 12
- 2026OpenMobile: Building Open Mobile Agents with Task and Trajectory Synthesiscited by 10
- 2026ToolCUA: Towards Optimal GUI-Tool Path Orchestration for Computer Use Agentscited by 10
- 2026OS-Themis: A Scalable Critic Framework for Generalist GUI Rewardscited by 9
- 2026GUIDE: Resolving Domain Bias in GUI Agents through Real-Time Web Video Retrieval and Plug-and-Play Annotationcited by 4
- 2026VLAA-GUI: Knowing When to Stop, Recover, and Search, A Modular Framework for GUI Automationcited by 4
- 2026StructAgent: Harness Long-horizon Digital Agents with Unified Causal Structurecited by 4
- 2026MAGA: Multi-Platform Self-Fusion of GUI Agents via Structured Action Distillationcited by 3
- 2026MacAgentBench: Benchmarking AI Agents on Real-World macOS Desktopcited by 2
- 2026Retrieval, Reward, and Training Protocols: What Matters in Training Search Agents?cited by 1
7 more
- 2026A History-Aware Visually Grounded Critic for Computer Use Agentscited by 1
- 2026Global Commander and Local Operative: A Dual-Agent Framework for Scene Navigationcited by 0
- 2026SynChain: Inducing Computer-Use Agent Systems to Construct Their Own Attack Chainscited by 0
- 2026LocalLSTC: A Long Short-Term Control Architecture for Locally Deployed GUI Agentscited by 0
- 2026Efficient GUI Agents: A Systems Survey of Observation, Memory, Action, and Runtime Optimizationcited by 0
- 2026AnyAct: Universal Action for Self-Evolving Agentscited by 0
- 2026GUI-HARVEST: Self-Improving GUI Agents through Evidence-Driven Harness Evolutioncited by 0
- 2026GUI-HARVEST: Self-Improving GUI Agents through Evidence-Driven Harness Evolution1 Oct 2026
- 2026AnyAct: Universal Action for Self-Evolving Agents29 Sep 2026
- 2026Efficient GUI Agents: A Systems Survey of Observation, Memory, Action, and Runtime Optimization2 Sep 2026
- 2026LocalLSTC: A Long Short-Term Control Architecture for Locally Deployed GUI Agents26 Aug 2026
- 2026SynChain: Inducing Computer-Use Agent Systems to Construct Their Own Attack Chains7 Aug 2026
- 2026MAGA: Multi-Platform Self-Fusion of GUI Agents via Structured Action Distillation31 Jul 2026
- 2026StructAgent: Harness Long-horizon Digital Agents with Unified Causal Structure13 Jul 2026
- 2026MacAgentBench: Benchmarking AI Agents on Real-World macOS Desktop21 Jun 2026
- 2026A History-Aware Visually Grounded Critic for Computer Use Agents9 Jun 2026
- 2026Retrieval, Reward, and Training Protocols: What Matters in Training Search Agents?27 May 2026
- 2026ToolCUA: Towards Optimal GUI-Tool Path Orchestration for Computer Use Agents12 May 2026
- 2026VLAA-GUI: Knowing When to Stop, Recover, and Search, A Modular Framework for GUI Automation23 Apr 2026
7 more
- 2026OpenMobile: Building Open Mobile Agents with Task and Trajectory Synthesis16 Apr 2026
- 2026GUIDE: Resolving Domain Bias in GUI Agents through Real-Time Web Video Retrieval and Plug-and-Play Annotation27 Mar 2026
- 2026Natural-Language Agent Harnesses26 Mar 2026
- 2026OS-Themis: A Scalable Critic Framework for Generalist GUI Rewards19 Mar 2026
- 2026Global Commander and Local Operative: A Dual-Agent Framework for Scene Navigation21 Feb 2026
- 2026Agent Alpha: Tree Search Unifying Generation, Exploration and Evaluation for Computer-Use Agents3 Feb 2026
- 2026OmegaUse: Building a General-Purpose GUI Agent for Autonomous Task Execution28 Jan 2026
- 2026OmegaUse: Building a General-Purpose GUI Agent for Autonomous Task Executioncited by 13
- 2026OpenMobile: Building Open Mobile Agents with Task and Trajectory Synthesiscited by 10
- 2026ToolCUA: Towards Optimal GUI-Tool Path Orchestration for Computer Use Agentscited by 10
- 2026OS-Themis: A Scalable Critic Framework for Generalist GUI Rewardscited by 9
- 2026VLAA-GUI: Knowing When to Stop, Recover, and Search, A Modular Framework for GUI Automationcited by 4
- 2026MAGA: Multi-Platform Self-Fusion of GUI Agents via Structured Action Distillationcited by 3
- 2026MacAgentBench: Benchmarking AI Agents on Real-World macOS Desktopcited by 2
- 2026Retrieval, Reward, and Training Protocols: What Matters in Training Search Agents?cited by 1
- 2026A History-Aware Visually Grounded Critic for Computer Use Agentscited by 1
- 2026Global Commander and Local Operative: A Dual-Agent Framework for Scene Navigationcited by 0
- 2026SynChain: Inducing Computer-Use Agent Systems to Construct Their Own Attack Chainscited by 0
- 2026LocalLSTC: A Long Short-Term Control Architecture for Locally Deployed GUI Agentscited by 0
12 most cited of 15 All 15 (JSON)
- 2026GUI-HARVEST: Self-Improving GUI Agents through Evidence-Driven Harness Evolution1 Oct 2026
- 2026AnyAct: Universal Action for Self-Evolving Agents29 Sep 2026
- 2026Efficient GUI Agents: A Systems Survey of Observation, Memory, Action, and Runtime Optimization2 Sep 2026
- 2026LocalLSTC: A Long Short-Term Control Architecture for Locally Deployed GUI Agents26 Aug 2026
- 2026SynChain: Inducing Computer-Use Agent Systems to Construct Their Own Attack Chains7 Aug 2026
- 2026MAGA: Multi-Platform Self-Fusion of GUI Agents via Structured Action Distillation31 Jul 2026
- 2026MacAgentBench: Benchmarking AI Agents on Real-World macOS Desktop21 Jun 2026
- 2026A History-Aware Visually Grounded Critic for Computer Use Agents9 Jun 2026
- 2026Retrieval, Reward, and Training Protocols: What Matters in Training Search Agents?27 May 2026
- 2026ToolCUA: Towards Optimal GUI-Tool Path Orchestration for Computer Use Agents12 May 2026
- 2026VLAA-GUI: Knowing When to Stop, Recover, and Search, A Modular Framework for GUI Automation23 Apr 2026
- 2026OpenMobile: Building Open Mobile Agents with Task and Trajectory Synthesis16 Apr 2026
12 newest of 15 All 15 (JSON)
- 2026Natural-Language Agent Harnessescited by 54
- 2026Natural-Language Agent Harnesses26 Mar 2026
Similar, not cited
Close in wording, no citation either way
- 2026OS-SPEAR: A Toolkit for the Safety, Performance,Efficiency, and Robustness Analysis of OS Agents
- 2026CUAAudit: Meta-Evaluation of Vision-Language Models as Auditors of Autonomous Computer-Use Agents
- 2026MUSE: A Unified Agentic Harness for MLLMs
- 2026MintAct: A Unified Visual Agent for Digital Environments
- 2026Adaptive Vision-Language Model Routing for Computer Use Agents
- 2024MMFactory: A Universal Solution Search Engine for Vision-Language Tasks
- 2026RoboHarness: A Memory-Augmented Policy Harness for Vision-Language-Action Model Robustness via In-Context Adaptation
- 2024ScreenAgent: A Vision Language Model-driven Computer Control Agent