mlsanityData to 14 Oct 2026

CODA: Coordinating the Cerebrum and Cerebellum for a Dual-Brain Computer Use Agent with Decoupled Reinforcement Learning

arXiv 2508.20096 ↗First posted 27 Aug 2025cs.CV cs.AI cs.LGcited by 6

On the map: LLM agents and tool use → GUI and mobile agents

also in 1 more
Abstract (arXiv)

Autonomous agents for Graphical User Interfaces (GUIs) face significant challenges in specialized domains such as scientific computing, where both long-horizon planning and precise execution are required. Existing approaches suffer from a trade-off: generalist agents excel at planning but perform poorly in execution, while specialized agents demonstrate the opposite weakness. Recent compositional frameworks attempt to bridge this gap by combining a planner and an actor, but they are typically static and non-trainable, which prevents adaptation from experience. This is a critical limitation given the scarcity of high-quality data in scientific domains. To address these limitations, we introduce CODA, a novel and trainable compositional framework that integrates a generalist planner (Cerebrum) with a specialist executor (Cerebellum), trained via a dedicated two-stage pipeline. In the first stage, Specialization, we apply a decoupled GRPO approach to train an expert planner for each scientific application individually, bootstrapping from a small set of task trajectories. In the second stage, Generalization, we aggregate all successful trajectories from the specialized experts to build a consolidated dataset, which is then used for supervised fine-tuning of the final planner. This equips CODA with both robust execution and cross-domain generalization. Evaluated on four challenging applications from the ScienceBoard benchmark, CODA significantly outperforms baselines and establishes a new state of the art among open-source models.

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Placed in GUI and mobile agents, one of 16 directions in LLM agents and tool use. Builds on 44 of its 66 references that are in the map. First posted on arXiv 27 Aug 2025.

Builds on

44 of 66 references.

Closest ancestors · 13

Same direction, newest first

  1. 2025GUI-Actor: Coordinate-Free Visual Grounding for GUI Agents3 Jun
  2. 2025Scaling Computer-Use Grounding via User Interface Decomposition and Synthesis19 May
  3. 2025Agent S2: A Compositional Generalist-Specialist Framework for Computer Use Agents1 Apr
  4. 2025UI-TARS: Pioneering Automated GUI Interaction with Native Agents21 Jan
  5. 2024OS-Genesis: Automating GUI Agent Trajectory Construction via Reverse Task Synthesis27 Dec
  6. 2024GUI Agents: A Survey18 Dec
  7. 2024OS-ATLAS: A Foundation Action Model for Generalist GUI Agents30 Oct
  8. 2024Agent S: An Open Agentic Framework that Uses Computers Like a Human10 Oct
  9. 2024Navigating the Digital World as Humans Do: Universal Visual Grounding for GUI Agents7 Oct
  10. 2024Android in the Zoo: Chain-of-Action-Thought for GUI Agents5 Mar
  11. 2024OS-Copilot: Towards Generalist Computer Agents with Self-Improvement12 Feb
  12. 2024SeeClick: Harnessing GUI Grounding for Advanced Visual GUI Agents17 Jan
1 more
  1. 2023CogAgent: A Visual Language Model for GUI Agents14 Dec

Other references · 31

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2024The Llama 3 Herd of Modelscited by 14,715
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  4. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  5. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,330
  6. 2025Qwen2.5-VL Technical Reportcited by 5,587
  7. 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,234
  8. 2023Voyager: An Open-Ended Embodied Agent with Large Language Modelscited by 2,191
  9. 2019Fine-Tuning Language Models from Human Preferencescited by 2,150
  10. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 2,026
  11. 2023WebArena: A Realistic Web Environment for Building Autonomous Agentscited by 2,006
  12. 2025InternVL3: Exploring Advanced Training and Test-Time Recipes for Open-Source Multimodal Modelscited by 1,834
19 more
  1. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,745
  2. 2023Gemini: A Family of Highly Capable Multimodal Modelscited by 872
  3. 2024WebVoyager: Building an End-to-End Web Agent with Large Multimodal Modelscited by 438
  4. 2024WebRL: Training LLM Web Agents via Self-Evolving Online Curriculum Reinforcement Learningcited by 256
  5. 2024ShowUI: One Vision-Language-Action Model for GUI Visual Agentcited by 232
  6. 2024ArCHer: Training Language Model Agents via Hierarchical Multi-Turn RLcited by 228
  7. 2024Agent Q: Advanced Reasoning and Learning for Autonomous AI Agentscited by 208
  8. 2023GPT-4V in Wonderland: Large Multimodal Models for Zero-Shot Smartphone GUI Navigationcited by 137
  9. 2024Autonomous Evaluation and Refinement of Digital Agentscited by 137
  10. 2025AgentRewardBench: Evaluating Automatic Evaluations of Web Agent Trajectoriescited by 108
  11. 2023BOLAA: Benchmarking and Orchestrating LLM-augmented Autonomous Agentscited by 84
  12. 2024OpenRLHF: An Easy-to-use, Scalable and High-performance RLHF Frameworkcited by 73
  13. 2025SEAgent: Self-Evolving Computer Use Agent with Autonomous Learning from Experiencecited by 69
  14. 2025ScienceBoard: Evaluating Multimodal Autonomous Agents in Realistic Scientific Workflowscited by 52
  15. 2025MM-IFEngine: Towards Multimodal Instruction Followingcited by 42
  16. 2025CoAct-1: Computer-using Multi-Agent System with Coding Actionscited by 35
  17. 2024X-Prompt: Towards Universal In-Context Image Generation in Auto-Regressive Vision Language Foundation Modelscited by 15
  18. 2025ScaleCap: Inference-Time Scalable Image Captioning via Dual-Modality Debiasingcited by 15
  19. 2024A Survey of Neural Code Intelligence: Paradigms, Advances and Beyondcited by 12

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 3; 2026: 3.

6 papers.

Similar, not cited

Close in wording, no citation either way

  1. 2025RoboOS: A Hierarchical Embodied Framework for Cross-Embodiment and Multi-Agent Collaboration
  2. 2019Plan Arithmetic: Compositional Plan Vectors for Multi-Task Control
  3. 2025OmniActor: A Generalist GUI and Embodied Agent for 2D&3D Worlds
  4. 2025Brain-Inspired Planning for Better Generalization in Reinforcement Learning
  5. 2026Skill2Real: Agentic Skill Learning for Zero-Shot Sim-to-Real Robot Manipulation
  6. 2025The brain-AI convergence: Predictive and generative world models for general-purpose computation
  7. 2025RoBridge: A Hierarchical Architecture Bridging Cognition and Execution for General Robotic Manipulation
  8. 2026NS-Copilot: An LLM-Driven Agent System for Autonomous Neuroscience Analysis