mlsanityData to 14 Oct 2026

VIKI-R: Coordinating Embodied Multi-Agent Cooperation via Reinforcement Learning

arXiv 2506.09049 ↗First posted 10 Jun 2025cs.AI cs.CV cs.ROcited by 18

On the map: Multimodal LLMs / vision-language models → Embodied and 3D vision-language models

also in 1 more
Abstract (arXiv)

Coordinating multiple embodied agents in dynamic environments remains a core challenge in artificial intelligence, requiring both perception-driven reasoning and scalable cooperation strategies. While recent works have leveraged large language models (LLMs) for multi-agent planning, a few have begun to explore vision-language models (VLMs) for visual reasoning. However, these VLM-based approaches remain limited in their support for diverse embodiment types. In this work, we introduce VIKI-Bench, the first hierarchical benchmark tailored for embodied multi-agent cooperation, featuring three structured levels: agent activation, task planning, and trajectory perception. VIKI-Bench includes diverse robot embodiments, multi-view visual observations, and structured supervision signals to evaluate reasoning grounded in visual inputs. To demonstrate the utility of VIKI-Bench, we propose VIKI-R, a two-stage framework that fine-tunes a pretrained vision-language model (VLM) using Chain-of-Thought annotated demonstrations, followed by reinforcement learning under multi-level reward signals. Our extensive experiments show that VIKI-R significantly outperforms baselines method across all task levels. Furthermore, we show that reinforcement learning enables the emergence of compositional cooperation patterns among heterogeneous agents. Together, VIKI-Bench and VIKI-R offer a unified testbed and method for advancing multi-agent, visual-driven cooperation in embodied AI systems.

Key facts. Cited by 18 papers in the map (data through 14 Oct 2026). Placed in Embodied and 3D vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 55 of its 70 references that are in the map. First posted on arXiv 10 Jun 2025.

Builds on

55 of 70 references.

Closest ancestors · 4

Same direction, newest first

  1. 2025RoboRefer: Towards Spatial Referring with Reasoning in Vision-Language Models for Robotics4 Jun
  2. 2025NoisyRollout: Reinforcing Visual Reasoning with Data Augmentation17 Apr
  3. 2025RoboBrain: A Unified Brain Model for Robotic Manipulation from Abstract to Concrete28 Feb
  4. 2023G-LLaVA: Solving Geometric Problem with Multi-Modal Large Language Model18 Dec

Other references · 51

  1. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  2. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  3. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
  4. 2025Qwen2.5-VL Technical Reportcited by 5,587
  5. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,182
  6. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scalecited by 2,810
  7. 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,719
  8. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,919
  9. 2023Improving Factuality and Reasoning in Language Models through Multiagent Debatecited by 1,784
  10. 2024WorldSimBench: Towards Video Generation Models as World Simulatorscited by 881
  11. 2024Large Language Model based Multi-Agents: A Survey of Progress and Challengescited by 769
  12. 2025Visual-RFT: Visual Reinforcement Fine-Tuningcited by 513
38 more
  1. 2024RoboCasa: Large-Scale Simulation of Everyday Tasks for Generalist Robotscited by 407
  2. 2023SOTOPIA: Interactive Evaluation for Social Intelligence in Language Agentscited by 367
  3. 2019On the Utility of Learning about Humans for Human-AI Coordinationcited by 349
  4. 2024Navigation World Modelscited by 272
  5. 2023Building Cooperative Embodied Agents Modularly with Large Language Modelscited by 237
  6. 2023OccWorld: Learning a 3D Occupancy World Model for Autonomous Drivingcited by 233
  7. 2023Large Language Models as Commonsense Knowledge for Large-Scale Task Planningcited by 231
  8. 2024ManiSkill3: GPU Parallelized Robotics Simulation and Rendering for Generalizable Embodied AIcited by 229
  9. 2023RoCo: Dialectic Multi-Robot Collaboration with Large Language Modelscited by 172
  10. 2025Context as Memory: Scene-Consistent Interactive Long Video Generation with Memory Retrievalcited by 163
  11. 2023Look Before You Leap: Unveiling the Power of GPT-4V in Robotic Vision-Language Planningcited by 161
  12. 2023SMART-LLM: Smart Multi-Agent Robot Task Planning using Large Language Modelscited by 153
  13. 2020Watch-And-Help: A Challenge for Social Perception and Human-AI Collaborationcited by 133
  14. 2024Math-LLaVA: Bootstrapping Mathematical Reasoning for Multimodal Large Language Modelscited by 133
  15. 2025GameFactory: Creating New Games with Generative Interactive Videoscited by 130
  16. 2024MAVIS: Mathematical Visual Instruction Tuning with an Automatic Data Enginecited by 113
  17. 2024Embodied LLM Agents Learn to Cooperate in Organized Teamscited by 82
  18. 2024PARTNR: A Benchmark for Planning and Reasoning in Embodied Multi-agent Taskscited by 82
  19. 2023LLM-Coordination: Evaluating and Analyzing Multi-agent Coordination Abilities in Large Language Modelscited by 68
  20. 2023MP5: A Multi-modal Open-ended Embodied System in Minecraft via Active Perceptioncited by 67
  21. 2024Code-as-Monitor: Constraint-aware Visual Programming for Reactive and Proactive Robotic Failure Detectioncited by 63
  22. 2024MineDreamer: Learning to Follow Instructions via Chain-of-Imagination for Simulated-World Controlcited by 53
  23. 2020A Cordial Sync: Going Beyond Marginal Policies for Multi-Agent Embodied Taskscited by 52
  24. 2024COMBO: Compositional World Models for Embodied Multi-Agent Cooperationcited by 49
  25. 2023Steve-Eye: Equipping LLM-based Embodied Agents with Visual Perception in Open Worldscited by 45
  26. 2024LLaMAR: Long-Horizon Planning for Multi-Agent Robots in Partially Observable Environmentscited by 32
  27. 2025ReSo: A Reward-driven Self-organizing LLM-based Multi-Agent System for Reasoning Taskscited by 31
  28. 2025RoboFactory: Exploring Embodied Agent Collaboration with Compositional Constraintscited by 31
  29. 2024LiP-LLM: Integrating Linear Programming and dependency graph with Large Language Models for multi-robot task planningcited by 24
  30. 2025A Survey of Interactive Generative Videocited by 23
  31. 2024DART-LLM: Dependency-Aware Multi-Robot Task Decomposition and Execution using Large Language Modelscited by 19
  32. 2025RAD: Retrieval-Augmented Decision-Making of Meta-Actions with Vision-Language Models in Autonomous Drivingcited by 18
  33. 2025LabUtopia: High-Fidelity Simulation and Hierarchical Benchmark for Scientific Embodied Agentscited by 15
  34. 2024Language-Driven Policy Distillation for Cooperative Driving in Multi-Agent Reinforcement Learningcited by 14
  35. 2025Position: Interactive Generative Video as Next-Generation Game Enginecited by 13
  36. 2025NavigateDiff: Visual Predictors are Zero-Shot Navigation Assistantscited by 11
  37. 2021Multi-Agent Embodied Visual Semantic Navigation with Scene Prior Knowledgecited by 10
  38. 2025Multi-Agent LLM Actor-Critic Framework for Social Robot Navigationcited by 8

50 most cited of 51 All 51 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 8; 2026: 10.

18 papers.

Sort:

By area:

  1. 2025The Landscape of Agentic Reinforcement Learning for LLMs: A Surveycited by 205
  2. 2026A Survey of Agentic Reasoning for Large Language Models: Towards Recursively Self-Improving and Collective Agentscited by 50
  3. 2025TIGeR: Tool-Integrated Geometric Reasoning in Vision-Language Models for Roboticscited by 27
  4. 2025Towards Spatial Trace with Reasoning in Vision-Language Models for Roboticscited by 12
  5. 2025Reinforcement Learning for Large Model: A Surveycited by 9
  6. 2025CDP: Towards Robust Autoregressive Visuomotor Policy Learning via Causal Diffusioncited by 8
  7. 2025Dissecting Embodied Abilities in Multimodal Language Models through Skill-level Evaluation and Diagnosiscited by 8
  8. 2026RoboAgent: Chaining Basic Capabilities for Embodied Task Planningcited by 7
  9. 2026From Perception to Action: An Interactive Benchmark for Vision Reasoningcited by 4
  10. 2025LiveSearchBench: An Automatically Constructed Benchmark for Retrieval and Reasoning over Dynamic Knowledgecited by 3
  11. 2025GauDP: Reinventing Multi-Agent Collaboration through Gaussian-Image Synergy in Diffusion Policiescited by 2
  12. 2026Advances and Innovations in the Multi-Agent Robotic System (MARS) Challengecited by 1
6 more
  1. 2026CoEditor++: Instruction-based Visual Editing via Cognitive Reasoningcited by 1
  2. 2026Ego to World: Collaborative Spatial Reasoning in Embodied Systems via Reinforcement Learningcited by 1
  3. 2026Rethinking Agentic Reinforcement Learning In Large Language Modelscited by 1
  4. 2026GuideFetch: A Task Coordination Framework for Concurrent Navigation and Object Retrieval in Assistive Robot Dogscited by 1
  5. 2026RoboFind: Multi-Agent Personalized Object Search for People Who Are Blind or Have Low Visioncited by 0
  6. 2026RoboTalk: Learning Multi-Robot Communication and Coordination from Multimodal Demonstrationscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025ERA: Transforming VLMs into Embodied Agents via Embodied Prior Learning and Online Reinforcement Learning
  2. 2026Plan-Bench: Can Existing VLMs Go Well Beyond Perception to Reason and Plan in Embodied Worlds?
  3. 2026EmbodiedSkills: A Unified Framework for Orchestrating, Training, and Deploying VLA Agents
  4. 2026ME-VLM: A Unified VLM for Embodied Cognition and Agent Coordination
  5. 2025Robot-R1: Reinforcement Learning for Enhanced Embodied Reasoning in Robotics
  6. 2026D-VLC: Decentralized Vision-Language Collaboration for Heterogeneous Embodied Multi-Robot Systems in Unknown Environments
  7. 2026How Foundational Skills Influence VLM-based Embodied Agents:A Native Perspective
  8. 2025Embodied-R: Collaborative Framework for Activating Embodied Spatial Reasoning in Foundation Models via Reinforcement Learning