mlsanityData to 14 Oct 2026

COMBO: Compositional World Models for Embodied Multi-Agent Cooperation

arXiv 2404.10775 ↗First posted 16 Apr 2024cs.CV cs.AI cs.MAcited by 49

On the map: World models → World models for language agents

also in 1 more
Abstract (arXiv)

In this paper, we investigate the problem of embodied multi-agent cooperation, where decentralized agents must cooperate given only egocentric views of the world. To effectively plan in this setting, in contrast to learning world dynamics in a single-agent scenario, we must simulate world dynamics conditioned on an arbitrary number of agents' actions given only partial egocentric visual observations of the world. To address this issue of partial observability, we first train generative models to estimate the overall world state given partial egocentric observations. To enable accurate simulation of multiple sets of actions on this world state, we then propose to learn a compositional world model for multi-agent cooperation by factorizing the naturally composable joint actions of multiple agents and compositionally generating the video conditioned on the world state. By leveraging this compositional world model, in combination with Vision Language Models to infer the actions of other agents, we can use a tree search procedure to integrate these modules and facilitate online cooperative planning. We evaluate our methods on three challenging benchmarks with 2-4 agents. The results show our compositional world model is effective and the framework enables the embodied agents to cooperate efficiently with different agents across various tasks and an arbitrary number of agents, showing the promising future of our proposed methods. More videos can be found at https://umass-embodied-agi.github.io/COMBO/.

Key facts. Cited by 49 papers in the map (data through 14 Oct 2026). Placed in World models for language agents, one of 11 directions in World models. Builds on 78 of its 94 references that are in the map. First posted on arXiv 16 Apr 2024.

Builds on

78 of 94 references.

Closest ancestors · 4

Same direction, newest first

  1. 2023Generative Agents: Interactive Simulacra of Human Behavior7 Apr
  2. 2023Language Is Not All You Need: Aligning Perception with Language Models27 Feb
  3. 2022Inner Monologue: Embodied Reasoning through Planning with Language Models12 Jul
  4. 2020Language Models are Few-Shot Learners28 May

Other references · 74

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  5. 2023Visual Instruction Tuningcited by 7,869
  6. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,058
  7. 2022Classifier-Free Diffusion Guidancecited by 5,847
  8. 2023Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detectioncited by 3,332
  9. 2023RT-2: Vision-Language-Action Models Transfer Web Knowledge to Robotic Controlcited by 3,080
  10. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,592
  11. 2023Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasetscited by 2,581
  12. 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,348
38 more
  1. 2022Video Diffusion Modelscited by 2,305
  2. 2023A Survey on Large Language Model based Autonomous Agentscited by 2,242
  3. 2023Voyager: An Open-Ended Embodied Agent with Large Language Modelscited by 2,191
  4. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,071
  5. 2022Imagen Video: High Definition Video Generation with Diffusion Modelscited by 1,778
  6. 2017Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environmentscited by 1,558
  7. 2023Align your Latents: High-Resolution Video Synthesis with Latent Diffusion Modelscited by 1,476
  8. 2023The Rise and Potential of Large Language Model Based Agents: A Surveycited by 1,439
  9. 2019Habitat: A Platform for Embodied AI Researchcited by 1,191
  10. 2022Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agentscited by 1,035
  11. 2021The Surprising Effectiveness of PPO in Cooperative, Multi-Agent Gamescited by 942
  12. 2017AI2-THOR: An Interactive 3D Environment for Visual AIcited by 869
  13. 2018Recurrent World Models Facilitate Policy Evolutioncited by 785
  14. 2019ALFRED: A Benchmark for Interpreting Grounded Instructions for Everyday Taskscited by 783
  15. 2024Genie: Generative Interactive Environmentscited by 747
  16. 2023Text2Video-Zero: Text-to-Image Diffusion Models are Zero-Shot Video Generatorscited by 670
  17. 2021VideoGPT: Video Generation using VQ-VAE and Transformerscited by 658
  18. 2022Compositional Visual Generation with Composable Diffusion Modelscited by 624
  19. 2019The StarCraft Multi-Agent Challengecited by 597
  20. 2020SAPIEN: A SimulAted Part-based Interactive ENvironmentcited by 595
  21. 2023Structure and Content-Guided Video Synthesis with Diffusion Modelscited by 590
  22. 2023Learning Universal Policies via Text-Guided Video Generationcited by 522
  23. 2022Phenaki: Variable Length Video Generation From Open Domain Textual Descriptioncited by 484
  24. 2018Gibson Env: Real-World Perception for Embodied Agentscited by 481
  25. 2023Learning Interactive Real-World Simulatorscited by 458
  26. 2023Cognitive Architectures for Language Agentscited by 429
  27. 2022VIMA: General Robot Manipulation with Multimodal Promptscited by 407
  28. 2016The CMA Evolution Strategy: A Tutorialcited by 377
  29. 2019On the Utility of Learning about Humans for Human-AI Coordinationcited by 349
  30. 2019Emergent Tool Use From Multi-Agent Autocurriculacited by 313
  31. 2023Zero-Shot Robotic Manipulation with Pretrained Image-Editing Diffusion Modelscited by 269
  32. 2018Human-level performance in first-person multiplayer games with population-based deep reinforcement learningcited by 264
  33. 2016Unsupervised Learning for Physical Interaction through Video Predictioncited by 260
  34. 2020ThreeDWorld: A Platform for Interactive Multi-Modal Physical Simulationcited by 250
  35. 2023Building Cooperative Embodied Agents Modularly with Large Language Modelscited by 237
  36. 2013The Complexity of Decentralized Control of Markov Decision Processescited by 231
  37. 2023Habitat 3.0: A Co-Habitat for Humans, Avatars and Robotscited by 228
  38. 2022Pre-Trained Language Models for Interactive Decision-Makingcited by 221

50 most cited of 74 All 74 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 5; 2025: 23; 2026: 21.

49 papers.

Sort:

By area:

  1. 2025TesserAct: Learning 4D Embodied World Modelscited by 94
  2. 2025A Comprehensive Survey on World Models for Embodied AIcited by 68
  3. 2026A Survey of Agentic Reasoning for Large Language Models: Towards Recursively Self-Improving and Collective Agentscited by 50
  4. 2025World-in-World: World Models in a Closed-Loop Worldcited by 47
  5. 2025VLM4D: Towards Spatiotemporal Awareness in Vision Language Modelscited by 46
  6. 2025Multi-agent Embodied AI: Advances and Future Directionscited by 36
  7. 2025Embodied AI: From LLMs to World Modelscited by 24
  8. 2025Heterogeneous Swarms: Jointly Optimizing Model Roles and Weights for Multi-LLM Systemscited by 23
  9. 2025A Survey of Interactive Generative Videocited by 23
  10. 2026MultiWorld: Scalable Multi-Agent Multi-View Video World Modelscited by 23
  11. 2024TeamCraft: A Benchmark for Multi-Modal Multi-Agent Systems in Minecraftcited by 21
  12. 2025MiLA: Multi-view Intensive-fidelity Long-term Video Generation World Model for Autonomous Drivingcited by 18
37 more
  1. 2025VIKI-R: Coordinating Embodied Multi-Agent Cooperation via Reinforcement Learningcited by 18
  2. 2025ORV: 4D Occupancy-centric Robot Video Generationcited by 17
  3. 2026MVISTA-4D: View-Consistent 4D World Model with Test-Time Action Inference for Robotic Manipulationcited by 17
  4. 2024Kubrick: Multimodal Agent Collaborations for Synthetic Video Generationcited by 16
  5. 2025A Step Toward World Models: A Survey on Robotic Manipulationcited by 15
  6. 2025Simulating the Visual World with Artificial Intelligence: A Roadmapcited by 15
  7. 2024Constrained Human-AI Cooperation: An Inclusive Embodied Social Intelligence Challengecited by 13
  8. 2025Generative Multi-Agent Collaboration in Embodied AI: A Systematic Reviewcited by 13
  9. 2024Infer Human's Intentions Before Following Natural Language Instructionscited by 10
  10. 2025Benchmarking LLMs' Swarm intelligencecited by 10
  11. 2026Towards Interactive Video World Modeling: Frontiers, Challenges, Benchmarks, and Future Trendscited by 8
  12. 2026CausalSpatial: A Benchmark for Object-Centric Causal Spatial Reasoningcited by 7
  13. 2024MindForge: Empowering Embodied Agents with Theory of Mind for Lifelong Cultural Learningcited by 5
  14. 2025MindPower: Enabling Theory-of-Mind Reasoning in VLM-based Embodied Agentscited by 5
  15. 2026From Assumptions to Actions: Turning LLM Reasoning into Uncertainty-Aware Planning for Embodied Agentscited by 5
  16. 2026AgenticCache: Cache-Driven Asynchronous Planning for Embodied AI Agentscited by 4
  17. 2025ANNIE: Be Careful of Your Robotscited by 3
  18. 2025Social World Model-Augmented Mechanism Design Policy Learningcited by 3
  19. 2026MetaMind: General and Cognitive World Models in Multi-Agent Systems by Meta-Theory of Mindcited by 3
  20. 2025Ego-centric Learning of Communicative World Models for Autonomous Drivingcited by 2
  21. 2025GauDP: Reinventing Multi-Agent Collaboration through Gaussian-Image Synergy in Diffusion Policiescited by 2
  22. 2026LLawCo: Learning Laws of Cooperation for Modeling Embodied Multi-Agent Behaviorcited by 2
  23. 2025GenAI-based Multi-Agent Reinforcement Learning towards Distributed Agent Intelligence: A Generative-RL Agent Perspectivecited by 1
  24. 2025Bridging the Gap Between Multimodal Foundation Models and World Modelscited by 1
  25. 2025IGen: Scalable Data Generation for Robot Learning from Open-World Imagescited by 1
  26. 2026Ego to World: Collaborative Spatial Reasoning in Embodied Systems via Reinforcement Learningcited by 1
  27. 20263D-Belief: Embodied Belief Inference via Generative 3D World Modelingcited by 1
  28. 2026When Replanning Becomes the Bottleneck: Budgeted Replanning for Embodied Agentscited by 1
  29. 2026IntentVLM: Open-Vocabulary Intention Recognition through Forward-Inverse Modeling with Video-Language Modelscited by 0
  30. 2026Sentinel: Embodied Cooperative Spatial Reasoning and Planningcited by 0
  31. 2026Multi-Agent Embodied Autonomous Driving (MAEAD): From V2X Information Exchange to Shared World Modelscited by 0
  32. 2026Learning Implicit Causal World Models from Multi-Agent Demonstrationscited by 0
  33. 2026MARS-RA: Rank Aggregation for Credit Assignment via Multimodal Comparisons in Embodied Multi-Agent Cooperationcited by 0
  34. 2026CyberWorld: World Models for Sample-Efficient Autonomous Cyber Defensecited by 0
  35. 2026JEPA-TTT: Persistent Test-Time Training of Latent World Models for Planning under Dynamics Shiftscited by 0
  36. 2026Esports World Model Datasetcited by 0
  37. 2026PAWA: Panoramic Action-conditioned World Modeling for Multiple Agentscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Multi-Agent Egocentric World Model with Fine-Grained Embodied Interaction
  2. 2026Embodied Multi-Agent Coordination by Aligning World Models Through Dialogue
  3. 2026When Multi-Robot Systems Meet Agentic AI:Towards Embodied Collective Intelligence
  4. 2026Gamma-World: Generative Multi-Agent World Modeling Beyond Two Players
  5. 2025Decentralized Collective World Model for Emergent Communication and Coordination
  6. 2026MetaWorld: Scaling Multi-Agent Video World Model from Single-view Video Data
  7. 2026CoEnv: Driving Embodied Multi-Agent Collaboration via Compositional Environment
  8. 2024Decentralized Transformers with Centralized Aggregation are Sample-Efficient Multi-Agent World Models