# TeamCraft: A Benchmark for Multi-Modal Multi-Agent Systems in Minecraft

arXiv: https://arxiv.org/abs/2412.05255 · HTML: https://mlsanity.com/paper/2412.05255/

## Key facts

- Cited by 21 papers in the map (data through 14 Oct 2026).
- Builds on 30 of its 67 references that are in the map.
- First posted on arXiv 6 Dec 2024.

## Builds on (most cited first, 30 of 30)

- [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023)
- [Voyager: An Open-Ended Embodied Agent with Large Language Models](https://mlsanity.com/paper/2305.16291/) (2023)
- [Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environments](https://mlsanity.com/paper/1706.02275/) (2017)
- [ALFWorld: Aligning Text and Embodied Environments for Interactive Learning](https://mlsanity.com/paper/2010.03768/) (2020)
- [The Surprising Effectiveness of PPO in Cooperative, Multi-Agent Games](https://mlsanity.com/paper/2103.01955/) (2021)
- [CLIPort: What and Where Pathways for Robotic Manipulation](https://mlsanity.com/paper/2109.12098/) (2021)
- [MineDojo: Building Open-Ended Embodied Agents with Internet-Scale Knowledge](https://mlsanity.com/paper/2206.08853/) (2022)
- [VIMA: General Robot Manipulation with Multimodal Prompts](https://mlsanity.com/paper/2210.03094/) (2022)
- [Describe, Explain, Plan and Select: Interactive Planning with Large Language Models Enables Open-World Multi-Task Agents](https://mlsanity.com/paper/2302.01560/) (2023)
- [On the Utility of Learning about Humans for Human-AI Coordination](https://mlsanity.com/paper/1910.05789/) (2019)
- [Video PreTraining (VPT): Learning to Act by Watching Unlabeled Online Videos](https://mlsanity.com/paper/2206.11795/) (2022)
- [Transporter Networks: Rearranging the Visual World for Robotic Manipulation](https://mlsanity.com/paper/2010.14406/) (2020)
- [SQA3D: Situated Question Answering in 3D Scenes](https://mlsanity.com/paper/2210.07474/) (2022)
- [Emergence of Grounded Compositional Language in Multi-Agent Populations](https://mlsanity.com/paper/1703.04908/) (2017)
- [Building Cooperative Embodied Agents Modularly with Large Language Models](https://mlsanity.com/paper/2307.02485/) (2023)
- [JARVIS-1: Open-World Multi-task Agents with Memory-Augmented Multimodal Language Models](https://mlsanity.com/paper/2311.05997/) (2023)
- [Watch-And-Help: A Challenge for Social Perception and Human-AI Collaboration](https://mlsanity.com/paper/2010.09890/) (2020)
- [Scalable Evaluation of Multi-Agent Reinforcement Learning with Melting Pot](https://mlsanity.com/paper/2107.06857/) (2021)
- [PARTNR: A Benchmark for Planning and Reasoning in Embodied Multi-agent Tasks](https://mlsanity.com/paper/2411.00081/) (2024)
- [VLMbench: A Compositional Benchmark for Vision-and-Language Manipulation](https://mlsanity.com/paper/2206.08522/) (2022)
- [COMBO: Compositional World Models for Embodied Multi-Agent Cooperation](https://mlsanity.com/paper/2404.10775/) (2024)
- [Evolutionary Population Curriculum for Scaling Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2003.10423/) (2020)
- [HandMeThat: Human-Robot Communication in Physical and Social Environments](https://mlsanity.com/paper/2310.03779/) (2023)
- [ROCKET-1: Mastering Open-World Interaction with Visual-Temporal Context Prompting](https://mlsanity.com/paper/2410.17856/) (2024)
- [Alexa Arena: A User-Centric Interactive Platform for Embodied AI](https://mlsanity.com/paper/2303.01586/) (2023)
- [MineLand: Simulating Large-Scale Multi-Agent Interactions with Limited Multimodal Senses and Physical Needs](https://mlsanity.com/paper/2403.19267/) (2024)
- [The Neural MMO Platform for Massively Multiagent Research](https://mlsanity.com/paper/2110.07594/) (2021)
- [The Multi-Agent Reinforcement Learning in Malm"O (MARL"O) Competition](https://mlsanity.com/paper/1901.08129/) (2019)
- [Inverse Attention Agents for Multi-Agent Systems](https://mlsanity.com/paper/2410.21794/) (2024)
- [SocialGFs: Learning Social Gradient Fields for Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2405.01839/) (2024)

## Built on by (most cited first, 21 of 21)

- [Why Do Multi-Agent LLM Systems Fail?](https://mlsanity.com/paper/2503.13657/) (2025)
- [From LLM Reasoning to Autonomous AI Agents: A Comprehensive Review](https://mlsanity.com/paper/2504.19678/) (2025)
- [Perception, Reason, Think, and Plan: A Survey on Large Multimodal Reasoning Models](https://mlsanity.com/paper/2505.04921/) (2025)
- [A Survey of Agentic Reasoning for Large Language Models: Towards Recursively Self-Improving and Collective Agents](https://mlsanity.com/paper/2601.12538/) (2026)
- [MultiWorld: Scalable Multi-Agent Multi-View Video World Models](https://mlsanity.com/paper/2604.18564/) (2026)
- [SimWorld: An Open-ended Realistic Simulator for Autonomous Agents in Physical and Social Worlds](https://mlsanity.com/paper/2512.01078/) (2025)
- [DeliveryBench: Can Agents Earn Profit in Real World?](https://mlsanity.com/paper/2512.19234/) (2025)
- [Seeing Together: Multi-Robot Cooperative Egocentric Spatial Reasoning with Multimodal Large Language Models](https://mlsanity.com/paper/2605.18431/) (2026)
- [MineAnyBuild: Benchmarking Spatial Planning for Open-world AI Agents](https://mlsanity.com/paper/2505.20148/) (2025)
- [GameplayQA: A Benchmarking Framework for Decision-Dense POV-Synced Multi-Video Understanding of 3D Virtual Agents](https://mlsanity.com/paper/2603.24329/) (2026)
- [Towards Generalist Game Players: An Investigation of Foundation Models in the Game Multiverse](https://mlsanity.com/paper/2605.09965/) (2026)
- [MECoBench: A Systematic Study of Multimodal Agent Collaboration in Embodied Environments](https://mlsanity.com/paper/2606.31966/) (2026)
- [Can Current Agents Close the Discovery-to-Application Gap? A Case Study in Minecraft](https://mlsanity.com/paper/2604.24697/) (2026)
- [Multi-agent Framework for Time-Sensitive Complementary Collaboration in Minecraft](https://mlsanity.com/paper/2606.15684/) (2026)
- [Dream-Cubed: Controllable Generative Modeling in Minecraft by Training on Billions of Cubes](https://mlsanity.com/paper/2604.22847/) (2026)
- [RTSGameBench: An RTS Benchmark for Strategic Reasoning by Vision-Language Models](https://mlsanity.com/paper/2606.18950/) (2026)
- [GenCoord: Executable Commitments for Embodied LLM Agents under Private Information](https://mlsanity.com/paper/2608.22055/) (2026)
- [CoCoBench: A Cooperative Coordination Benchmark for Embodied Multi-Agent Task Planning](https://mlsanity.com/paper/2608.28266/) (2026)
- [Lies We Can See: Joint Verbal and Non-Verbal Deception by VLM Agents in Embodied Social Interactions](https://mlsanity.com/paper/2608.30428/) (2026)
- [RoboTalk: Learning Multi-Robot Communication and Coordination from Multimodal Demonstrations](https://mlsanity.com/paper/2609.23997/) (2026)
- [MASBench: Benchmarking LLM-based Multi-Agent Collaboration under Partial Observability](https://mlsanity.com/paper/2610.04672/) (2026)

## Abstract (arXiv)

Collaboration is a cornerstone of society. In the real world, human teammates make use of multi-sensory data to tackle challenging tasks in ever-changing environments. It is essential for embodied agents collaborating in visually-rich environments replete with dynamic interactions to understand multi-modal observations and task specifications. To evaluate the performance of generalizable multi-modal collaborative agents, we present TeamCraft, a multi-modal multi-agent benchmark built on top of the open-world video game Minecraft. The benchmark features 55,000 task variants specified by multi-modal prompts, procedurally-generated expert demonstrations for imitation learning, and carefully designed protocols to evaluate model generalization capabilities. We also perform extensive analyses to better understand the limitations and strengths of existing approaches. Our results indicate that existing models continue to face significant challenges in generalizing to novel goals, scenes, and unseen numbers of agents. These findings underscore the need for further research in this area. The TeamCraft platform and dataset are publicly available at https://github.com/teamcraft-bench/teamcraft.