# CRAFT: Grounded Multi-Agent Coordination Under Partial Information

arXiv: https://arxiv.org/abs/2603.25268 · HTML: https://mlsanity.com/paper/2603.25268/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Placed in General LLM agent frameworks and benchmarks, one of 16 directions in LLM agents and tool use.
- Builds on 34 of its 91 references that are in the map.
- First posted on arXiv 26 Mar 2026.

## Placed in

- LLM agents and tool use / [General LLM agent frameworks and benchmarks](https://mlsanity.com/agents/general-llm-agent-frameworks-and-benchmarks/)

## Builds on (most cited first, 30 of 34)

- [DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Models](https://mlsanity.com/paper/2402.03300/) (2024)
- [Qwen2.5 Technical Report](https://mlsanity.com/paper/2412.15115/) (2024)
- [GPT-4o System Card](https://mlsanity.com/paper/2410.21276/) (2024)
- [Gemma 2: Improving Open Language Models at a Practical Size](https://mlsanity.com/paper/2408.00118/) (2024)
- [Multi-Agent Collaboration Mechanisms: A Survey of LLMs](https://mlsanity.com/paper/2501.06322/) (2025)
- [Camels in a Changing Climate: Enhancing LM Adaptation with Tulu 2](https://mlsanity.com/paper/2311.10702/) (2023)
- [CollabLLM: From Passive Responders to Active Collaborators](https://mlsanity.com/paper/2502.00640/) (2025)
- [ViewSpatial-Bench: Evaluating Multi-perspective Spatial Localization in Vision-Language Models](https://mlsanity.com/paper/2505.21500/) (2025)
- [Off-Belief Learning](https://mlsanity.com/paper/2103.04000/) (2021)
- [Artificial Intelligence Index Report 2025](https://mlsanity.com/paper/2504.07139/) (2025)
- [Evaluating Language Model Agency through Negotiations](https://mlsanity.com/paper/2401.04536/) (2024)
- [SpatialBench: Benchmarking Multimodal Large Language Models for Spatial Cognition](https://mlsanity.com/paper/2511.21471/) (2025)
- [The Traitors: Deception and Trust in Multi-Agent Language Model Simulations](https://mlsanity.com/paper/2505.12923/) (2025)
- [Multi-agent Reinforcement Learning: A Comprehensive Survey](https://mlsanity.com/paper/2312.10256/) (2023)
- [Position: Theory of Mind Benchmarks are Broken for Large Language Models](https://mlsanity.com/paper/2412.19726/) (2024)
- [AgentsNet: Coordination and Collaborative Reasoning in Multi-Agent LLMs](https://mlsanity.com/paper/2507.08616/) (2025)
- [Dissociating language and thought in large language models](https://mlsanity.com/paper/2301.06627/) (2023)
- [Building Better AI Agents: A Provocation on the Utilisation of Persona in LLM-based Conversational Agents](https://mlsanity.com/paper/2407.11977/) (2024)
- [Emergent Coordination in Multi-Agent Language Models](https://mlsanity.com/paper/2510.05174/) (2025)
- [Spatial Reasoning in Multimodal Large Language Models: A Survey of Tasks, Benchmarks and Methods](https://mlsanity.com/paper/2511.15722/) (2025)
- [The Collaboration Gap: Exploration and Benchmarking of Open-World Agentic Cooperation](https://mlsanity.com/paper/2511.02687/) (2025)
- [Planning, Inference and Pragmatics in Sequential Language Games](https://mlsanity.com/paper/1805.11774/) (2018)
- [Tag Map: A Text-Based Map for Spatial Reasoning and Navigation with Large Language Models](https://mlsanity.com/paper/2409.15451/) (2024)
- [Language Models are Bounded Pragmatic Speakers: Understanding RLHF from a Bayesian Cognitive Modeling Perspective](https://mlsanity.com/paper/2305.17760/) (2023)
- [From Text to Space: Mapping Abstract Spatial Models in LLMs during a Grid-World Navigation Task](https://mlsanity.com/paper/2502.16690/) (2025)
- [The Five Ws of Multi-Agent Communication: Who Talks to Whom, When, What, and Why -- A Survey from MARL to Emergent Language and LLMs](https://mlsanity.com/paper/2602.11583/) (2026)
- [Pragmatic Competence Evaluation of Large Language Models for the Korean Language](https://mlsanity.com/paper/2403.12675/) (2024)
- [LLM-Hanabi: Evaluating Multi-Agent Gameplays with Theory-of-Mind and Rationale Inference in Imperfect Information Collaboration Game](https://mlsanity.com/paper/2510.04980/) (2025)
- [MT-PingEval: Evaluating Multi-Turn Collaboration with Private Information Games](https://mlsanity.com/paper/2602.24188/) (2026)
- [Collaborate, Deliberate, Evaluate: How LLM Alignment Affects Coordinated Multi-Agent Outcomes](https://mlsanity.com/paper/2509.05882/) (2025)

## Built on by (most cited first, 3 of 3)

- [CalBench: Evaluating Coordination-Privacy Trade-offs in Multi-Agent LLMs](https://mlsanity.com/paper/2605.09823/) (2026)
- [Flout at Your Own Risk: LLMs Struggle with Pragmatic Cooperativity Under Epistemic Asymmetry](https://mlsanity.com/paper/2607.11053/) (2026)
- [Mind the Gap: Theory-of-Mind-Grounded Friction for Epistemic Alignment](https://mlsanity.com/paper/2608.30719/) (2026)

## Abstract (arXiv)

We introduce CRAFT, a multi-agent benchmark for evaluating pragmatic communication in large language models under strict partial information. In this setting, multiple agents with complementary but incomplete views must coordinate through natural language to construct a shared 3D structure that no single agent can fully observe. We formalize this problem as a multi-sender Bounded Pragmatic Speaker problem and provide a diagnostic framework that decomposes failures into spatial grounding, belief modeling and pragmatic communication errors, including a taxonomy of behavioral failure profiles in both frontier and open-weight models. Across a diverse set of models, including 8 open-weight and 7 frontier including reasoning models, we find that stronger reasoning ability does not reliably translate to better coordination: smaller open-weight models often match or outperform frontier systems, and improved individual communication does not guarantee successful collaboration. These results suggest that multi-agent coordination remains a fundamentally unsolved challenge for current language models. Our code can be found at https://github.com/csu-signal/CRAFT