mlsanityData to 14 Oct 2026

CRAFT: Grounded Multi-Agent Coordination Under Partial Information

arXiv 2603.25268 ↗First posted 26 Mar 2026cs.CL cs.AIcited by 3

On the map: LLM agents and tool use → General LLM agent frameworks and benchmarks

Abstract (arXiv)

We introduce CRAFT, a multi-agent benchmark for evaluating pragmatic communication in large language models under strict partial information. In this setting, multiple agents with complementary but incomplete views must coordinate through natural language to construct a shared 3D structure that no single agent can fully observe. We formalize this problem as a multi-sender Bounded Pragmatic Speaker problem and provide a diagnostic framework that decomposes failures into spatial grounding, belief modeling and pragmatic communication errors, including a taxonomy of behavioral failure profiles in both frontier and open-weight models. Across a diverse set of models, including 8 open-weight and 7 frontier including reasoning models, we find that stronger reasoning ability does not reliably translate to better coordination: smaller open-weight models often match or outperform frontier systems, and improved individual communication does not guarantee successful collaboration. These results suggest that multi-agent coordination remains a fundamentally unsolved challenge for current language models. Our code can be found at https://github.com/csu-signal/CRAFT

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Placed in General LLM agent frameworks and benchmarks, one of 16 directions in LLM agents and tool use. Builds on 34 of its 91 references that are in the map. First posted on arXiv 26 Mar 2026.

Builds on

34 of 91 references.

Closest ancestors · 5

Same direction, newest first

  1. 2026The Five Ws of Multi-Agent Communication: Who Talks to Whom, When, What, and Why -- A Survey from MARL to Emergent Language and LLMs12 Feb
  2. 2025The Collaboration Gap: Exploration and Benchmarking of Open-World Agentic Cooperation4 Nov
  3. 2025Collaborate, Deliberate, Evaluate: How LLM Alignment Affects Coordinated Multi-Agent Outcomes7 Sep
  4. 2025AgentsNet: Coordination and Collaborative Reasoning in Multi-Agent LLMs11 Jul
  5. 2025Multi-Agent Collaboration Mechanisms: A Survey of LLMs10 Jan

Other references · 29

  1. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  2. 2024Qwen2.5 Technical Reportcited by 4,942
  3. 2024GPT-4o System Cardcited by 4,588
  4. 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,998
  5. 2023Camels in a Changing Climate: Enhancing LM Adaptation with Tulu 2cited by 229
  6. 2025CollabLLM: From Passive Responders to Active Collaboratorscited by 124
  7. 2025ViewSpatial-Bench: Evaluating Multi-perspective Spatial Localization in Vision-Language Modelscited by 118
  8. 2021Off-Belief Learningcited by 75
  9. 2025Artificial Intelligence Index Report 2025cited by 64
  10. 2024Evaluating Language Model Agency through Negotiationscited by 48
  11. 2025SpatialBench: Benchmarking Multimodal Large Language Models for Spatial Cognitioncited by 35
  12. 2025The Traitors: Deception and Trust in Multi-Agent Language Model Simulationscited by 34
17 more
  1. 2023Multi-agent Reinforcement Learning: A Comprehensive Surveycited by 33
  2. 2024Position: Theory of Mind Benchmarks are Broken for Large Language Modelscited by 31
  3. 2023Dissociating language and thought in large language modelscited by 29
  4. 2024Building Better AI Agents: A Provocation on the Utilisation of Persona in LLM-based Conversational Agentscited by 26
  5. 2025Emergent Coordination in Multi-Agent Language Modelscited by 23
  6. 2025Spatial Reasoning in Multimodal Large Language Models: A Survey of Tasks, Benchmarks and Methodscited by 23
  7. 2018Planning, Inference and Pragmatics in Sequential Language Gamescited by 12
  8. 2024Tag Map: A Text-Based Map for Spatial Reasoning and Navigation with Large Language Modelscited by 12
  9. 2023Language Models are Bounded Pragmatic Speakers: Understanding RLHF from a Bayesian Cognitive Modeling Perspectivecited by 10
  10. 2025From Text to Space: Mapping Abstract Spatial Models in LLMs during a Grid-World Navigation Taskcited by 10
  11. 2024Pragmatic Competence Evaluation of Large Language Models for the Korean Languagecited by 8
  12. 2025LLM-Hanabi: Evaluating Multi-Agent Gameplays with Theory-of-Mind and Rationale Inference in Imperfect Information Collaboration Gamecited by 8
  13. 2026MT-PingEval: Evaluating Multi-Turn Collaboration with Private Information Gamescited by 8
  14. 2025Spatio-Temporal LLM: Reasoning about Environments and Actionscited by 6
  15. 2024Fully Decentralized Cooperative Multi-Agent Reinforcement Learning: A Surveycited by 5
  16. 2026Owen-Shapley Policy Optimization: A Principled RL Algorithm for Generative Search LLMscited by 1
  17. 2026Distributed Partial Information Puzzles: Examining Common Ground Construction Under Epistemic Asymmetrycited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026CalBench: Evaluating Coordination-Privacy Trade-offs in Multi-Agent LLMscited by 7
  2. 2026Flout at Your Own Risk: LLMs Struggle with Pragmatic Cooperativity Under Epistemic Asymmetrycited by 0
  3. 2026Mind the Gap: Theory-of-Mind-Grounded Friction for Epistemic Alignmentcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024COMMA: A Communicative Multimodal Multi-Agent Benchmark
  2. 2025Systematic Failures in Collective Reasoning under Distributed Information in Multi-Agent LLMs
  3. 2026Silo-Bench: A Scalable Environment for Evaluating Distributed Coordination in Multi-Agent LLM Systems
  4. 2026Can Small Agents Collaborate to Beat a Single Large Language Model?
  5. 2025AsymPuzl: An Asymmetric Puzzle for multi-agent cooperation
  6. 2024Improving Multi-Agent Debate with Sparse Communication Topology
  7. 2025Benefits and Limitations of Communication in Multi-Agent Reasoning
  8. 2025Reasoning-Aware Prompt Orchestration: A Foundation Model for Multi-Agent Language Model Coordination