mlsanityData to 14 Oct 2026

Sanzaru: Benchmarking Protocol Formation under Epistemic & Communication Asymmetry

arXiv 2610.15163 ↗First posted 10 Oct 2026cs.CL cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

A language model's ability to solve a task individually does not ensure that a team can solve it under unequal access to information and communication. SANZARU examines this gap through cooperative bomb-defusal puzzles adapted from BOMBANANA! (Lefto Studio, 2026). Task information and action authority are divided among three agents: one cannot hear, another cannot speak, and the third cannot see. The benchmark contains 120 puzzle instances across seven families, each targeting a distinct coordination ability. Across 1512 episodes, 7 models achieve an overall success rate of 49%, ranging from 20% to 86% under the evaluated configurations. Teams develop or adopt diverse communication protocols, from simple gesture codes to Morse and encodings of entire manuals. Yet trace analysis attributes about 60% of 771 team losses to communication, and agents attempt to use channels unavailable to their teammates despite explicit role instructions. Together, task outcomes and interaction traces enable evaluation of both collaborative performance and perspective taking throughout cooperation. The benchmark is available at https://github.com/markli404/sanzaru-benchmark.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 18 of its 27 references that are in the map. First posted on arXiv 10 Oct 2026.

Builds on

18 of 27 references.

References · 18

  1. 2023PersonaLLM: Investigating the Ability of Large Language Models to Express Personality Traitscited by 234
  2. 2019The Hanabi Challenge: A New Frontier for AI Researchcited by 197
  3. 2016Multi-Agent Cooperation and the Emergence of (Natural) Languagecited by 185
  4. 2023FANToM: A Benchmark for Stress-testing Machine Theory of Mind in Interactionscited by 171
  5. 2024Emergent social conventions and collective bias in LLM populationscited by 123
  6. 2024PersonaGym: Evaluating Persona Agents and LLMscited by 97
  7. 2019On the Pitfalls of Measuring Emergent Communicationcited by 71
  8. 2023LLM-Coordination: Evaluating and Analyzing Multi-agent Coordination Abilities in Large Language Modelscited by 68
  9. 2024SimpleToM: Exposing the Gap between Explicit ToM Inference and Implicit ToM Application in LLMscited by 57
  10. 2025Collab-Overcooked: Benchmarking and Evaluating Large Language Models as Collaborative Agentscited by 41
  11. 2024COMMA: A Communicative Multimodal Multi-Agent Benchmarkcited by 14
  12. 2025Benchmarking LLMs' Swarm intelligencecited by 10
6 more
  1. 2026PAC-BENCH: Evaluating Multi-Agent Collaboration under Privacy Constraintscited by 7
  2. 2024Using Game Play to Investigate Multimodal and Conversational Grounding in Large Multimodal Modelscited by 3
  3. 2025Communication and Verification in LLM Agents towards Collaboration under Information Asymmetrycited by 3
  4. 2026SOTOPIA-TOM: Evaluating Privacy and Information Management in Multi-Agent Interaction with Theory of Mindcited by 3
  5. 2026LLM Agents for Deliberative Collaboration: A Study on Joint Decision Making Under Partial Observabilitycited by 2
  6. 2026GPTNT: Benchmarking Real-Time Collaboration Between Multimodal Agents on Keep Talking And Nobody Explodescited by 1

Built on by

Not cited yet.