# MT-PingEval: Evaluating Multi-Turn Collaboration with Private Information Games

arXiv: https://arxiv.org/abs/2602.24188 · HTML: https://mlsanity.com/paper/2602.24188/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 48 references that are in the map.
- First posted on arXiv 27 Feb 2026.

## Builds on (most cited first, 24 of 24)

- [Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena](https://mlsanity.com/paper/2306.05685/) (2023)
- [Discovering Language Model Behaviors with Model-Written Evaluations](https://mlsanity.com/paper/2212.09251/) (2022)
- [Why Do Multi-Agent LLM Systems Fail?](https://mlsanity.com/paper/2503.13657/) (2025)
- [LLMs Get Lost In Multi-Turn Conversation](https://mlsanity.com/paper/2505.06120/) (2025)
- [MINT: Evaluating LLMs in Multi-turn Interaction with Tools and Language Feedback](https://mlsanity.com/paper/2309.10691/) (2023)
- [MT-Bench-101: A Fine-Grained Benchmark for Evaluating Large Language Models in Multi-Turn Dialogues](https://mlsanity.com/paper/2402.14762/) (2024)
- [MediQ: Question-Asking LLMs and a Benchmark for Reliable Interactive Clinical Reasoning](https://mlsanity.com/paper/2406.00922/) (2024)
- [ELEPHANT: Measuring and understanding social sycophancy in LLMs](https://mlsanity.com/paper/2505.13995/) (2025)
- [Towards an Automatic Turing Test: Learning to Evaluate Dialogue Responses](https://mlsanity.com/paper/1708.07149/) (2017)
- [MT-Eval: A Multi-Turn Capabilities Evaluation Benchmark for Large Language Models](https://mlsanity.com/paper/2401.16745/) (2024)
- [Learning Symmetric Collaborative Dialogue Agents with Dynamic Knowledge Graph Embeddings](https://mlsanity.com/paper/1704.07130/) (2017)
- [SimulatorArena: Are User Simulators Reliable Proxies for Multi-Turn Evaluation of AI Assistants?](https://mlsanity.com/paper/2510.05444/) (2025)
- [Grounding Gaps in Language Model Generations](https://mlsanity.com/paper/2311.09144/) (2023)
- [Abstract Visual Reasoning with Tangram Shapes](https://mlsanity.com/paper/2211.16492/) (2022)
- [Approximating Interactive Human Evaluation with Self-Play for Open-Domain Dialog Systems](https://mlsanity.com/paper/1906.09308/) (2019)
- [BotChat: Evaluating LLMs' Capabilities of Having Multi-Turn Dialogues](https://mlsanity.com/paper/2310.13650/) (2023)
- [Decision-Oriented Dialogue for Human-AI Collaboration](https://mlsanity.com/paper/2305.20076/) (2023)
- [COVR: A test-bed for Visually Grounded Compositional Generalization with real images](https://mlsanity.com/paper/2109.10613/) (2021)
- [CooperBench: Why Coding Agents Cannot be Your Teammates Yet](https://mlsanity.com/paper/2601.13295/) (2026)
- [Teaching Models to Balance Resisting and Accepting Persuasion](https://mlsanity.com/paper/2410.14596/) (2024)
- [MD3: The Multi-Dialect Dataset of Dialogues](https://mlsanity.com/paper/2305.11355/) (2023)
- [The Collaboration Gap: Exploration and Benchmarking of Open-World Agentic Cooperation](https://mlsanity.com/paper/2511.02687/) (2025)
- [Contextual Dialogue Act Classification for Open-Domain Conversational Agents](https://mlsanity.com/paper/2005.13804/) (2020)
- [Interaction Dynamics as a Reward Signal for LLMs](https://mlsanity.com/paper/2511.08394/) (2025)

## Built on by (most cited first, 8 of 8)

- [CRAFT: Grounded Multi-Agent Coordination Under Partial Information](https://mlsanity.com/paper/2603.25268/) (2026)
- [Sycophancy Undermines Epistemic Vigilance in Cooperative Vision-Language Tasks](https://mlsanity.com/paper/2607.29585/) (2026)
- [The Collaboration Tax: How Much LLM Multi-Agent Systems Pay to Coordinate](https://mlsanity.com/paper/2608.22152/) (2026)
- [Talk is Cheap, Communication is Hard: Dynamic Grounding Failures and Repair in Multi-Agent Negotiation](https://mlsanity.com/paper/2605.01750/) (2026)
- [RankJudge: A Multi-Turn LLM-as-a-Judge Synthetic Benchmark Generator](https://mlsanity.com/paper/2605.21748/) (2026)
- [Flout at Your Own Risk: LLMs Struggle with Pragmatic Cooperativity Under Epistemic Asymmetry](https://mlsanity.com/paper/2607.11053/) (2026)
- [Which one is banana man? Evaluating vision-language models in multi-turn pragmatic interpretation](https://mlsanity.com/paper/2608.29571/) (2026)
- [Narrowing the Collaboration Gap, Probably](https://mlsanity.com/paper/2610.19215/) (2026)

## Abstract (arXiv)

We present a scalable and verifiable methodology for evaluating language models in multi-turn interactions, using a suite of collaborative games that require effective communication about private information. This enables an interactive scaling analysis, in which a fixed token budget is divided over a variable number of turns. We find that language models often fail to use interactive collaboration to improve over the non-interactive baseline in which one agent summarizes its information and the other agent immediately acts, despite substantial headroom. This suggests that state-of-the-art models still suffer from significant weaknesses in planning and executing multi-turn collaborative conversations. We analyze the linguistic features of these dialogues, assessing the roles of sycophancy, information density, and discourse coherence. While there is no single linguistic explanation for the collaborative weaknesses of contemporary language models, we note that humans achieve comparable task success at superior token efficiency by producing more coherent dialogues. The proactive management of private information is a defining feature of real-world communication, and this work is designed to drive further progress on this capability.