# Don't lie to your friends: Learning what you know from collaborative self-play

arXiv: https://arxiv.org/abs/2503.14481 · HTML: https://mlsanity.com/paper/2503.14481/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 50 of its 72 references that are in the map.
- First posted on arXiv 18 Mar 2025.

## Builds on (most cited first, 30 of 50)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Toolformer: Language Models Can Teach Themselves to Use Tools](https://mlsanity.com/paper/2302.04761/) (2023)
- [Gemini 1.5: Unlocking multimodal understanding across millions of tokens of context](https://mlsanity.com/paper/2403.05530/) (2024)
- [Domain-Adversarial Training of Neural Networks](https://mlsanity.com/paper/1505.07818/) (2015)
- [Gemma 2: Improving Open Language Models at a Practical Size](https://mlsanity.com/paper/2408.00118/) (2024)
- [Language Models (Mostly) Know What They Know](https://mlsanity.com/paper/2207.05221/) (2022)
- [Improving Factuality and Reasoning in Language Models through Multiagent Debate](https://mlsanity.com/paper/2305.14325/) (2023)
- [When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memories](https://mlsanity.com/paper/2212.10511/) (2022)
- [Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMs](https://mlsanity.com/paper/2306.13063/) (2023)
- [Aleatoric and Epistemic Uncertainty in Machine Learning: An Introduction to Concepts and Methods](https://mlsanity.com/paper/1910.09457/) (2019)
- [Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedback](https://mlsanity.com/paper/2305.14975/) (2023)
- [Large Language Model based Multi-Agents: A Survey of Progress and Challenges](https://mlsanity.com/paper/2402.01680/) (2024)
- [Discovering Latent Knowledge in Language Models Without Supervision](https://mlsanity.com/paper/2212.03827/) (2022)
- [Teaching Models to Express Their Uncertainty in Words](https://mlsanity.com/paper/2205.14334/) (2022)
- [ChatEval: Towards Better LLM-based Evaluators through Multi-Agent Debate](https://mlsanity.com/paper/2308.07201/) (2023)
- [Learning to Communicate with Deep Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/1605.06676/) (2016)
- [Reinforced Self-Training (ReST) for Language Modeling](https://mlsanity.com/paper/2308.08998/) (2023)
- [AmbigQA: Answering Ambiguous Open-domain Questions](https://mlsanity.com/paper/2004.10645/) (2020)
- [Making Retrieval-Augmented Language Models Robust to Irrelevant Context](https://mlsanity.com/paper/2310.01558/) (2023)
- [Learning Multiagent Communication with Backpropagation](https://mlsanity.com/paper/1605.07736/) (2016)
- [ReST-MCTS*: LLM Self-Training via Process Reward Guided Tree Search](https://mlsanity.com/paper/2406.03816/) (2024)
- [ReConcile: Round-Table Conference Improves Reasoning via Consensus among Diverse LLMs](https://mlsanity.com/paper/2309.13007/) (2023)
- [Debating with More Persuasive LLMs Leads to More Truthful Answers](https://mlsanity.com/paper/2402.06782/) (2024)
- [Exploring Large Language Models for Communication Games: An Empirical Study on Werewolf](https://mlsanity.com/paper/2309.04658/) (2023)
- [Selective Question Answering under Domain Shift](https://mlsanity.com/paper/2006.09462/) (2020)
- [Reducing conversational agents' overconfidence through linguistic calibration](https://mlsanity.com/paper/2012.14983/) (2020)
- [The Hanabi Challenge: A New Frontier for AI Research](https://mlsanity.com/paper/1902.00506/) (2019)
- [TALM: Tool Augmented Language Models](https://mlsanity.com/paper/2205.12255/) (2022)
- [Don't Hallucinate, Abstain: Identifying LLM Knowledge Gaps via Multi-LLM Collaboration](https://mlsanity.com/paper/2402.00367/) (2024)
- [Conformal Language Modeling](https://mlsanity.com/paper/2306.10193/) (2023)

## Built on by (most cited first, 14 of 14)

- [Calibrate-Then-Act: Cost-Aware Exploration in LLM Agents](https://mlsanity.com/paper/2602.16699/) (2026)
- [SPARTA ALIGNMENT: Collectively Aligning Multiple Language Models through Combat](https://mlsanity.com/paper/2506.04721/) (2025)
- [To Call or Not to Call: A Framework to Assess and Optimize LLM Tool Calling](https://mlsanity.com/paper/2605.00737/) (2026)
- [MASH: Modeling Abstention via Selective Help-Seeking](https://mlsanity.com/paper/2510.01152/) (2025)
- [Be My Eyes: Extending Large Language Models to New Modalities Through Multi-Agent Collaboration](https://mlsanity.com/paper/2511.19417/) (2025)
- [Learning Steerable Clarification Policies with Collaborative Self-play](https://mlsanity.com/paper/2512.04068/) (2025)
- [Hallucinations Undermine Trust; Metacognition is a Way Forward](https://mlsanity.com/paper/2605.01428/) (2026)
- [Balancing Faithfulness and Performance in Reasoning via Multi-Listener Soft Execution](https://mlsanity.com/paper/2602.16154/) (2026)
- [Search, Do not Guess: Teaching Small Language Models to Be Effective Search Agents](https://mlsanity.com/paper/2604.04651/) (2026)
- [Representational Similarity and Model Behavior in Multi-Agent Interaction](https://mlsanity.com/paper/2606.07818/) (2026)
- [Pandora's AI Model Routing Box: Efficient Allocation with Costly Value Estimation](https://mlsanity.com/paper/2608.20316/) (2026)
- [Beyond Accuracy: Rethinking Hallucination and Regulatory Response in Generative AI](https://mlsanity.com/paper/2509.13345/) (2025)
- [Interplay: Training Independent Simulators for Reference-Free Conversational Recommendation](https://mlsanity.com/paper/2603.18573/) (2026)
- [Uncertainty as a Planning Signal: Multi-Turn Decision Making for Goal-Oriented Conversation](https://mlsanity.com/paper/2604.03924/) (2026)

## Abstract (arXiv)

To be helpful assistants, AI agents must be aware of their own capabilities and limitations. This includes knowing when to answer from parametric knowledge versus using tools, when to trust tool outputs, and when to abstain or hedge. Such capabilities are hard to teach through supervised fine-tuning because they require constructing examples that reflect the agent's specific capabilities. We therefore propose a radically new approach to teaching agents what they know: collaborative self-play. We construct multi-agent collaborations in which the group is rewarded for collectively arriving at correct answers. The desired meta-knowledge emerges from the incentives built into the structure of the interaction. We focus on small societies of agents that have access to heterogeneous tools (corpus-specific retrieval), and therefore must collaborate to maximize their success while minimizing their effort. Experiments show that group-level rewards for multi-agent communities can induce policies that transfer to improve tool use and selective prediction in settings where individual agents are deployed in isolation.