mlsanityData to 7 Oct 2026

Don't lie to your friends: Learning what you know from collaborative self-play

arXiv 2503.14481 ↗First posted 18 Mar 2025cs.LG cs.CLcited by 14

Not on the map. Why?

Abstract (arXiv)

To be helpful assistants, AI agents must be aware of their own capabilities and limitations. This includes knowing when to answer from parametric knowledge versus using tools, when to trust tool outputs, and when to abstain or hedge. Such capabilities are hard to teach through supervised fine-tuning because they require constructing examples that reflect the agent's specific capabilities. We therefore propose a radically new approach to teaching agents what they know: collaborative self-play. We construct multi-agent collaborations in which the group is rewarded for collectively arriving at correct answers. The desired meta-knowledge emerges from the incentives built into the structure of the interaction. We focus on small societies of agents that have access to heterogeneous tools (corpus-specific retrieval), and therefore must collaborate to maximize their success while minimizing their effort. Experiments show that group-level rewards for multi-agent communities can induce policies that transfer to improve tool use and selective prediction in settings where individual agents are deployed in isolation.

Key facts. Cited by 14 papers in the map (data through 7 Oct 2026). Builds on 50 of its 72 references that are in the map. First posted on arXiv 18 Mar 2025.

Builds on

50 of 72 references.

References · 50

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,072
  3. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
  4. 2015Domain-Adversarial Training of Neural Networkscited by 3,023
  5. 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,928
  6. 2022Language Models (Mostly) Know What They Knowcited by 1,783
  7. 2023Improving Factuality and Reasoning in Language Models through Multiagent Debatecited by 1,739
  8. 2022When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memoriescited by 1,103
  9. 2023Can LLMs Express Their Uncertainty? An Empirical Evaluation of Confidence Elicitation in LLMscited by 946
  10. 2019Aleatoric and Epistemic Uncertainty in Machine Learning: An Introduction to Concepts and Methodscited by 903
  11. 2023Just Ask for Calibration: Strategies for Eliciting Calibrated Confidence Scores from Language Models Fine-Tuned with Human Feedbackcited by 788
  12. 2024Large Language Model based Multi-Agents: A Survey of Progress and Challengescited by 752
38 more
  1. 2022Discovering Latent Knowledge in Language Models Without Supervisioncited by 733
  2. 2022Teaching Models to Express Their Uncertainty in Wordscited by 731
  3. 2023ChatEval: Towards Better LLM-based Evaluators through Multi-Agent Debatecited by 724
  4. 2016Learning to Communicate with Deep Multi-Agent Reinforcement Learningcited by 561
  5. 2023Reinforced Self-Training (ReST) for Language Modelingcited by 486
  6. 2020AmbigQA: Answering Ambiguous Open-domain Questionscited by 385
  7. 2023Making Retrieval-Augmented Language Models Robust to Irrelevant Contextcited by 364
  8. 2016Learning Multiagent Communication with Backpropagationcited by 355
  9. 2024ReST-MCTS*: LLM Self-Training via Process Reward Guided Tree Searchcited by 338
  10. 2023ReConcile: Round-Table Conference Improves Reasoning via Consensus among Diverse LLMscited by 297
  11. 2024Debating with More Persuasive LLMs Leads to More Truthful Answerscited by 255
  12. 2023Exploring Large Language Models for Communication Games: An Empirical Study on Werewolfcited by 238
  13. 2020Selective Question Answering under Domain Shiftcited by 228
  14. 2020Reducing conversational agents' overconfidence through linguistic calibrationcited by 209
  15. 2022TALM: Tool Augmented Language Modelscited by 197
  16. 2019The Hanabi Challenge: A New Frontier for AI Researchcited by 194
  17. 2024Don't Hallucinate, Abstain: Identifying LLM Knowledge Gaps via Multi-LLM Collaborationcited by 176
  18. 2022Large Language Models with Controllable Working Memorycited by 169
  19. 2023Conformal Language Modelingcited by 163
  20. 2021Simple Entity-Centric Questions Challenge Dense Retrieverscited by 140
  21. 2024Language Models with Conformal Factuality Guaranteescited by 128
  22. 2024Know Your Limits: A Survey of Abstention in Large Language Modelscited by 116
  23. 2024ClashEval: Quantifying the tug-of-war between an LLM's internal prior and external evidencecited by 109
  24. 2023R-Tuning: Instructing Large Language Models to Say `I Don't Know'cited by 106
  25. 2024Relying on the Unreliable: The Impact of Language Models' Reluctance to Express Uncertaintycited by 106
  26. 2023Mindstorms in Natural Language-Based Societies of Mindcited by 105
  27. 2024Large Language Models Must Be Taught to Know What They Don't Knowcited by 99
  28. 2025Multiagent Finetuning: Self Improvement with Diverse Reasoning Chainscited by 90
  29. 2024SOTOPIA-π: Interactive Learning of Socially Intelligent Language Agentscited by 89
  30. 2024Can Large Language Models Faithfully Express Their Intrinsic Uncertainty in Words?cited by 85
  31. 2024Gecko: Versatile Text Embeddings Distilled from Large Language Modelscited by 81
  32. 2023Alignment for Honestycited by 72
  33. 2024Multi-turn Reinforcement Learning from Preference Human Feedbackcited by 65
  34. 2024Linguistic Calibration of Long-Form Generationscited by 59
  35. 2023Grounding Gaps in Language Model Generationscited by 45
  36. 2024MAGDi: Structured Distillation of Multi-Agent Interaction Graphs Improves Reasoning in Smaller Language Modelscited by 41
  37. 2023Social Learning: Towards Collaborative Learning with Large Language Modelscited by 15
  38. 2024Efficacy of Language Model Self-Play in Non-Zero-Sum Gamescited by 14

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 5; 2026: 9.

14 papers.

Sort:

By area:

  1. 2026Calibrate-Then-Act: Cost-Aware Exploration in LLM Agentscited by 16
  2. 2025SPARTA ALIGNMENT: Collectively Aligning Multiple Language Models through Combatcited by 8
  3. 2026To Call or Not to Call: A Framework to Assess and Optimize LLM Tool Callingcited by 7
  4. 2025Be My Eyes: Extending Large Language Models to New Modalities Through Multi-Agent Collaborationcited by 5
  5. 2025Learning Steerable Clarification Policies with Collaborative Self-playcited by 5
  6. 2026Hallucinations Undermine Trust; Metacognition is a Way Forwardcited by 4
  7. 2025MASH: Modeling Abstention via Selective Help-Seekingcited by 3
  8. 2026Balancing Faithfulness and Performance in Reasoning via Multi-Listener Soft Executioncited by 2
  9. 2026Representational Similarity and Model Behavior in Multi-Agent Interactioncited by 2
  10. 2025Beyond Accuracy: Rethinking Hallucination and Regulatory Response in Generative AIcited by 1
  11. 2026Search, Do not Guess: Teaching Small Language Models to Be Effective Search Agentscited by 1
  12. 2026Pandora's AI Model Routing Box: Efficient Allocation with Costly Value Estimationcited by 1
2 more
  1. 2026Interplay: Training Independent Simulators for Reference-Free Conversational Recommendationcited by 0
  2. 2026Uncertainty as a Planning Signal: Multi-Turn Decision Making for Goal-Oriented Conversationcited by 0