# Training LLMs for Multi-Agent Collaboration with Verifiable Information Rewards

arXiv: https://arxiv.org/abs/2610.17893 · HTML: https://mlsanity.com/paper/2610.17893/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Reinforcement learning for LLM agents, one of 16 directions in LLM agents and tool use.
- Builds on 19 of its 28 references that are in the map.
- First posted on arXiv 13 Oct 2026.

## Placed in

- LLM agents and tool use / [Reinforcement learning for LLM agents](https://mlsanity.com/agents/reinforcement-learning-for-llm-agents/)

## Builds on (most cited first, 19 of 19)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering](https://mlsanity.com/paper/1809.09600/) (2018)
- [Why Do Multi-Agent LLM Systems Fail?](https://mlsanity.com/paper/2503.13657/) (2025)
- [8-bit Optimizers via Block-wise Quantization](https://mlsanity.com/paper/2110.02861/) (2021)
- [RAGEN: Understanding Self-Evolution in LLM Agents via Multi-Turn Reinforcement Learning](https://mlsanity.com/paper/2504.20073/) (2025)
- [SmolLM2: When Smol Goes Big -- Data-Centric Training of a Small Language Model](https://mlsanity.com/paper/2502.02737/) (2025)
- [Mastering Complex Control in MOBA Games with Deep Reinforcement Learning](https://mlsanity.com/paper/1912.09729/) (2019)
- [SPIRAL: Self-Play on Zero-Sum Games Incentivizes Reasoning via Multi-Agent Multi-Turn Reinforcement Learning](https://mlsanity.com/paper/2506.24119/) (2025)
- [MAPoRL: Multi-Agent Post-Co-Training for Collaborative Large Language Models with Reinforcement Learning](https://mlsanity.com/paper/2502.18439/) (2025)
- [MathChat: Converse to Tackle Challenging Math Problems with LLM Agents](https://mlsanity.com/paper/2306.01337/) (2023)
- [Do as We Do, Not as You Think: the Conformity of Large Language Models](https://mlsanity.com/paper/2501.13381/) (2025)
- [Silo-Bench: A Scalable Environment for Evaluating Distributed Coordination in Multi-Agent LLM Systems](https://mlsanity.com/paper/2603.01045/) (2026)
- [Systematic Failures in Collective Reasoning under Distributed Information in Multi-Agent LLMs](https://mlsanity.com/paper/2505.11556/) (2025)
- [Learning Decentralized LLM Collaboration with Multi-Agent Actor Critic](https://mlsanity.com/paper/2601.21972/) (2026)
- [HIPO: Instruction Hierarchy via Constrained Reinforcement Learning](https://mlsanity.com/paper/2603.16152/) (2026)
- [The Collaboration Tax: How Much LLM Multi-Agent Systems Pay to Coordinate](https://mlsanity.com/paper/2608.22152/) (2026)

## Abstract (arXiv)

Effective collaboration among large language model (LLM) agents in multi-agent systems requires sharing relevant evidence, requesting missing information, and incorporating teammates' contributions into decisions. Training these behaviors is challenging because final task outcomes provide limited feedback on the value of individual messages. We introduce CoSIR (Collaboration via Surprisal-based Information Rewards), a post-training approach that uses a simple, verifiable environment to teach collaboration behaviors that generalize to more complex, unseen tasks. We propose a Bayesian evaluator that scores shared information by its reduction in team surprisal, defined as the sum of the negative log probabilities assigned to the correct option given each agent's available evidence. Requests are scored by the potential reduction in the same quantity if the requested information were truthfully provided. Separate rewards encourage agents to use the received information in their private choices and calculations. We post-train Qwen3-4B, Qwen3-8B, and Gemma-3-4B-it through self-play and evaluate transfer to seven unseen natural-language collaboration tasks. CoSIR achieves the highest aggregate score, while its performance in general tasks remains competitive. Behavioral analyses show increased disclosure of private evidence and fewer judged failures to use teammates' input.