# Training Language Models for Social Deduction with Multi-Agent Reinforcement Learning

arXiv: https://arxiv.org/abs/2502.06060 · HTML: https://mlsanity.com/paper/2502.06060/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Placed in General LLM agent frameworks and benchmarks, one of 16 directions in LLM agents and tool use.
- Builds on 15 of its 37 references that are in the map.
- First posted on arXiv 9 Feb 2025.

## Placed in

- LLM agents and tool use / [General LLM agent frameworks and benchmarks](https://mlsanity.com/agents/general-llm-agent-frameworks-and-benchmarks/)

## Builds on (most cited first, 15 of 15)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [On the Utility of Learning about Humans for Human-AI Coordination](https://mlsanity.com/paper/1910.05789/) (2019)
- [Text2Motion: From Natural Language Instructions to Feasible Plans](https://mlsanity.com/paper/2303.12153/) (2023)
- ["Other-Play" for Zero-Shot Coordination](https://mlsanity.com/paper/2003.02979/) (2020)
- [Large Language Models as General Pattern Machines](https://mlsanity.com/paper/2307.04721/) (2023)
- [Multi-Agent Cooperation and the Emergence of (Natural) Language](https://mlsanity.com/paper/1612.07182/) (2016)
- [Emergence of Language with Multi-agent Games: Learning to Communicate with Sequences of Symbols](https://mlsanity.com/paper/1705.11192/) (2017)
- [GPT-4 Doesn't Know It's Wrong: An Analysis of Iterative Prompting for Reasoning Problems](https://mlsanity.com/paper/2310.12397/) (2023)
- [Continual adaptation for efficient machine communication](https://mlsanity.com/paper/1911.09896/) (2019)
- [Sample-Efficient Reinforcement Learning of Partially Observable Markov Games](https://mlsanity.com/paper/2206.01315/) (2022)
- [Learning to communicate about shared procedural abstractions](https://mlsanity.com/paper/2107.00077/) (2021)
- [How Well Can a Long Sequence Model Model Long Sequences? Comparing Architechtural Inductive Biases on Long-Context Abilities](https://mlsanity.com/paper/2407.08112/) (2024)
- [Hidden Agenda: a Social Deduction Game with Diverse Learned Equilibria](https://mlsanity.com/paper/2201.01816/) (2022)
- [Hidden-Role Games: Equilibrium Concepts and Computation](https://mlsanity.com/paper/2308.16017/) (2023)
- [Optimizing Robotic Manipulation with Decision-RWKV: A Recurrent Sequence Modeling Approach for Lifelong Learning](https://mlsanity.com/paper/2407.16306/) (2024)

## Built on by (most cited first, 16 of 16)

- [SPIRAL: Self-Play on Zero-Sum Games Incentivizes Reasoning via Multi-Agent Multi-Turn Reinforcement Learning](https://mlsanity.com/paper/2506.24119/) (2025)
- [SPICE: Self-Play In Corpus Environments Improves Reasoning](https://mlsanity.com/paper/2510.24684/) (2025)
- [LLM Collaboration With Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2508.04652/) (2025)
- [Chasing Moving Targets with Online Self-Play Reinforcement Learning for Safer Language Models](https://mlsanity.com/paper/2506.07468/) (2025)
- [Among Us: A Sandbox for Measuring and Detecting Agentic Deception](https://mlsanity.com/paper/2504.04072/) (2025)
- [MetaMind: Modeling Human Social Thoughts with Metacognitive Multi-Agent Systems](https://mlsanity.com/paper/2505.18943/) (2025)
- [SPADE: Self-Play in Adaptive Synthetic Executable Environments](https://mlsanity.com/paper/2608.19197/) (2026)
- [Bayesian Social Deduction with Graph-Informed Language Models](https://mlsanity.com/paper/2506.17788/) (2025)
- [Learning Decentralized LLM Collaboration with Multi-Agent Actor Critic](https://mlsanity.com/paper/2601.21972/) (2026)
- [AI Agent Behavioral Science](https://mlsanity.com/paper/2506.06366/) (2025)
- [QUACK: Questioning, Understanding, and Auditing Communicated Knowledge in Multimodal Social Deduction Agents](https://mlsanity.com/paper/2605.27068/) (2026)
- [Shall We Play a Game? Language Models for Open-ended Wargames](https://mlsanity.com/paper/2509.17192/) (2025)
- [Evaluating Large Language Models in a Complex Hidden Role Game](https://mlsanity.com/paper/2605.22826/) (2026)
- [From RLVR to RLSVR: Task Transformation Induces Self-Verifiable Rewards for Open-Ended LLM Self-Improvement](https://mlsanity.com/paper/2607.23802/) (2026)
- [Revac: A Social Deduction Reasoning Agent](https://mlsanity.com/paper/2604.19523/) (2026)
- [Synthetic Linguistic Agency: How an Embodied Mortal Agent Learns Linguistic Affordances through Consequential Social Experience](https://mlsanity.com/paper/2608.27843/) (2026)

## Abstract (arXiv)

Communicating in natural language is a powerful tool in multi-agent settings, as it enables independent agents to share information in partially observable settings and allows zero-shot coordination with humans. However, most prior works are limited as they either rely on training with large amounts of human demonstrations or lack the ability to generate natural and useful communication strategies. In this work, we train language models to have productive discussions about their environment in natural language without any human demonstrations. We decompose the communication problem into listening and speaking. Our key idea is to leverage the agent's goal to predict useful information about the world as a dense reward signal that guides communication. Specifically, we improve a model's listening skills by training them to predict information about the environment based on discussions, and we simultaneously improve a model's speaking skills with multi-agent reinforcement learning by rewarding messages based on their influence on other agents. To investigate the role and necessity of communication in complex social settings, we study an embodied social deduction game based on Among Us, where the key question to answer is the identity of an adversarial imposter. We analyze emergent behaviors due to our technique, such as accusing suspects and providing evidence, and find that it enables strong discussions, doubling the win rates compared to standard RL. We release our code and models at https://socialdeductionllm.github.io/