mlsanityData to 14 Oct 2026

Multi-agent KTO: Reinforcing Strategic Interactions of Large Language Model in Language Game

arXiv 2501.14225 ↗First posted 24 Jan 2025cs.CL cs.AI cs.HCcited by 10

On the map: LLM agents and tool use → Reinforcement learning for LLM agents

Abstract (arXiv)

Achieving Artificial General Intelligence (AGI) requires AI agents that can not only make stratigic decisions but also engage in flexible and meaningful communication. Inspired by Wittgenstein's language game theory in Philosophical Investigations, we propose that language agents can learn through in-context interaction rather than traditional multi-stage frameworks that separate decision-making from language expression. Using Werewolf, a social deduction game that tests language understanding, strategic interaction, and adaptability, we develop the Multi-agent Kahneman & Tversky's Optimization (MaKTO). MaKTO engages diverse models in extensive gameplay to generate unpaired desirable and unacceptable responses, then employs KTO to refine the model's decision-making process. In 9-player Werewolf games, MaKTO achieves a 61% average win rate across various models, outperforming GPT-4o and two-stage RL agents by relative improvements of 23.0% and 10.9%, respectively. Notably, MaKTO also demonstrates human-like performance, winning 60% against expert players and showing only 49% detectability in Turing-style blind tests.

Key facts. Cited by 10 papers in the map (data through 14 Oct 2026). Placed in Reinforcement learning for LLM agents, one of 16 directions in LLM agents and tool use. Builds on 36 of its 60 references that are in the map. First posted on arXiv 24 Jan 2025.

Builds on

36 of 60 references.

Closest ancestors · 6

Same direction, newest first

  1. 2024An Implementation of Werewolf Agent That does not Truly Trust LLMs3 Sep
  2. 2024Trial and Error: Exploration-Based Trajectory Optimization for LLM Agents4 Mar
  3. 2024Agent-Pro: Learning to Evolve via Policy-Level Reflection and Optimization27 Feb
  4. 2024Enhance Reasoning for Large Language Models in the Game Werewolf4 Feb
  5. 2023Deciphering Digital Detectives: Understanding LLM Behaviors and Capabilities in Multi-Agent Mystery Games1 Dec
  6. 2023Language Agents with Reinforcement Learning for Strategic Play in the Werewolf Game29 Oct

Other references · 30

  1. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  2. 2023Generative Agents: Interactive Simulacra of Human Behaviorcited by 3,568
  3. 2019Dota 2 with Large Scale Deep Reinforcement Learningcited by 1,103
  4. 2024SimPO: Simple Preference Optimization with a Reference-Free Rewardcited by 963
  5. 2023ExpeL: LLM Agents Are Experiential Learnerscited by 876
  6. 2024ORPO: Monolithic Preference Optimization without Reference Modelcited by 562
  7. 2017Population Based Training of Neural Networkscited by 367
  8. 2023Character-LLM: A Trainable Agent for Role-Playingcited by 339
  9. 2023AgentTuning: Enabling Generalized Agent Abilities for LLMscited by 338
  10. 2024Step-DPO: Step-wise Preference Optimization for Long-chain Reasoning of LLMscited by 273
  11. 2023Exploring Large Language Models for Communication Games: An Empirical Study on Werewolfcited by 241
  12. 2024Smaug: Fixing Failure Modes of Preference Optimisation with DPO-Positivecited by 241
18 more
  1. 2024From Persona to Personalization: A Survey on Role-Playing Language Agentscited by 212
  2. 2024Agent Q: Advanced Reasoning and Learning for Autonomous AI Agentscited by 208
  3. 2018Decoupling Strategy and Generation in Negotiation Dialoguescited by 162
  4. 2024AgentGym: Evolving Large Language Model-based Agents across Diverse Environmentscited by 120
  5. 2023Avalon's Game of Thoughts: Battle Against Deception through Recursive Contemplationcited by 104
  6. 2024SOTOPIA-π: Interactive Learning of Socially Intelligent Language Agentscited by 93
  7. 2024Watch Every Step! LLM Agent Learning via Iterative Step-Level Process Refinementcited by 88
  8. 2024Self-playing Adversarial Language Game Enhances LLM Reasoningcited by 83
  9. 2023LLM-Based Agent Society Investigation: Collaboration and Confrontation in Avalon Gameplaycited by 63
  10. 2024Werewolf Arena: A Case Study in LLM Evaluation via Social Deductioncited by 45
  11. 2016Learning Language Games through Interactioncited by 43
  12. 2019Finding Friend and Foe in Multi-Agent Gamescited by 42
  13. 2024AgentSense: Benchmarking Social Intelligence of Language Agents through Interactive Scenarioscited by 40
  14. 2024AMONGAGENTS: Evaluating Large Language Models in the Interactive Text-Based Social Deduction Gamecited by 28
  15. 2024Generative Emergent Communication: Large Language Model is a Collective World Modelcited by 15
  16. 2024Helmsman of the Masses? Evaluate the Opinion Leadership of Large Language Models in the Werewolf Gamecited by 13
  17. 2023Dialogue Games for Benchmarking Language Understanding: Motivation, Taxonomy, Strategycited by 12
  18. 2024Boundless Socratic Learning with Language Gamescited by 12

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 6; 2026: 4.

10 papers.

  1. 2025Advances and Challenges in Foundation Agents: From Brain-Inspired Intelligence to Evolutionary, Collaborative, and Safe Systemscited by 126
  2. 2025SocioVerse: A World Model for Social Simulation Powered by LLM Agents and A Pool of 10 Million Real-World Userscited by 55
  3. 2025ARIA: Training Language Agents with Intention-Driven Reward Aggregationcited by 9
  4. 2025Beyond Nash Equilibrium: Bounded Rationality of LLMs and humans in Strategic Decision-makingcited by 9
  5. 2025Tree-of-Reasoning: Towards Complex Medical Diagnosis via Multi-Agent Reasoning with Evidence Treecited by 7
  6. 2026One Model, All Roles: Multi-Turn, Multi-Agent Self-Play Reinforcement Learning for Conversational Social Intelligencecited by 7
  7. 2026Beyond Individual Intelligence: Surveying Collaboration, Failure Attribution, and Self-Evolution in LLM-based Multi-Agent Systemscited by 6
  8. 2025Enhancing Language Agent Strategic Reasoning through Self-Play in Adversarial Gamescited by 1
  9. 2026MARO: Learning Stronger Reasoning from Social Interactioncited by 1
  10. 2026MARBO: Relational Belief Grounding for LLM Agents in Social Deduction Gamescited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Learning Strategic Language Agents in the Werewolf Game with Iterative Latent Space Policy Optimization
  2. 2023Strategic Reasoning with Language Models
  3. 2024A Survey on Large Language Model-Based Game Agents
  4. 2025Playing games with Large language models: Randomness and strategy
  5. 2025Grounding Natural Language for Multi-agent Decision-Making with Multi-agentic LLMs
  6. 2024Large Language Models as Agents in Two-Player Games
  7. 2026LinguaGame: A Linguistically Grounded Game-Theoretic Paradigm for Multi-Agent Dialogue Generation
  8. 2025The Decrypto Benchmark for Multi-Agent Reasoning and Theory of Mind