mlsanityData to 14 Oct 2026

Modeling Strong and Human-Like Gameplay with KL-Regularized Search

arXiv 2112.07544 ↗First posted 14 Dec 2021cs.MA cs.AI cs.GTcited by 50

On the map: Offline RL and imitation learning from data → Imitation learning methods

Abstract (arXiv)

We consider the task of building strong but human-like policies in multi-agent decision-making problems, given examples of human behavior. Imitation learning is effective at predicting human actions but may not match the strength of expert humans, while self-play learning and search techniques (e.g. AlphaZero) lead to strong performance but may produce policies that are difficult for humans to understand and coordinate with. We show in chess and Go that regularizing search based on the KL divergence from an imitation-learned policy results in higher human prediction accuracy and stronger performance than imitation learning alone. We then introduce a novel regret minimization algorithm that is regularized based on the KL divergence from an imitation-learned policy, and show that using this algorithm for search in no-press Diplomacy yields a policy that matches the human prediction accuracy of imitation learning while being substantially stronger.

Key facts. Cited by 50 papers in the map (data through 14 Oct 2026). Placed in Imitation learning methods, one of 13 directions in Offline RL and imitation learning from data. Builds on 35 of its 80 references that are in the map. First posted on arXiv 14 Dec 2021.

Builds on

35 of 80 references.

References · 35

  1. 2017Squeeze-and-Excitation Networkscited by 4,789
  2. 2019Mastering Atari, Go, Chess and Shogi by Planning with a Learned Modelcited by 1,353
  3. 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 782
  4. 2019Behavior Regularized Offline Reinforcement Learningcited by 662
  5. 2017Overcoming Exploration in Reinforcement Learning with Demonstrationscited by 340
  6. 2017Leveraging Demonstrations for Deep Reinforcement Learning on Robotics Problems with Sparse Rewardscited by 319
  7. 2017DeepStack: Expert-Level Artificial Intelligence in No-Limit Pokercited by 233
  8. 2020Keep Doing What Worked: Behavioral Modelling Priors for Offline Reinforcement Learningcited by 215
  9. 2017Deal or No Deal? End-to-End Learning for Negotiation Dialoguescited by 209
  10. 2020"Other-Play" for Zero-Shot Coordinationcited by 198
  11. 2019The Hanabi Challenge: A New Frontier for AI Researchcited by 197
  12. 2020Human-centric Dialog Training via Offline Reinforcement Learningcited by 93
23 more
  1. 2020Aligning Superhuman AI with Human Behavior: Chess as a Model Systemcited by 79
  2. 2021Off-Belief Learningcited by 75
  3. 2016Reasoning About Pragmatics with Neural Listeners and Speakerscited by 68
  4. 2020Multi-agent Communication meets Natural Language: Synergies between Functional and Structural Language Learningcited by 66
  5. 2019Accelerating Self-Play Learning in Gocited by 64
  6. 2021Fast Policy Extragradient Methods for Competitive Games with Entropy Regularizationcited by 63
  7. 2020Monte-Carlo Tree Search as Regularized Policy Optimizationcited by 61
  8. 2019Countering Language Drift via Visual Groundingcited by 55
  9. 2017Context-aware Captions from Context-agnostic Supervisioncited by 43
  10. 2020Human-Level Performance in No-Press Diplomacy via Equilibrium Searchcited by 41
  11. 2021Evaluation of Human-AI Teams for Learned and Rule-Based Agents in Hanabicited by 40
  12. 2019No Press Diplomacy: Modeling Multi-Agent Gameplaycited by 38
  13. 2019Improving Policies via Search in Cooperative Partially Observable Gamescited by 33
  14. 2021No-Press Diplomacy from Scratchcited by 31
  15. 2020Learning to Play No-Press Diplomacy with Best Response Policy Iterationcited by 26
  16. 2017Hierarchical Text Generation and Planning for Strategic Dialoguecited by 23
  17. 2018What game are we playing? End-to-end learning in normal and extensive form gamescited by 22
  18. 2018Online Convex Optimization for Sequential Decision Processes and Extensive-Form Gamescited by 21
  19. 2021Scalable Online Planning via Reinforcement Learning Fine-Tuningcited by 18
  20. 2021Learned Belief Search: Efficiently Improving Policies in Partially Observable Settingscited by 10
  21. 2021Multitasking Inhibits Semantic Driftcited by 8
  22. 2015Better Computer Go Player with Neural Network and Long-term Predictioncited by 4
  23. 2017Beyond Monte Carlo Tree Search: Playing Go with Deep Alternative Neural Network and Long-Term Evaluationcited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2022: 8; 2023: 13; 2024: 14; 2025: 7; 2026: 8.

50 papers.

Sort:

By area:

  1. 2022A Unified Approach to Reinforcement Learning, Quantal Response Equilibria, and Two-Player Zero-Sum Gamescited by 66
  2. 2023Imitation Bootstrapped Reinforcement Learningcited by 64
  3. 2022Mastering the Game of No-Press Diplomacy via Human-Regularized Reinforcement Learning and Planningcited by 48
  4. 2023Inverse Reinforcement Learning without Reinforcement Learningcited by 36
  5. 2023The Consensus Game: Language Model Generation via Equilibrium Searchcited by 36
  6. 2024Richelieu: Self-Evolving LLM-Based Agents for AI Diplomacycited by 36
  7. 2025Amulet: ReAlignment During Test Time for Personalized Preference Adaptation of LLMscited by 29
  8. 2024Maia-2: A Unified Model for Human-AI Alignment in Chesscited by 25
  9. 2024Human-compatible driving partners through data-regularized self-play reinforcement learningcited by 17
  10. 2023Diverse Conventions for Human-AI Collaborationcited by 15
  11. 2022Mimetic Models: Ethical Implications of AI that Acts Like Youcited by 11
  12. 2022Automated Play-Testing Through RL Based Human-Like Play-Styles Generationcited by 11
38 more
  1. 2023Navigates Like Me: Understanding How People Evaluate Human-Like AI in Video Gamescited by 11
  2. 2023Scaling Laws for Imitation Learning in Single-Agent Gamescited by 11
  3. 2024Human-aligned Chess with a Bit of Searchcited by 11
  4. 2024Designing Skill-Compatible AI: Methodologies and Frameworks in Chesscited by 10
  5. 2025AssistanceZero: Scalably Solving Assistance Gamescited by 10
  6. 2022Human-AI Coordination via Human-Regularized Search and Learningcited by 8
  7. 2022Optimal Behavior Prior: Data-Efficient Human Models for Improved Human-AI Collaborationcited by 8
  8. 2025Ad-Hoc Human-AI Coordination Challengecited by 8
  9. 2025Learning to Imitate with Less: Efficient Individual Behavior Modeling in Chesscited by 7
  10. 2024Human-Agent Cooperation in Games under Incomplete Information through Natural Language Communicationcited by 6
  11. 2026Chessformer: A Unified Architecture for Chess Modelingcited by 6
  12. 2025RSPO: Regularized Self-Play Alignment of Large Language Modelscited by 5
  13. 2023Towards Human-Like RL: Taming Non-Naturalistic Behavior in Deep RL via Adaptive Behavioral Costs in 3D Gamescited by 4
  14. 2023Regularized Conventions: Equilibrium Computation as a Model of Pragmatic Reasoningcited by 4
  15. 2024Tractable Equilibrium Computation in Markov Games through Risk Aversioncited by 4
  16. 2024Explaining Decisions of Agents in Mixed-Motive Gamescited by 4
  17. 2025DipLLM: Fine-Tuning LLM for Strategic Decision-making in Diplomacycited by 4
  18. 2026Human-like autonomy emerges from self-play and a pinch of human datacited by 4
  19. 2023The Update-Equivalence Framework for Decision-Time Planningcited by 3
  20. 2023Smooth Nash Equilibria: Algorithms and Complexitycited by 3
  21. 2023Modeling Boundedly Rational Agents with Latent Inference Budgetscited by 3
  22. 2024Truth or Deceit? A Bayesian Decoding Game Enhances Consistency and Reliabilitycited by 3
  23. 2024Convex Markov Games: A New Frontier for Multi-Agent Reinforcement Learningcited by 3
  24. 2024Personalized Help for Optimizing Low-Skilled Users' Strategycited by 3
  25. 2022AutoReply: Detecting Nonsense in Dialogue Introspectively with Discriminative Repliescited by 2
  26. 2024Inverse Concave-Utility Reinforcement Learning is Inverse Game Theorycited by 2
  27. 2025Modelling bounded rational decision-making through Wasserstein constraintscited by 2
  28. 2026ALIGN: Aligned Delegation with Performance Guarantees for Multi-Agent LLM Reasoningcited by 2
  29. 2026Deep Incentive Design with Differentiable Equilibrium Blockscited by 2
  30. 2022Towards automating Codenames spymasters with deep reinforcement learningcited by 1
  31. 2023Revisiting Estimation Bias in Policy Gradients for Deep Reinforcement Learningcited by 1
  32. 2026LinguaGame: A Linguistically Grounded Game-Theoretic Paradigm for Multi-Agent Dialogue Generationcited by 1
  33. 2026JaxAHT: A JAX-Based Library for Ad Hoc Teamworkcited by 1
  34. 2023Learned Tree Search for Long-Horizon Social Robot Navigation in Shared Airspacecited by 0
  35. 2024Securing Equal Share: A Principled Approach for Learning Multiplayer Symmetric Gamescited by 0
  36. 2024Noisy Zero-Shot Coordination: Breaking The Common Knowledge Assumption In Zero-Shot Coordination Gamescited by 0
  37. 2026Improving Human Performance with Value-Aware Interventions: A Case Study in Chesscited by 0
  38. 2026AI Coaching for Accelerating Human Skill Development with Reinforcement Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Mimicking To Dominate: Imitation Learning Strategies for Success in Multiagent Competitive Games
  2. 2022A Ranking Game for Imitation Learning
  3. 2021Of Moments and Matching: A Game-Theoretic Framework for Closing the Imitation Gap
  4. 2024A Minimaximalist Approach to Reinforcement Learning from Human Feedback
  5. 2026Beyond Search-Imitation: Prior-Directed Exploration for Searchless Chess
  6. 2026Matching Multiple Experts: On the Exploitability of Multi-Agent Imitation Learning
  7. 2020Self-Imitation Advantage Learning
  8. 2019A New Framework for Query Efficient Active Imitation Learning