mlsanityData to 7 Oct 2026

Discovering Diverse Multi-Agent Strategic Behavior via Reward Randomization

arXiv 2103.04564 ↗First posted 8 Mar 2021cs.AI cs.LGcited by 35

Not on the map. Why?

Abstract (arXiv)

We propose a simple, general and effective technique, Reward Randomization for discovering diverse strategic policies in complex multi-agent games. Combining reward randomization and policy gradient, we derive a new algorithm, Reward-Randomized Policy Gradient (RPG). RPG is able to discover multiple distinctive human-interpretable strategies in challenging temporal trust dilemmas, including grid-world games and a real-world game Agar.io, where multiple equilibria exist but standard multi-agent policy gradient algorithms always converge to a fixed one with a sub-optimal payoff for every player even using state-of-the-art exploration techniques. Furthermore, with the set of diverse strategies from RPG, we can (1) achieve higher payoffs by fine-tuning the best policy from the set; and (2) obtain an adaptive agent by using this set of strategies as its training opponents. The source code and example videos can be found in our website: https://sites.google.com/view/staghuntrpg.

Key facts. Cited by 35 papers in the map (data through 7 Oct 2026). Builds on 41 of its 82 references that are in the map. First posted on arXiv 8 Mar 2021.

Builds on

41 of 82 references.

References · 41

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2017Multi-Agent Actor-Critic for Mixed Cooperative-Competitive Environmentscited by 1,519
  3. 2017Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real Worldcited by 1,401
  4. 2017Curiosity-driven Exploration by Self-supervised Predictioncited by 1,173
  5. 2019Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learningcited by 1,168
  6. 2019Dota 2 with Large Scale Deep Reinforcement Learningcited by 1,084
  7. 2018Exploration by Random Network Distillationcited by 966
  8. 2015End-to-End Training of Deep Visuomotor Policiescited by 847
  9. 2018Diversity is All You Need: Learning Skills without a Reward Functioncited by 734
  10. 2017Population Based Training of Neural Networkscited by 351
  11. 2019Dynamics-Aware Unsupervised Discovery of Skillscited by 308
  12. 2019Emergent Tool Use From Multi-Agent Autocurriculacited by 308
29 more
  1. 2016#Exploration: A Study of Count-Based Exploration for Deep Reinforcement Learningcited by 307
  2. 2017StarCraft II: A New Challenge for Reinforcement Learningcited by 268
  3. 2017A Unified Game-Theoretic Approach to Multiagent Reinforcement Learningcited by 265
  4. 2018Human-level performance in first-person multiplayer games with population-based deep reinforcement learningcited by 260
  5. 2017Inverse Reward Designcited by 246
  6. 2015Escaping From Saddle Points --- Online Stochastic Gradient for Tensor Decompositioncited by 241
  7. 2017Learning with Opponent-Learning Awarenesscited by 218
  8. 2017Multi-agent Reinforcement Learning in Sequential Social Dilemmascited by 206
  9. 2014Robots that can adapt like animalscited by 203
  10. 2020"Other-Play" for Zero-Shot Coordinationcited by 194
  11. 2016Learning to Navigate in Complex Environmentscited by 192
  12. 2019MAVEN: Multi-Agent Variational Explorationcited by 181
  13. 2016Deep Reinforcement Learning from Self-Play in Imperfect-Information Gamescited by 153
  14. 2018An Alternative View: When Does SGD Escape Local Minima?cited by 115
  15. 2020Provable Self-Play Algorithms for Competitive Reinforcement Learningcited by 112
  16. 2018Building Generalizable Agents with a Realistic and Rich 3D Environmentcited by 109
  17. 2019Open-ended Learning in Symmetric Zero-sum Gamescited by 96
  18. 2017Intrinsically Motivated Goal Exploration Processes with Automatic Curriculum Learningcited by 79
  19. 2020From Poincar'e Recurrence to Convergence in Imperfect Information Games: Finding Equilibrium via Regularizationcited by 59
  20. 2019Towards Practical Multi-Object Manipulation using Relational Reinforcement Learningcited by 54
  21. 2020Social diversity and social preferences in mixed-motive reinforcement learningcited by 49
  22. 2017Prosocial learning agents solve generalized Stag Hunts better than selfish onescited by 47
  23. 2020Evolutionary Population Curriculum for Scaling Multi-Agent Reinforcement Learningcited by 47
  24. 2017Consequentialist conditional cooperation in social dilemmas with imperfect informationcited by 23
  25. 2019Learning to Interactively Learn and Assistcited by 20
  26. 2018Deep Reinforcement Learning for Green Security Games with Real-Time Informationcited by 19
  27. 2013Nash Convergence of Gradient Dynamics in Iterated General-Sum Gamescited by 18
  28. 2019Deep Fictitious Play for Finding Markovian Nash Equilibrium in Multi-Agent Gamescited by 16
  29. 2019Robust Opponent Modeling via Adversarial Ensemble Reinforcement Learning in Asymmetric Imperfect-Information Gamescited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 4; 2022: 6; 2023: 12; 2024: 6; 2025: 5; 2026: 2.

35 papers.

Sort:

By area:

  1. 2021Maximum Entropy Population-Based Training for Zero-Shot Human-AI Coordinationcited by 69
  2. 2025Diversity-Aware Policy Optimization for Large Language Model Reasoningcited by 39
  3. 2023Learning Zero-Shot Cooperation with Humans, Assuming Humans Are Biasedcited by 37
  4. 2022Revisiting Some Common Practices in Cooperative Multi-Agent Reinforcement Learningcited by 32
  5. 2022Continuously Discovering Novel Strategies via Reward-Switching Policy Optimizationcited by 23
  6. 2024Learning to Cooperate with Humans using Generative Agentscited by 23
  7. 2021Model-Based Opponent Modelingcited by 21
  8. 2022Generating Teammates for Training Robust Ad Hoc Teamwork Agents via Best-Response Diversitycited by 17
  9. 2023Policy Space Diversity for Non-Transitive Gamescited by 15
  10. 2023Diverse Conventions for Human-AI Collaborationcited by 14
  11. 2021Robust Reinforcement Learning Under Minimax Regret for Green Securitycited by 12
  12. 2022DGPO: Discovering Multiple Strategies with Diversity-Guided Policy Optimizationcited by 10
23 more
  1. 2024Rethinking Robustness Assessment: Adversarial Attacks on Learning-based Quadrupedal Locomotion Controllerscited by 9
  2. 2022Human-AI Coordination via Human-Regularized Search and Learningcited by 8
  3. 2026Vector Policy Optimization: Training for Diversity Improves Test-Time Searchcited by 8
  4. 2024Dreaming to Assist: Learning to Align with Human Objectives for Shared Control in High-Speed Racingcited by 7
  5. 2021Learning Meta Representations for Agents in Multi-Agent Reinforcement Learningcited by 6
  6. 2023Learning Diverse Risk Preferences in Population-based Self-playcited by 6
  7. 2023Robust Driving Policy Learning with Guided Meta Reinforcement Learningcited by 5
  8. 2025VS-Bench: Evaluating VLMs for Strategic Abilities in Multi-Agent Environmentscited by 5
  9. 2022Beyond Rewards: a Hierarchical Perspective on Offline Multiagent Behavioral Analysiscited by 3
  10. 2023Iteratively Learn Diverse Strategies with State Distance Informationcited by 3
  11. 2024Importance Sampling-Guided Meta-Training for Intelligent Agents in Highly Interactive Environmentscited by 3
  12. 2025Improving Human-AI Coordination through Online Adversarial Training and Generative Modelscited by 3
  13. 2025Adaptively Coordinating with Novel Partners via Learned Latent Strategiescited by 3
  14. 2023Heterogeneous Social Value Orientation Leads to Meaningful Diversity in Sequential Social Dilemmascited by 2
  15. 2023Context-Aware Bayesian Network Actor-Critic Methods for Cooperative Multi-Agent Reinforcement Learningcited by 2
  16. 2026Benchmarking the Limits of In-Context Reinforcement Learning for Ad-Hoc Teamworkcited by 2
  17. 2023Differentiable Arbitrating in Zero-sum Markov Gamescited by 1
  18. 2023Diverse Policy Optimization for Structured Action Spacecited by 1
  19. 2023Accelerate Multi-Agent Reinforcement Learning in Zero-Sum Games with Subgame Curriculum Learningcited by 1
  20. 2025LLMs as Policy-Agnostic Teammates: A Case Study in Human Proxy Design for Heterogeneous Agent Teamscited by 1
  21. 2023Benchmarking Robustness and Generalization in Multi-Agent Systems: A Case Study on Neural MMOcited by 0
  22. 2024Phasic Diversity Optimization for Population-Based Reinforcement Learningcited by 0
  23. 2024Preference-based opponent shaping in differentiable gamescited by 0