mlsanityData to 13 Oct 2026

Continuously Discovering Novel Strategies via Reward-Switching Policy Optimization

arXiv 2204.02246 ↗First posted 4 Apr 2022cs.LG cs.AIcited by 23

Not on the map. Why?

Abstract (arXiv)

We present Reward-Switching Policy Optimization (RSPO), a paradigm to discover diverse strategies in complex RL environments by iteratively finding novel policies that are both locally optimal and sufficiently different from existing ones. To encourage the learning policy to consistently converge towards a previously undiscovered local optimum, RSPO switches between extrinsic and intrinsic rewards via a trajectory-based novelty measurement during the optimization process. When a sampled trajectory is sufficiently distinct, RSPO performs standard policy optimization with extrinsic rewards. For trajectories with high likelihood under existing policies, RSPO utilizes an intrinsic diversity reward to promote exploration. Experiments show that RSPO is able to discover a wide spectrum of strategies in a variety of domains, ranging from single-agent particle-world tasks and MuJoCo continuous control to multi-agent stag-hunt games and StarCraftII challenges.

Key facts. Cited by 23 papers in the map (data through 13 Oct 2026). Builds on 36 of its 54 references that are in the map. First posted on arXiv 4 Apr 2022.

Builds on

36 of 54 references.

References · 36

  1. 2017Proximal Policy Optimization Algorithmscited by 14,592
  2. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,656
  3. 2015Trust Region Policy Optimizationcited by 2,513
  4. 2014Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modelingcited by 2,352
  5. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,310
  6. 2017Curiosity-driven Exploration by Self-supervised Predictioncited by 1,180
  7. 2018Exploration by Random Network Distillationcited by 971
  8. 2018Diversity is All You Need: Learning Skills without a Reward Functioncited by 739
  9. 2019The StarCraft Multi-Agent Challengecited by 592
  10. 2019Emergent Tool Use From Multi-Agent Autocurriculacited by 310
  11. 2016Deep Reinforcement Learning for Dialogue Generationcited by 287
  12. 2017StarCraft II: A New Challenge for Reinforcement Learningcited by 269
24 more
  1. 2017A Unified Game-Theoretic Approach to Multiagent Reinforcement Learningcited by 265
  2. 2018Human-level performance in first-person multiplayer games with population-based deep reinforcement learningcited by 260
  3. 2017Emergence of Grounded Compositional Language in Multi-Agent Populationscited by 259
  4. 2014Robots that can adapt like animalscited by 203
  5. 2019The Hanabi Challenge: A New Frontier for AI Researchcited by 194
  6. 2019MAVEN: Multi-Agent Variational Explorationcited by 183
  7. 2020Effective Diversity in Population Based Reinforcement Learningcited by 116
  8. 2018On Learning Intrinsic Rewards for Policy Gradient Methodscited by 101
  9. 2020One Solution is Not All You Need: Few-Shot Extrapolation via Structured MaxEnt RLcited by 79
  10. 2019Influence-Based Multi-Agent Explorationcited by 75
  11. 2020The AI Economist: Improving Equality and Productivity with AI-Driven Tax Policiescited by 75
  12. 2019Dynamical Distance Learning for Semi-Supervised and Unsupervised Skill Discoverycited by 67
  13. 2018Diversity-Driven Exploration Strategy for Deep Reinforcement Learningcited by 65
  14. 2019Emergent Coordination Through Competitioncited by 57
  15. 2020Efficient Continuous Pareto Exploration in Multi-Task Learningcited by 57
  16. 2021Modelling Behavioural Diversity for Learning in Open-Ended Gamescited by 39
  17. 2021Discovering Diverse Multi-Agent Strategic Behavior via Reward Randomizationcited by 35
  18. 2019Diversity-Inducing Policy Gradient: Using Maximum Mean Discrepancy to Find a Set of Diverse Policiescited by 34
  19. 2020Ridge Rider: Finding Diverse Solutions by Following Eigenvectors of the Hessiancited by 20
  20. 2021Discovering Diverse Nearly Optimal Policies with Successor Featurescited by 20
  21. 2021Unifying Behavioral and Response Diversity for Open-ended Learning in Zero-sum Gamescited by 14
  22. 2020Novel Policy Seeking with Constrained Optimizationcited by 11
  23. 2021Why Do Local Methods Solve Nonconvex Problems?cited by 8
  24. 2019Reward Prediction Error as an Exploration Objective in Deep RLcited by 2

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2022: 7; 2023: 9; 2024: 3; 2025: 3; 2026: 1.

23 papers.

Sort:

By area:

  1. 2025Diversity-Aware Policy Optimization for Large Language Model Reasoningcited by 40
  2. 2023Learning Zero-Shot Cooperation with Humans, Assuming Humans Are Biasedcited by 37
  3. 2022Discovering Policies with DOMiNO: Diversity Optimization Maintaining Near Optimalitycited by 35
  4. 2022Revisiting Some Common Practices in Cooperative Multi-Agent Reinforcement Learningcited by 32
  5. 2022Neuroevolution is a Competitive Alternative to Reinforcement Learning for Skill Discoverycited by 23
  6. 2022Learning General World Models in a Handful of Reward-Free Deploymentscited by 12
  7. 2022DGPO: Discovering Multiple Strategies with Diversity-Guided Policy Optimizationcited by 10
  8. 2025Diversity-Enhanced Reasoning for Subjective Questionscited by 9
  9. 2024Do's and Don'ts: Learning Desirable Skills with Instruction Videoscited by 8
  10. 2023Communication-Robust Multi-Agent Learning by Adaptable Auxiliary Multi-Agent Adversary Generationcited by 6
  11. 2023Learning Diverse Risk Preferences in Population-based Self-playcited by 6
  12. 2023Keep Various Trajectories: Promoting Exploration of Ensemble Policies in Continuous Controlcited by 5
11 more
  1. 2022Beyond Rewards: a Hierarchical Perspective on Offline Multiagent Behavioral Analysiscited by 3
  2. 2023Iteratively Learn Diverse Strategies with State Distance Informationcited by 3
  3. 2022Open-Ended Diverse Solution Discovery with Regulated Behavior Patterns for Cross-Domain Adaptationcited by 1
  4. 2023Diverse Policy Optimization for Structured Action Spacecited by 1
  5. 2023Policy Dispersion in Non-Markovian Environmentcited by 0
  6. 2023Diverse Policies Converge in Reward-free Markov Decision Processecited by 0
  7. 2023Diversity for Contingency: Learning Diverse Behaviors for Efficient Adaptation and Transfercited by 0
  8. 2024Uniformly Safe RL with Objective Suppression for Multi-Constraint Safety-Critical Applicationscited by 0
  9. 2024Phasic Diversity Optimization for Population-Based Reinforcement Learningcited by 0
  10. 2025Trajectory First: A Curriculum for Discovering Diverse Policiescited by 0
  11. 2026Explicit Trajectory Diversity for RL-Based Post-Training of LLM Agentscited by 0