mlsanityData to 7 Oct 2026

Discovering Policies with DOMiNO: Diversity Optimization Maintaining Near Optimality

arXiv 2205.13521 ↗First posted 26 May 2022cs.AI cs.LGcited by 35

Not on the map. Why?

Abstract (arXiv)

Finding different solutions to the same problem is a key aspect of intelligence associated with creativity and adaptation to novel situations. In reinforcement learning, a set of diverse policies can be useful for exploration, transfer, hierarchy, and robustness. We propose DOMiNO, a method for Diversity Optimization Maintaining Near Optimality. We formalize the problem as a Constrained Markov Decision Process where the objective is to find diverse policies, measured by the distance between the state occupancies of the policies in the set, while remaining near-optimal with respect to the extrinsic reward. We demonstrate that the method can discover diverse and meaningful behaviors in various domains, such as different locomotion patterns in the DeepMind Control Suite. We perform extensive analysis of our approach, compare it with other multi-objective baselines, demonstrate that we can control both the quality and the diversity of the set via interpretable hyperparameters, and show that the discovered set is robust to perturbations.

Key facts. Cited by 35 papers in the map (data through 7 Oct 2026). Builds on 50 of its 79 references that are in the map. First posted on arXiv 26 May 2022.

Builds on

50 of 79 references.

References · 50

  1. 2016Generative Adversarial Imitation Learningcited by 1,576
  2. 2017Curiosity-driven Exploration by Self-supervised Predictioncited by 1,173
  3. 2018DeepMind Control Suitecited by 935
  4. 2018IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architecturescited by 866
  5. 2018Diversity is All You Need: Learning Skills without a Reward Functioncited by 734
  6. 2017In-Datacenter Performance Analysis of a Tensor Processing Unitcited by 610
  7. 2015Illuminating search spaces by mapping elitescited by 406
  8. 2018Reward Constrained Policy Optimizationcited by 397
  9. 2016Successor Features for Transfer in Reinforcement Learningcited by 331
  10. 2019Dynamics-Aware Unsupervised Discovery of Skillscited by 308
  11. 2020Responsive Safety in Reinforcement Learning by PID Lagrangian Methodscited by 302
  12. 2019Challenges of Real-World Reinforcement Learningcited by 282
38 more
  1. 2016Variational Intrinsic Controlcited by 262
  2. 2018Provably Efficient Maximum Entropy Explorationcited by 240
  3. 2020Never Give Up: Learning Directed Exploration Strategiescited by 210
  4. 2014Robots that can adapt like animalscited by 203
  5. 2020Effective Diversity in Population Based Reinforcement Learningcited by 116
  6. 2017Quality and Diversity Optimization: A Unifying Modular Frameworkcited by 111
  7. 2020Variational Policy Gradient Method for Reinforcement Learning with General Utilitiescited by 108
  8. 2019Transfer in Deep Reinforcement Learning Using Successor Features and Generalised Policy Improvementcited by 88
  9. 2021Reward is enough for convex MDPscited by 82
  10. 2020One Solution is Not All You Need: Few-Shot Extrapolation via Structured MaxEnt RLcited by 79
  11. 2018Diversity-Driven Exploration Strategy for Deep Reinforcement Learningcited by 65
  12. 2021The Option Keyboard: Combining Skills in Reinforcement Learningcited by 65
  13. 2019Autonomous skill discovery with Quality-Diversity and Unsupervised Descriptorscited by 57
  14. 2018Reinforcement Learning for Improving Agent Designcited by 55
  15. 2021Concave Utility Reinforcement Learning: the Mean-Field Game Viewpointcited by 54
  16. 2020Diversity Policy Gradient for Sample Efficient Quality-Diversity Optimizationcited by 53
  17. 2021Adversarially Guided Actor-Criticcited by 47
  18. 2020A Boolean Task Algebra for Reinforcement Learningcited by 46
  19. 2017Stein Variational Policy Gradientcited by 42
  20. 2019Off-Policy Actor-Critic with Shared Experience Replaycited by 39
  21. 2021Podracer architectures for scalable Reinforcement Learningcited by 37
  22. 2022Approximating Gradients for Differentiable Quality Diversity in Reinforcement Learningcited by 37
  23. 2021The Information Geometry of Unsupervised Reinforcement Learningcited by 36
  24. 2019Diversity-Inducing Policy Gradient: Using Maximum Mean Discrepancy to Find a Set of Diverse Policiescited by 34
  25. 2020Relative Variational Intrinsic Controlcited by 34
  26. 2022Challenging Common Assumptions in Convex Reinforcement Learningcited by 29
  27. 2021Online Apprenticeship Learningcited by 28
  28. 2022Continuously Discovering Novel Strategies via Reward-Switching Policy Optimizationcited by 23
  29. 2021Discovering a set of policies for the worst case rewardcited by 21
  30. 2019Learning Novel Policies For Taskscited by 20
  31. 2020Balancing Constraints and Rewards with Meta-Gradient D4PGcited by 18
  32. 2021Constructing a Good Behavior Basis for Transfer using Generalized Policy Updatescited by 14
  33. 2019Apprenticeship Learning via Frank-Wolfecited by 13
  34. 2020Non-local Policy Optimization via Diversity-regularized Collaborative Explorationcited by 13
  35. 2019Inverse Reinforcement Learning in Contextual MDPscited by 12
  36. 2020Novel Policy Seeking with Constrained Optimizationcited by 11
  37. 2019Unknown mixing times in apprenticeship and reinforcement learningcited by 7
  38. 2020Harnessing Distribution Ratio Estimators for Learning Agents with Quality and Diversitycited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 3; 2023: 14; 2024: 5; 2025: 11; 2026: 2.

35 papers.

Sort:

By area:

  1. 2023Motif: Intrinsic Motivation from Artificial Intelligence Feedbackcited by 99
  2. 2023Confronting Reward Model Overoptimization with Constrained RLHFcited by 96
  3. 2025Fast Adaptation with Behavioral Foundation Modelscited by 20
  4. 2023ReLOAD: Reinforcement Learning with Optimistic Ascent-Descent for Last-Iterate Convergence in Constrained MDPscited by 19
  5. 2024Proto Successor Measure: Representing the Behavior Space of an RL Agentcited by 17
  6. 2023Diversifying AI: Towards Creative Chess with AlphaZerocited by 15
  7. 2024Quality-Diversity Actor-Critic: Learning High-Performing and Diverse Behaviors via Value and Successor Features Criticscited by 13
  8. 2022Learning General World Models in a Handful of Reward-Free Deploymentscited by 12
  9. 2024Do's and Don'ts: Learning Desirable Skills with Instruction Videoscited by 8
  10. 2025The impact of intrinsic rewards on exploration in Reinforcement Learningcited by 8
  11. 2022Quality Diversity Evolutionary Learning of Decision Treescited by 7
  12. 2023Learning Diverse Skills for Local Navigation under Multi-constraint Optimalitycited by 7
23 more
  1. 2023Understanding the Synergies between Quality-Diversity and Deep Reinforcement Learningcited by 5
  2. 2023Keep Various Trajectories: Promoting Exploration of Ensemble Policies in Continuous Controlcited by 5
  3. 2023Offline Diversity Maximization Under Imitation Constraintscited by 4
  4. 2023Natural and Robust Walking using Reinforcement Learning without Demonstrations in High-Dimensional Musculoskeletal Modelscited by 4
  5. 2024Robot Air Hockey: A Manipulation Testbed for Robot Learning with Reinforcement Learningcited by 4
  6. 2025Enhancing Diversity in Parallel Agents: A Maximum State Entropy Exploration Storycited by 4
  7. 2025From Parameters to Behaviors: Unsupervised Compression of the Policy Spacecited by 4
  8. 2023Iteratively Learn Diverse Strategies with State Distance Informationcited by 3
  9. 2024Convex Markov Games: A New Frontier for Multi-Agent Reinforcement Learningcited by 3
  10. 2025Human-Aligned Skill Discovery: Balancing Behaviour Exploration and Alignmentcited by 3
  11. 2025AutoQD: Automatic Discovery of Diverse Behaviors with Quality-Diversity Optimizationcited by 3
  12. 2023Diversity Through Exclusion (DTE): Niche Identification for Reinforcement Learning through Value-Decompositioncited by 2
  13. 2023Heterogeneous Social Value Orientation Leads to Meaningful Diversity in Sequential Social Dilemmascited by 2
  14. 2025Dual-Force: Enhanced Offline Diversity Maximization under Imitation Constraintscited by 2
  15. 2023Diverse Policy Optimization for Structured Action Spacecited by 1
  16. 2025The Geometry of Nonlinear Reinforcement Learningcited by 1
  17. 2025Soft Quality-Diversity Optimizationcited by 1
  18. 2022Efficient Exploration using Model-Based Quality-Diversity with Gradientscited by 0
  19. 2023APART: Diverse Skill Discovery using All Pairs with Ascending Reward and DropouTcited by 0
  20. 2025Trajectory First: A Curriculum for Discovering Diverse Policiescited by 0
  21. 2025From Tabula Rasa to Emergent Abilities: Discovering Robot Skills via Real-World Unsupervised Quality-Diversitycited by 0
  22. 2026K-Myriad: Jump-starting reinforcement learning with unsupervised parallel agentscited by 0
  23. 2026Reward-Adaptive Iterative Discovery: A Case Study on Automated Game Testing for NHL26cited by 0