# Diversity-Inducing Policy Gradient: Using Maximum Mean Discrepancy to Find a Set of Diverse Policies

arXiv: https://arxiv.org/abs/1906.00088 · HTML: https://mlsanity.com/paper/1906.00088/

## Key facts

- Cited by 34 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 31 May 2019.

## Built on by (most cited first, 30 of 34)

- [Maximum Entropy Population-Based Training for Zero-Shot Human-AI Coordination](https://mlsanity.com/paper/2112.11701/) (2021)
- [Diversity-Aware Policy Optimization for Large Language Model Reasoning](https://mlsanity.com/paper/2505.23433/) (2025)
- [Discovering Policies with DOMiNO: Diversity Optimization Maintaining Near Optimality](https://mlsanity.com/paper/2205.13521/) (2022)
- [Revisiting Some Common Practices in Cooperative Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2206.07505/) (2022)
- [PECAN: Leveraging Policy Ensemble for Context-Aware Zero-Shot Human-AI Coordination](https://mlsanity.com/paper/2301.06387/) (2023)
- [Continuously Discovering Novel Strategies via Reward-Switching Policy Optimization](https://mlsanity.com/paper/2204.02246/) (2022)
- [Neuroevolution is a Competitive Alternative to Reinforcement Learning for Skill Discovery](https://mlsanity.com/paper/2210.03516/) (2022)
- [Discovering Diverse Nearly Optimal Policies with Successor Features](https://mlsanity.com/paper/2106.00669/) (2021)
- [Towards Applicable Reinforcement Learning: Improving the Generalization and Sample Efficiency with Policy Ensemble](https://mlsanity.com/paper/2205.09284/) (2022)
- [Diversifying AI: Towards Creative Chess with AlphaZero](https://mlsanity.com/paper/2308.09175/) (2023)
- [Unifying Behavioral and Response Diversity for Open-ended Learning in Zero-sum Games](https://mlsanity.com/paper/2106.04958/) (2021)
- [System Neural Diversity: Measuring Behavioral Heterogeneity in Multi-Agent Learning](https://mlsanity.com/paper/2305.02128/) (2023)
- [Adaptable Agent Populations via a Generative Model of Policies](https://mlsanity.com/paper/2107.07506/) (2021)
- [SetPO: Set-Level Policy Optimization for Diversity-Preserving LLM Reasoning](https://mlsanity.com/paper/2602.01062/) (2026)
- [Diversity-Enhanced Reasoning for Subjective Questions](https://mlsanity.com/paper/2507.20187/) (2025)
- [Learning Diverse Risk Preferences in Population-based Self-play](https://mlsanity.com/paper/2305.11476/) (2023)
- [Keep Various Trajectories: Promoting Exploration of Ensemble Policies in Continuous Control](https://mlsanity.com/paper/2310.11138/) (2023)
- [Measuring Policy Distance for Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2401.11257/) (2024)
- [Iteratively Learn Diverse Strategies with State Distance Information](https://mlsanity.com/paper/2310.14509/) (2023)
- [Learning Diverse Policies with Soft Self-Generated Guidance](https://mlsanity.com/paper/2402.04539/) (2024)
- [Diversity Through Exclusion (DTE): Niche Identification for Reinforcement Learning through Value-Decomposition](https://mlsanity.com/paper/2302.01180/) (2023)
- [Policy Diversity for Cooperative Agents](https://mlsanity.com/paper/2308.14308/) (2023)
- [Adaptive trajectory-constrained exploration strategy for deep reinforcement learning](https://mlsanity.com/paper/2312.16456/) (2023)
- [Safe adaptation in multiagent competition](https://mlsanity.com/paper/2203.07562/) (2022)
- [Open-Ended Diverse Solution Discovery with Regulated Behavior Patterns for Cross-Domain Adaptation](https://mlsanity.com/paper/2209.12029/) (2022)
- [Diverse Policy Optimization for Structured Action Space](https://mlsanity.com/paper/2302.11917/) (2023)
- [Policy Dispersion in Non-Markovian Environment](https://mlsanity.com/paper/2302.14509/) (2023)
- [Increasing Entropy to Boost Policy Gradient Performance on Personalization Tasks](https://mlsanity.com/paper/2310.05324/) (2023)
- [Policy Optimization with Smooth Guidance Learned from State-Only Demonstrations](https://mlsanity.com/paper/2401.00162/) (2023)
- [Phasic Diversity Optimization for Population-Based Reinforcement Learning](https://mlsanity.com/paper/2403.11114/) (2024)

## Abstract (arXiv)

Standard reinforcement learning methods aim to master one way of solving a task whereas there may exist multiple near-optimal policies. Being able to identify this collection of near-optimal policies can allow a domain expert to efficiently explore the space of reasonable solutions. Unfortunately, existing approaches that quantify uncertainty over policies are not ultimately relevant to finding policies with qualitatively distinct behaviors. In this work, we formalize the difference between policies as a difference between the distribution of trajectories induced by each policy, which encourages diversity with respect to both state visitation and action choices. We derive a gradient-based optimization technique that can be combined with existing policy gradient methods to now identify diverse collections of well-performing policies. We demonstrate our approach on benchmarks and a healthcare task.