# Harnessing Distribution Ratio Estimators for Learning Agents with Quality and Diversity

arXiv: https://arxiv.org/abs/2011.02614 · HTML: https://mlsanity.com/paper/2011.02614/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 5 Nov 2020.

## Built on by (most cited first, 7 of 7)

- [Modelling Behavioural Diversity for Learning in Open-Ended Games](https://mlsanity.com/paper/2103.07927/) (2021)
- [Discovering Policies with DOMiNO: Diversity Optimization Maintaining Near Optimality](https://mlsanity.com/paper/2205.13521/) (2022)
- [Discovering Diverse Nearly Optimal Policies with Successor Features](https://mlsanity.com/paper/2106.00669/) (2021)
- [Discovering Diverse Solutions in Deep Reinforcement Learning by Maximizing State-Action-Based Mutual Information](https://mlsanity.com/paper/2103.07084/) (2021)
- [A Coupled Flow Approach to Imitation Learning](https://mlsanity.com/paper/2305.00303/) (2023)
- [Diverse Policy Optimization for Structured Action Space](https://mlsanity.com/paper/2302.11917/) (2023)
- [Policy Dispersion in Non-Markovian Environment](https://mlsanity.com/paper/2302.14509/) (2023)

## Abstract (arXiv)

Quality-Diversity (QD) is a concept from Neuroevolution with some intriguing applications to Reinforcement Learning. It facilitates learning a population of agents where each member is optimized to simultaneously accumulate high task-returns and exhibit behavioral diversity compared to other members. In this paper, we build on a recent kernel-based method for training a QD policy ensemble with Stein variational gradient descent. With kernels based on f-divergence between the stationary distributions of policies, we convert the problem to that of efficient estimation of the ratio of these stationary distributions. We then study various distribution ratio estimators used previously for off-policy evaluation and imitation and re-purpose them to compute the gradients for policies in an ensemble such that the resultant population is diverse and of high-quality.