# "What are my options?": Explaining RL Agents with Diverse Near-Optimal Alternatives (Extended)

arXiv: https://arxiv.org/abs/2506.09901 · HTML: https://mlsanity.com/paper/2506.09901/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Deep RL algorithms and representations, one of 15 directions in Deep reinforcement learning methods.
- Builds on 13 of its 21 references that are in the map.
- First posted on arXiv 11 Jun 2025.

## Placed in

- Deep reinforcement learning methods / [Deep RL algorithms and representations](https://mlsanity.com/deeprl/deep-rl-algorithms-and-representations/)

## Builds on (most cited first, 13 of 13)

- [Illuminating search spaces by mapping elites](https://mlsanity.com/paper/1504.04909/) (2015)
- [Explainable Reinforcement Learning Through a Causal Lens](https://mlsanity.com/paper/1905.10958/) (2019)
- [Lipschitz Continuity in Model-based Reinforcement Learning](https://mlsanity.com/paper/1804.07193/) (2018)
- [Quality-Diversity Optimization: a novel branch of stochastic optimization](https://mlsanity.com/paper/2012.04322/) (2020)
- [Establishing Appropriate Trust via Critical States](https://mlsanity.com/paper/1810.08174/) (2018)
- [The Emerging Landscape of Explainable AI Planning and Decision Making](https://mlsanity.com/paper/2002.11697/) (2020)
- [Dot-to-Dot: Explainable Hierarchical Reinforcement Learning for Robotic Manipulation](https://mlsanity.com/paper/1904.06703/) (2019)
- [A Survey of Explainable Reinforcement Learning](https://mlsanity.com/paper/2202.08434/) (2022)
- [pyribs: A Bare-Bones Python Library for Quality Diversity Optimization](https://mlsanity.com/paper/2303.00191/) (2023)
- [Uncertain Quality-Diversity: Evaluation methodology and new methods for Quality-Diversity in Uncertain Domains](https://mlsanity.com/paper/2302.00463/) (2023)
- [Explainable Reinforcement Learning via Model Transforms](https://mlsanity.com/paper/2209.12006/) (2022)
- [Training Diverse High-Dimensional Controllers by Scaling Covariance Matrix Adaptation MAP-Annealing](https://mlsanity.com/paper/2210.02622/) (2022)
- [Pointwise-in-Time Explanation for Linear Temporal Logic Rules](https://mlsanity.com/paper/2306.13956/) (2023)

## Abstract (arXiv)

In this work, we provide an extended discussion of a new approach to explainable Reinforcement Learning called Diverse Near-Optimal Alternatives (DNA), first proposed at L4DC 2025. DNA seeks a set of reasonable "options" for trajectory-planning agents, optimizing policies to produce qualitatively diverse trajectories in Euclidean space. In the spirit of explainability, these distinct policies are used to "explain" an agent's options in terms of available trajectory shapes from which a human user may choose. In particular, DNA applies to value function-based policies on Markov decision processes where agents are limited to continuous trajectories. Here, we describe DNA, which uses reward shaping in local, modified Q-learning problems to solve for distinct policies with guaranteed epsilon-optimality. We show that it successfully returns qualitatively different policies that constitute meaningfully different "options" in simulation, including a brief comparison to related approaches in the stochastic optimization field of Quality Diversity. Beyond the explanatory motivation, this work opens new possibilities for exploration and adaptive planning in RL.