# Quality-Diversity Actor-Critic: Learning High-Performing and Diverse Behaviors via Value and Successor Features Critics

arXiv: https://arxiv.org/abs/2403.09930 · HTML: https://mlsanity.com/paper/2403.09930/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Placed in RL theory and sample complexity, one of 15 directions in Deep reinforcement learning methods.
- Builds on 49 of its 68 references that are in the map.
- First posted on arXiv 15 Mar 2024.

## Placed in

- Deep reinforcement learning methods / [RL theory and sample complexity](https://mlsanity.com/deeprl/rl-theory-and-sample-complexity/)

## Builds on (most cited first, 30 of 49)

- [Model-Agnostic Meta-Learning for Fast Adaptation of Deep Networks](https://mlsanity.com/paper/1703.03400/) (2017)
- [Continuous control with deep reinforcement learning](https://mlsanity.com/paper/1509.02971/) (2015)
- [Playing Atari with Deep Reinforcement Learning](https://mlsanity.com/paper/1312.5602/) (2013)
- [Addressing Function Approximation Error in Actor-Critic Methods](https://mlsanity.com/paper/1802.09477/) (2018)
- [Dream to Control: Learning Behaviors by Latent Imagination](https://mlsanity.com/paper/1912.01603/) (2019)
- [World Models](https://mlsanity.com/paper/1803.10122/) (2018)
- [Learning Latent Dynamics for Planning from Pixels](https://mlsanity.com/paper/1811.04551/) (2018)
- [Mastering Diverse Domains through World Models](https://mlsanity.com/paper/2301.04104/) (2023)
- [Soft Actor-Critic Algorithms and Applications](https://mlsanity.com/paper/1812.05905/) (2018)
- [Deep Reinforcement Learning at the Edge of the Statistical Precipice](https://mlsanity.com/paper/2108.13264/) (2021)
- [Diversity is All You Need: Learning Skills without a Reward Function](https://mlsanity.com/paper/1802.06070/) (2018)
- [Solving Rubik's Cube with a Robot Hand](https://mlsanity.com/paper/1910.07113/) (2019)
- [Illuminating search spaces by mapping elites](https://mlsanity.com/paper/1504.04909/) (2015)
- [Brax -- A Differentiable Physics Engine for Large Scale Rigid Body Simulation](https://mlsanity.com/paper/2106.13281/) (2021)
- [Successor Features for Transfer in Reinforcement Learning](https://mlsanity.com/paper/1606.05312/) (2016)
- [Maximum a Posteriori Policy Optimisation](https://mlsanity.com/paper/1806.06920/) (2018)
- [Dynamics-Aware Unsupervised Discovery of Skills](https://mlsanity.com/paper/1907.01657/) (2019)
- [Variational Intrinsic Control](https://mlsanity.com/paper/1611.07507/) (2016)
- [First return, then explore](https://mlsanity.com/paper/2004.12919/) (2020)
- [Robots that can adapt like animals](https://mlsanity.com/paper/1407.3501/) (2014)
- [Behavior From the Void: Unsupervised Active Pre-Training](https://mlsanity.com/paper/2103.04551/) (2021)
- [Rapid Locomotion via Reinforcement Learning](https://mlsanity.com/paper/2205.02824/) (2022)
- [Emergence of Locomotion Behaviours in Rich Environments](https://mlsanity.com/paper/1707.02286/) (2017)
- [Walk These Ways: Tuning Robot Control for Generalization with Multiplicity of Behavior](https://mlsanity.com/paper/2212.03238/) (2022)
- [Quality and Diversity Optimization: A Unifying Modular Framework](https://mlsanity.com/paper/1708.09251/) (2017)
- [Universal Successor Features Approximators](https://mlsanity.com/paper/1812.07626/) (2018)
- [One Solution is Not All You Need: Few-Shot Extrapolation via Structured MaxEnt RL](https://mlsanity.com/paper/2010.14484/) (2020)
- [Covariance Matrix Adaptation for the Rapid Illumination of Behavior Space](https://mlsanity.com/paper/1912.02400/) (2019)
- [Diversity Policy Gradient for Sample Efficient Quality-Diversity Optimization](https://mlsanity.com/paper/2006.08505/) (2020)
- [Scaling MAP-Elites to Deep Neuroevolution](https://mlsanity.com/paper/2003.01825/) (2020)

## Built on by (most cited first, 13 of 13)

- [Diversity-Incentivized Exploration for Versatile Reasoning](https://mlsanity.com/paper/2509.26209/) (2025)
- [The impact of intrinsic rewards on exploration in Reinforcement Learning](https://mlsanity.com/paper/2501.11533/) (2025)
- [Zero-Shot Adaptation of Behavioral Foundation Models to Unseen Dynamics](https://mlsanity.com/paper/2505.13150/) (2025)
- [Diversifying Policy Behaviors with Extrinsic Behavioral Curiosity](https://mlsanity.com/paper/2410.06151/) (2024)
- [Walk Wisely on Graph: Knowledge Graph Reasoning with Dual Agents via Efficient Guidance-Exploration](https://mlsanity.com/paper/2408.01880/) (2024)
- [Full Domain Analysis in Fluid Dynamics](https://mlsanity.com/paper/2505.22275/) (2025)
- [Is On-Policy Data always the Best Choice for Direct Preference Optimization-based LM Alignment?](https://mlsanity.com/paper/2508.10530/) (2025)
- [The Geometry of Nonlinear Reinforcement Learning](https://mlsanity.com/paper/2509.01432/) (2025)
- [Discount Model Search for Quality Diversity Optimization in High-Dimensional Measure Spaces](https://mlsanity.com/paper/2601.01082/) (2026)
- [Trajectory First: A Curriculum for Discovering Diverse Policies](https://mlsanity.com/paper/2506.01568/) (2025)
- [From Tabula Rasa to Emergent Abilities: Discovering Robot Skills via Real-World Unsupervised Quality-Diversity](https://mlsanity.com/paper/2508.19172/) (2025)
- [Structure-Conditioned Actor-Critic Branches for Quality-Diversity Reinforcement Learning](https://mlsanity.com/paper/2606.08735/) (2026)
- [Contextual Quality-Diversity Evolutionary Reinforcement Learning for HVAC Control in Tropical Commercial Buildings](https://mlsanity.com/paper/2608.11324/) (2026)

## Abstract (arXiv)

A key aspect of intelligence is the ability to demonstrate a broad spectrum of behaviors for adapting to unexpected situations. Over the past decade, advancements in deep reinforcement learning have led to groundbreaking achievements to solve complex continuous control tasks. However, most approaches return only one solution specialized for a specific problem. We introduce Quality-Diversity Actor-Critic (QDAC), an off-policy actor-critic deep reinforcement learning algorithm that leverages a value function critic and a successor features critic to learn high-performing and diverse behaviors. In this framework, the actor optimizes an objective that seamlessly unifies both critics using constrained optimization to (1) maximize return, while (2) executing diverse skills. Compared with other Quality-Diversity methods, QDAC achieves significantly higher performance and more diverse behaviors on six challenging continuous control locomotion tasks. We also demonstrate that we can harness the learned skills to adapt better than other baselines to five perturbed environments. Finally, qualitative analyses showcase a range of remarkable behaviors: adaptive-intelligent-robotics.github.io/QDAC.