mlsanityData to 14 Oct 2026

Distributional Value Estimation Without Target Networks for Robust Quality-Diversity

arXiv 2604.20381 ↗First posted 22 Apr 2026cs.LG cs.NE cs.ROcited by 0

On the map: Deep reinforcement learning methods → Deep RL algorithms and representations

Abstract (arXiv)

Quality-Diversity (QD) algorithms excel at discovering diverse repertoires of skills, but are hindered by poor sample efficiency and often require tens of millions of environment steps to solve complex locomotion tasks. Recent advances in Reinforcement Learning (RL) have shown that high Update-to-Data (UTD) ratios accelerate Actor-Critic learning. While effective, standard high-UTD algorithms typically utilise target networks to stabilise training. This requirement introduces a significant computational bottleneck, rendering them impractical for resource-intensive Quality-Diversity (QD) tasks where sample efficiency and rapid population adaptation are critical. In this paper, we introduce QDHUAC, a sample-efficient, target-free and distributional QD-RL algorithm that provides dense and low-variance gradient signals, which enables high-UTD training for Dominated Novelty Search whilst requiring an order of magnitude fewer environment steps. We demonstrate that our method enables stable training at high UTD ratios, achieving competitive coverage and fitness on high-dimensional Brax environments with an order of magnitude fewer samples than baselines. Our results suggest that combining target-free distributional critics with dominance-based selection is a key enabler for the next generation of sample-efficient evolutionary RL algorithms.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Deep RL algorithms and representations, one of 15 directions in Deep reinforcement learning methods. Builds on 27 of its 40 references that are in the map. First posted on arXiv 22 Apr 2026.

Builds on

27 of 40 references.

Closest ancestors · 5

Same direction, newest first

  1. 2025When Does Neuroevolution Outcompete Reinforcement Learning in Transfer Learning Tasks?28 May
  2. 2025SPEQ: Offline Stabilization Phases for Efficient Q-Learning in High Update-To-Data Ratio Reinforcement Learning15 Jan
  3. 2024Beyond The Rainbow: High Performance Deep Reinforcement Learning on a Desktop PC6 Nov
  4. 2024Bigger, Regularized, Optimistic: scaling for compute and sample-efficient continuous control25 May
  5. 2024Dissecting Deep RL with High Update Ratios: Combatting Value Divergence9 Mar

Other references · 22

  1. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 7,005
  2. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,417
  3. 2017Improved Training of Wasserstein GANscited by 2,235
  4. 2015Prioritized Experience Replaycited by 1,099
  5. 2017Rainbow: Combining Improvements in Deep Reinforcement Learningcited by 786
  6. 2017A Distributional Perspective on Reinforcement Learningcited by 765
  7. 2017Distributional Reinforcement Learning with Quantile Regressioncited by 474
  8. 2021Brax -- A Differentiable Physics Engine for Large Scale Rigid Body Simulationcited by 407
  9. 2021Randomized Ensembled Double Q-Learning: Learning Fast Without a Modelcited by 288
  10. 2020Revisiting Fundamentals of Experience Replaycited by 175
  11. 2017The Shattered Gradients Problem: If resnets are the answer, then what is the question?cited by 157
  12. 2023Bigger, Better, Faster: Human-level Atari with human-level efficiencycited by 147
10 more
  1. 2016Using Centroidal Voronoi Tessellations to Scale Up the Multi-dimensional Archive of Phenotypic Elites Algorithmcited by 66
  2. 2021Spectral Normalisation for Deep Reinforcement Learning: an Optimisation Perspectivecited by 65
  3. 2020Diversity Policy Gradient for Sample Efficient Quality-Diversity Optimizationcited by 54
  4. 2019A Comparative Analysis of Expected and Distributional Reinforcement Learningcited by 47
  5. 2020D2RL: Deep Dense Architectures in Reinforcement Learningcited by 42
  6. 2019Distributional Policy Optimization: An Alternative Approach for Continuous Controlcited by 22
  7. 2023QDax: A Library for Quality-Diversity and Population-based Algorithms with Hardware Accelerationcited by 17
  8. 2025Dominated Novelty Search: Rethinking Local Competition in Quality-Diversitycited by 8
  9. 2023Understanding the Synergies between Quality-Diversity and Deep Reinforcement Learningcited by 6
  10. 2025Multi-Objective Quality-Diversity in Unstructured and Unbounded Spacescited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024Quality-Diversity Actor-Critic: Learning High-Performing and Diverse Behaviors via Value and Successor Features Critics
  2. 2026Learning Hierarchical Skill Policies with Offline Quality-Diversity Reinforcement Learning
  3. 2026Structure-Conditioned Actor-Critic Branches for Quality-Diversity Reinforcement Learning
  4. 2023Quality Diversity through Human Feedback: Towards Open-Ended Diversity-Driven Optimization
  5. 2024Quality-Diversity Algorithms Can Provably Be Helpful for Optimization
  6. 2026Behavioral Foundation Models for Quality Diversity
  7. 2025Overcoming Deceptiveness in Fitness Optimization with Unsupervised Quality-Diversity ↗
  8. 2024Towards Adapting Reinforcement Learning Agents to New Tasks: Insights from Q-Values