mlsanityData to 14 Oct 2026

Generating Behaviorally Diverse Policies with Latent Diffusion Models

arXiv 2305.18738 ↗First posted 30 May 2023cs.LG cs.AI cs.ROcited by 19

On the map: Diffusion models → Diffusion and flow policies for robotics

Abstract (arXiv)

Recent progress in Quality Diversity Reinforcement Learning (QD-RL) has enabled learning a collection of behaviorally diverse, high performing policies. However, these methods typically involve storing thousands of policies, which results in high space-complexity and poor scaling to additional behaviors. Condensing the archive into a single model while retaining the performance and coverage of the original collection of policies has proved challenging. In this work, we propose using diffusion models to distill the archive into a single generative model over policy parameters. We show that our method achieves a compression ratio of 13x while recovering 98% of the original rewards and 89% of the original coverage. Further, the conditioning mechanism of diffusion models allows for flexibly selecting and sequencing behaviors, including using language. Project website: https://sites.google.com/view/policydiffusion/home

Key facts. Cited by 19 papers in the map (data through 14 Oct 2026). Placed in Diffusion and flow policies for robotics, one of 14 directions in Diffusion models. Builds on 28 of its 34 references that are in the map. First posted on arXiv 30 May 2023.

Builds on

28 of 34 references.

References · 28

  1. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  2. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  3. 2020Denoising Diffusion Implicit Modelscited by 9,781
  4. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 9,122
  5. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,255
  6. 2022Elucidating the Design Space of Diffusion-Based Generative Modelscited by 3,045
  7. 2022DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Stepscited by 1,979
  8. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,546
  9. 2016HyperNetworkscited by 1,027
  10. 2015Illuminating search spaces by mapping elitescited by 425
  11. 2021Brax -- A Differentiable Physics Engine for Large Scale Rigid Body Simulationcited by 407
  12. 2019Continual learning with hypernetworkscited by 273
16 more
  1. 2019Go-Explore: a New Approach for Hard-Exploration Problemscited by 212
  2. 2014Robots that can adapt like animalscited by 204
  3. 2018Graph HyperNetworks for Neural Architecture Searchcited by 132
  4. 2021Parameter Prediction for Unseen Deep Architecturescited by 89
  5. 2021Differentiable Quality Diversitycited by 72
  6. 2023Permutation Equivariant Neural Functionalscited by 70
  7. 2016Using Centroidal Voronoi Tessellations to Scale Up the Multi-dimensional Archive of Phenotypic Elites Algorithmcited by 66
  8. 2020Diversity Policy Gradient for Sample Efficient Quality-Diversity Optimizationcited by 54
  9. 2018Discovering the Elite Hypervolume by Leveraging Interspecies Correlationcited by 41
  10. 2022Deep Surrogate Assisted Generation of Environmentscited by 40
  11. 2022Approximating Gradients for Differentiable Quality Diversity in Reinforcement Learningcited by 38
  12. 2021Policy Manifold Search: Exploring the Manifold Hypothesis for Diversity-based Neuroevolutioncited by 20
  13. 2023MAP-Elites with Descriptor-Conditioned Gradients and Archive Distillation into a Single Policycited by 20
  14. 2023Proximal Policy Gradient Arborescence for Quality Diversity Reinforcement Learningcited by 20
  15. 2023The Quality-Diversity Transformer: Generating Behavior-Conditioned Trajectories with Decision Transformerscited by 10
  16. 2022Efficiently Learning Small Policies for Locomotion and Manipulationcited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 3; 2024: 6; 2025: 6; 2026: 4.

19 papers.

Sort:

By area:

  1. 2023Diffusion Models for Reinforcement Learning: A Surveycited by 91
  2. 2023AlignDiff: Aligning Diverse Human Preferences via Behavior-Customisable Diffusion Modelcited by 37
  3. 2024Diffusion Meets DAgger: Supercharging Eye-in-hand Imitation Learningcited by 35
  4. 2025Efficient Diffusion Models: A Surveycited by 35
  5. 2025Generative Modeling of Weights: Generalization or Memorization?cited by 14
  6. 2024Learning from Random Demonstrations: Offline Reinforcement Learning with Importance-Sampled Diffusion Modelscited by 8
  7. 2024The Duality of Generative AI and Reinforcement Learning in Robotics: A Reviewcited by 6
  8. 2023HyperPPO: A scalable method for finding small policies for robotic controlcited by 4
  9. 2024WARPD: World model Assisted Reactive Policy Diffusioncited by 4
  10. 2025Dynamics-aware Diffusion Models for Planning and Controlcited by 4
  11. 2025From Parameters to Behaviors: Unsupervised Compression of the Policy Spacecited by 4
  12. 2025Learning Intractable Multimodal Policies with Reparameterization and Diversity Regularizationcited by 3
7 more
  1. 2024Denoising Diffusion-Based Control of Nonlinear Systemscited by 2
  2. 2025Diffusion Models for Reinforcement Learning: Foundations, Taxonomy, and Developmentcited by 2
  3. 2024Quality-Diversity with Limited Resourcescited by 1
  4. 2026Discount Model Search for Quality Diversity Optimization in High-Dimensional Measure Spacescited by 1
  5. 2026Unsupervised Behavioral Compression: Learning Low-Dimensional Policy Manifolds through State-Occupancy Matchingcited by 1
  6. 2026TeNet: Text-to-Network for Compact Policy Synthesiscited by 0
  7. 2026Modeling and Understanding Neural Weights of Humanoid Policies with Diffusioncited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2022Diffusion Policies as an Expressive Policy Class for Offline Reinforcement Learning
  2. 2024Revisiting Generative Policies: A Simpler Reinforcement Learning Algorithmic Perspective
  3. 2026Behavioral Foundation Models for Quality Diversity
  4. 2023Score Regularized Policy Optimization through Diffusion Behavior
  5. 2024Diffusion Policies creating a Trust Region for Offline Reinforcement Learning
  6. 2023Learning a Diffusion Model Policy from Rewards via Q-Score Matching
  7. 2025Diffusion Policies with Value-Conditional Optimization for Offline Reinforcement Learning
  8. 2026Diffusion Policy through Conditional Proximal Policy Optimization