# Generating Behaviorally Diverse Policies with Latent Diffusion Models

arXiv: https://arxiv.org/abs/2305.18738 · HTML: https://mlsanity.com/paper/2305.18738/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- Placed in Diffusion and flow policies for robotics, one of 14 directions in Diffusion models.
- Builds on 28 of its 34 references that are in the map.
- First posted on arXiv 30 May 2023.

## Placed in

- Diffusion models / [Diffusion and flow policies for robotics](https://mlsanity.com/diffusion/diffusion-and-flow-policies-for-robotics/)

## Builds on (most cited first, 28 of 28)

- [Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2006.11239/) (2020)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [Score-Based Generative Modeling through Stochastic Differential Equations](https://mlsanity.com/paper/2011.13456/) (2020)
- [Diffusion Models Beat GANs on Image Synthesis](https://mlsanity.com/paper/2105.05233/) (2021)
- [Elucidating the Design Space of Diffusion-Based Generative Models](https://mlsanity.com/paper/2206.00364/) (2022)
- [DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Steps](https://mlsanity.com/paper/2206.00927/) (2022)
- [Decision Transformer: Reinforcement Learning via Sequence Modeling](https://mlsanity.com/paper/2106.01345/) (2021)
- [HyperNetworks](https://mlsanity.com/paper/1609.09106/) (2016)
- [Illuminating search spaces by mapping elites](https://mlsanity.com/paper/1504.04909/) (2015)
- [Brax -- A Differentiable Physics Engine for Large Scale Rigid Body Simulation](https://mlsanity.com/paper/2106.13281/) (2021)
- [Continual learning with hypernetworks](https://mlsanity.com/paper/1906.00695/) (2019)
- [Go-Explore: a New Approach for Hard-Exploration Problems](https://mlsanity.com/paper/1901.10995/) (2019)
- [Robots that can adapt like animals](https://mlsanity.com/paper/1407.3501/) (2014)
- [Graph HyperNetworks for Neural Architecture Search](https://mlsanity.com/paper/1810.05749/) (2018)
- [Parameter Prediction for Unseen Deep Architectures](https://mlsanity.com/paper/2110.13100/) (2021)
- [Differentiable Quality Diversity](https://mlsanity.com/paper/2106.03894/) (2021)
- [Permutation Equivariant Neural Functionals](https://mlsanity.com/paper/2302.14040/) (2023)
- [Using Centroidal Voronoi Tessellations to Scale Up the Multi-dimensional Archive of Phenotypic Elites Algorithm](https://mlsanity.com/paper/1610.05729/) (2016)
- [Diversity Policy Gradient for Sample Efficient Quality-Diversity Optimization](https://mlsanity.com/paper/2006.08505/) (2020)
- [Discovering the Elite Hypervolume by Leveraging Interspecies Correlation](https://mlsanity.com/paper/1804.03906/) (2018)
- [Deep Surrogate Assisted Generation of Environments](https://mlsanity.com/paper/2206.04199/) (2022)
- [Approximating Gradients for Differentiable Quality Diversity in Reinforcement Learning](https://mlsanity.com/paper/2202.03666/) (2022)
- [Policy Manifold Search: Exploring the Manifold Hypothesis for Diversity-based Neuroevolution](https://mlsanity.com/paper/2104.13424/) (2021)
- [MAP-Elites with Descriptor-Conditioned Gradients and Archive Distillation into a Single Policy](https://mlsanity.com/paper/2303.03832/) (2023)
- [Proximal Policy Gradient Arborescence for Quality Diversity Reinforcement Learning](https://mlsanity.com/paper/2305.13795/) (2023)
- [The Quality-Diversity Transformer: Generating Behavior-Conditioned Trajectories with Decision Transformers](https://mlsanity.com/paper/2303.16207/) (2023)
- [Efficiently Learning Small Policies for Locomotion and Manipulation](https://mlsanity.com/paper/2210.00140/) (2022)

## Built on by (most cited first, 19 of 19)

- [Diffusion Models for Reinforcement Learning: A Survey](https://mlsanity.com/paper/2311.01223/) (2023)
- [AlignDiff: Aligning Diverse Human Preferences via Behavior-Customisable Diffusion Model](https://mlsanity.com/paper/2310.02054/) (2023)
- [Diffusion Meets DAgger: Supercharging Eye-in-hand Imitation Learning](https://mlsanity.com/paper/2402.17768/) (2024)
- [Efficient Diffusion Models: A Survey](https://mlsanity.com/paper/2502.06805/) (2025)
- [Generative Modeling of Weights: Generalization or Memorization?](https://mlsanity.com/paper/2506.07998/) (2025)
- [Learning from Random Demonstrations: Offline Reinforcement Learning with Importance-Sampled Diffusion Models](https://mlsanity.com/paper/2405.19878/) (2024)
- [The Duality of Generative AI and Reinforcement Learning in Robotics: A Review](https://mlsanity.com/paper/2410.16411/) (2024)
- [HyperPPO: A scalable method for finding small policies for robotic control](https://mlsanity.com/paper/2309.16663/) (2023)
- [WARPD: World model Assisted Reactive Policy Diffusion](https://mlsanity.com/paper/2410.14040/) (2024)
- [Dynamics-aware Diffusion Models for Planning and Control](https://mlsanity.com/paper/2504.00236/) (2025)
- [From Parameters to Behaviors: Unsupervised Compression of the Policy Space](https://mlsanity.com/paper/2509.22566/) (2025)
- [Learning Intractable Multimodal Policies with Reparameterization and Diversity Regularization](https://mlsanity.com/paper/2511.01374/) (2025)
- [Denoising Diffusion-Based Control of Nonlinear Systems](https://mlsanity.com/paper/2402.02297/) (2024)
- [Diffusion Models for Reinforcement Learning: Foundations, Taxonomy, and Development](https://mlsanity.com/paper/2510.12253/) (2025)
- [Quality-Diversity with Limited Resources](https://mlsanity.com/paper/2406.03731/) (2024)
- [Discount Model Search for Quality Diversity Optimization in High-Dimensional Measure Spaces](https://mlsanity.com/paper/2601.01082/) (2026)
- [Unsupervised Behavioral Compression: Learning Low-Dimensional Policy Manifolds through State-Occupancy Matching](https://mlsanity.com/paper/2603.27044/) (2026)
- [TeNet: Text-to-Network for Compact Policy Synthesis](https://mlsanity.com/paper/2601.15912/) (2026)
- [Modeling and Understanding Neural Weights of Humanoid Policies with Diffusion](https://mlsanity.com/paper/2610.20606/) (2026)

## Abstract (arXiv)

Recent progress in Quality Diversity Reinforcement Learning (QD-RL) has enabled learning a collection of behaviorally diverse, high performing policies. However, these methods typically involve storing thousands of policies, which results in high space-complexity and poor scaling to additional behaviors. Condensing the archive into a single model while retaining the performance and coverage of the original collection of policies has proved challenging. In this work, we propose using diffusion models to distill the archive into a single generative model over policy parameters. We show that our method achieves a compression ratio of 13x while recovering 98% of the original rewards and 89% of the original coverage. Further, the conditioning mechanism of diffusion models allows for flexibly selecting and sequencing behaviors, including using language. Project website: https://sites.google.com/view/policydiffusion/home