mlsanityData to 14 Oct 2026

Reasoning with Latent Diffusion in Offline Reinforcement Learning

arXiv 2309.06599 ↗First posted 12 Sep 2023cs.LGcited by 33

On the map: Diffusion models → Diffusion and flow policies for robotics

also in 1 more
Abstract (arXiv)

Offline reinforcement learning (RL) holds promise as a means to learn high-reward policies from a static dataset, without the need for further environment interactions. However, a key challenge in offline RL lies in effectively stitching portions of suboptimal trajectories from the static dataset while avoiding extrapolation errors arising due to a lack of support in the dataset. Existing approaches use conservative methods that are tricky to tune and struggle with multi-modal data (as we show) or rely on noisy Monte Carlo return-to-go samples for reward conditioning. In this work, we propose a novel approach that leverages the expressiveness of latent diffusion to model in-support trajectory sequences as compressed latent skills. This facilitates learning a Q-function while avoiding extrapolation error via batch-constraining. The latent space is also expressive and gracefully copes with multi-modal data. We show that the learned temporally-abstract latent space encodes richer task-specific information for offline RL tasks as compared to raw state-actions. This improves credit assignment and facilitates faster reward propagation during Q-learning. Our method demonstrates state-of-the-art performance on the D4RL benchmarks, particularly excelling in long-horizon, sparse-reward tasks.

Key facts. Cited by 33 papers in the map (data through 14 Oct 2026). Placed in Diffusion and flow policies for robotics, one of 14 directions in Diffusion models. Builds on 41 of its 47 references that are in the map. First posted on arXiv 12 Sep 2023.

Builds on

41 of 47 references.

Closest ancestors · 2

Same direction, newest first

  1. 2023Imitating Human Behaviour with Diffusion Models25 Jan
  2. 2022Is Conditional Generative Modeling all you need for Decision-Making?28 Nov

Other references · 39

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  4. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,713
  5. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  6. 2020Denoising Diffusion Implicit Modelscited by 9,781
  7. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,058
  8. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,585
  9. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,355
  10. 2022Classifier-Free Diffusion Guidancecited by 5,847
  11. 2019Generative Modeling by Estimating Gradients of the Data Distributioncited by 4,101
  12. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,417
27 more
  1. 2022DPM-Solver: A Fast ODE Solver for Diffusion Probabilistic Model Sampling in Around 10 Stepscited by 1,979
  2. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,920
  3. 2023Consistency Modelscited by 1,822
  4. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,809
  5. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,546
  6. 2021Image Super-Resolution via Iterative Refinementcited by 1,511
  7. 2022RePaint: Inpainting using Denoising Diffusion Probabilistic Modelscited by 1,430
  8. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,427
  9. 2020DiffWave: A Versatile Diffusion Model for Audio Synthesiscited by 1,354
  10. 2021Offline Reinforcement Learning with Implicit Q-Learningcited by 1,293
  11. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,241
  12. 2015Prioritized Experience Replaycited by 1,099
  13. 2022Planning with Diffusion for Flexible Behavior Synthesiscited by 1,061
  14. 2021Offline Reinforcement Learning as One Big Sequence Modeling Problemcited by 644
  15. 2021Tackling the Generative Learning Trilemma with Denoising Diffusion GANscited by 516
  16. 2023Shap-E: Generating Conditional 3D Implicit Functionscited by 385
  17. 2023Efficient Diffusion Training via Min-SNR Weighting Strategycited by 246
  18. 2021Score-Based Generative Modeling with Critically-Damped Langevin Diffusioncited by 233
  19. 2020Accelerating Reinforcement Learning with Learned Skill Priorscited by 223
  20. 2020OPAL: Offline Primitive Discovery for Accelerating Offline Reinforcement Learningcited by 166
  21. 2020PLAS: Latent Action Space for Offline Reinforcement Learningcited by 133
  22. 2020EMaQ: Expected-Max Q-Learning Operator for Simple Yet Effective Offline and Online RLcited by 115
  23. 2019Training Agents using Upside-Down Reinforcement Learningcited by 107
  24. 2019Reward-Conditioned Policiescited by 102
  25. 2021Temporal Abstraction in Reinforcement Learning with the Successor Representationcited by 54
  26. 2022Flow to Control: Offline Reinforcement Learning with Lossless Primitive Discoverycited by 12
  27. 2021Implicitly Regularized RL with Implicit Q-Valuescited by 9

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 3; 2024: 11; 2025: 14; 2026: 5.

33 papers.

Sort:

By area:

  1. 2024Diffusion Policy Policy Optimizationcited by 280
  2. 2025Flow Q-Learningcited by 184
  3. 2025Steering Your Diffusion Policy with Latent Space Reinforcement Learningcited by 169
  4. 2023Consistency Models as a Rich and Efficient Policy Class for Reinforcement Learningcited by 100
  5. 2023Diffusion Models for Reinforcement Learning: A Surveycited by 91
  6. 2024Learning Multimodal Behaviors from Scratch with Diffusion Policy Gradientcited by 51
  7. 2025VideoWorld: Exploring Knowledge Learning from Unlabeled Videoscited by 43
  8. 2025Generative Trajectory Stitching through Diffusion Compositioncited by 41
  9. 2025Diffusion Models for Robotic Manipulation: A Surveycited by 41
  10. 2024Scaling Diffusion Policy in Transformer to 1 Billion Parameters for Robotic Manipulationcited by 38
  11. 2024Deep Generative Models for Offline Policy Learning: Tutorial, Survey, and Perspectives on Future Directionscited by 20
  12. 2024Discrete Policy: Learning Disentangled Action Space for Multi-Task Robotic Manipulationcited by 19
21 more
  1. 2025Normalizing Flows are Capable Models for Continuous Controlcited by 14
  2. 2025Guided Flow Policy: Learning from High-Value Actions in Offline Reinforcement Learningcited by 9
  3. 2025Uncertainty Comes for Free: Human-in-the-Loop Policies with Diffusion Modelscited by 8
  4. 2025Unleashing Flow Policies with Distributional Criticscited by 8
  5. 2024The Duality of Generative AI and Reinforcement Learning in Robotics: A Reviewcited by 6
  6. 2023iDesigner: A High-Resolution and Complex-Prompt Following Text-to-Image Diffusion Model for Interior Designcited by 4
  7. 2024DiffLight: A Partial Rewards Conditioned Diffusion Model for Traffic Signal Control with Missing Datacited by 4
  8. 2024Enhancing Exploration with Diffusion Policies in Hybrid Off-Policy RL: Application to Non-Prehensile Manipulationcited by 3
  9. 2025Pick-and-place Manipulation Across Grippers Without Retraining: A Learning-optimization Diffusion Policy Approachcited by 3
  10. 2024DIAR: Diffusion-model-guided Implicit Q-learning with Adaptive Revaluationcited by 2
  11. 2025Diffusion-based Planning with Learned Viability Filterscited by 2
  12. 2025Inference-stage Adaptation-projection Strategy Adapts Diffusion Policy to Cross-manipulators Scenarioscited by 2
  13. 2025Diffusion Models for Reinforcement Learning: Foundations, Taxonomy, and Developmentcited by 2
  14. 2026Fisher Decorator: Refining Flow Policy via a Local Transport Mapcited by 2
  15. 2026Towards Efficient and Expressive Offline RL via Flow-Anchored Noise-conditioned Q-Learningcited by 2
  16. 2024Enhanced Safety in Autonomous Driving: Integrating Latent State Diffusion Model for End-to-End Navigationcited by 1
  17. 2024Diffusion-Based Offline RL for Improved Decision-Making in Augmented ARC Taskcited by 1
  18. 2025Offline Reinforcement Learning with Discrete Diffusion Skillscited by 1
  19. 2026Offline Discovery of Interpretable Skills from Multi-Task Trajectoriescited by 1
  20. 2026Improving Diffusion Planners by Self-Supervised Action Gating with Energiescited by 1
  21. 2026Decoupled Guidance Diffusion for Adaptive Offline Safe Reinforcement Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Latent Policy Steering through One-Step Flow Policies
  2. 2026LaDi-RL: Latent Diffusion Reasoning Prevents Entropy Collapse in Reinforcement Learning
  3. 2025Learning from Reward-Free Offline Data: A Case for Planning with Latent Dynamics Models
  4. 2026Efficient and Uncertainty-Aware Diffusion Framework for Offline-to-Online Reinforcement Learning
  5. 2023Efficient Planning with Latent Diffusion
  6. 2026From Suboptimal to Optimal: One-Step Latent Flow Policies for Offline Reinforcement Learning
  7. 2024Preferred-Action-Optimized Diffusion Policies for Offline Reinforcement Learning
  8. 2026Pessimistic Latent Task-aware Optimization for Robust Offline Meta-Reinforcement Learning