mlsanityData to 14 Oct 2026

Consistency Models as a Rich and Efficient Policy Class for Reinforcement Learning

arXiv 2309.16984 ↗First posted 29 Sep 2023cs.LGcited by 100

On the map: Diffusion models → Diffusion and flow policies for robotics

also in 1 more
Abstract (arXiv)

Score-based generative models like the diffusion model have been testified to be effective in modeling multi-modal data from image generation to reinforcement learning (RL). However, the inference process of diffusion model can be slow, which hinders its usage in RL with iterative sampling. We propose to apply the consistency model as an efficient yet expressive policy representation, namely consistency policy, with an actor-critic style algorithm for three typical RL settings: offline, offline-to-online and online. For offline RL, we demonstrate the expressiveness of generative models as policies from multi-modal data. For offline-to-online RL, the consistency policy is shown to be more computational efficient than diffusion policy, with a comparable performance. For online RL, the consistency policy demonstrates significant speedup and even higher average performances than the diffusion policy.

Key facts. Cited by 100 papers in the map (data through 14 Oct 2026). Placed in Diffusion and flow policies for robotics, one of 14 directions in Diffusion models. Builds on 40 of its 50 references that are in the map. First posted on arXiv 29 Sep 2023.

Builds on

40 of 50 references.

Closest ancestors · 8

Same direction, newest first

  1. 2023Reasoning with Latent Diffusion in Offline Reinforcement Learning12 Sep
  2. 2023Contrastive Energy Prediction for Exact Energy-Guided Diffusion Sampling in Offline Reinforcement Learning25 Apr
  3. 2023IDQL: Implicit Q-Learning as an Actor-Critic Method with Diffusion Policies20 Apr
  4. 2023Goal-Conditioned Imitation Learning using Score-based Diffusion Policies5 Apr
  5. 2023Diffusion Policy: Visuomotor Policy Learning via Action Diffusion7 Mar
  6. 2023Imitating Human Behaviour with Diffusion Models25 Jan
  7. 2022Is Conditional Generative Modeling all you need for Decision-Making?28 Nov
  8. 2022Diffusion Policies as an Expressive Policy Class for Offline Reinforcement Learning12 Aug

Other references · 32

  1. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  2. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 9,122
  3. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,058
  4. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  5. 2022Elucidating the Design Space of Diffusion-Based Generative Modelscited by 3,045
  6. 2016Categorical Reparameterization with Gumbel-Softmaxcited by 2,739
  7. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,417
  8. 2023Consistency Modelscited by 1,822
  9. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,809
  10. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,546
  11. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,427
  12. 2021Offline Reinforcement Learning with Implicit Q-Learningcited by 1,293
20 more
  1. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,241
  2. 2022Planning with Diffusion for Flexible Behavior Synthesiscited by 1,061
  3. 2021A Minimalist Approach to Offline Reinforcement Learningcited by 899
  4. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 840
  5. 2020AWAC: Accelerating Online Reinforcement Learning with Offline Datasetscited by 671
  6. 2019Behavior Regularized Offline Reinforcement Learningcited by 662
  7. 2020MOPO: Model-based Offline Policy Optimizationcited by 636
  8. 2020MOReL : Model-Based Offline Reinforcement Learningcited by 590
  9. 2017A Brief Survey of Deep Reinforcement Learningcited by 483
  10. 2023Efficient Online Reinforcement Learning with Offline Datacited by 363
  11. 2023Cal-QL: Calibrated Offline RL Pre-Training for Efficient Online Fine-Tuningcited by 237
  12. 2017Distilling a Neural Network Into a Soft Decision Treecited by 236
  13. 2021Offline-to-Online Reinforcement Learning via Balanced Replay and Pessimistic Q-Ensemblecited by 214
  14. 2022Hybrid RL: Using Both Offline and Online Data Can Make RL Efficientcited by 161
  15. 2021Offline RL Without Off-Policy Evaluationcited by 151
  16. 2023Extreme Q-Learning: MaxEnt RL without Entropycited by 116
  17. 2021Probabilistic Mixture-of-Experts for Efficient Deep Reinforcement Learningcited by 27
  18. 2022Know Your Boundaries: The Necessity of Explicit Behavioral Cloning in Offline RLcited by 21
  19. 2020CDT: Cascading Decision Trees for Explainable Reinforcement Learningcited by 15
  20. 2023Guiding Online Reinforcement Learning with Action-Free Offline Pretrainingcited by 11

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 18; 2025: 33; 2026: 49.

100 papers.

Sort:

By area:

  1. 2024Diffusion Policy Policy Optimizationcited by 280
  2. 2025Flow Q-Learningcited by 184
  3. 2025Steering Your Diffusion Policy with Latent Space Reinforcement Learningcited by 169
  4. 2024Consistency Policy: Accelerated Visuomotor Policies via Consistency Distillationcited by 166
  5. 2024FlowPolicy: Enabling Fast and Robust 3D Flow-based Policy via Consistency Flow Matching for Robot Manipulationcited by 104
  6. 2025Inference-Time Alignment in Diffusion Models with Reward-Guided Generation: Tutorial and Reviewcited by 88
  7. 2024An Overview of Diffusion Models: Applications, Guided Generation, Statistical Rates and Optimizationcited by 73
  8. 2025A Vision-Language-Action-Critic Model for Robotic Real-World Reinforcement Learningcited by 71
  9. 2024Policy Decorator: Model-Agnostic Online Refinement for Large Policy Modelcited by 70
  10. 2025RL-100: Performant Robotic Manipulation with Real-World Reinforcement Learningcited by 59
  11. 2025DIME:Diffusion-Based Maximum Entropy Reinforcement Learningcited by 58
  12. 2025Diffusion Guidance Is a Controllable Policy Improvement Operatorcited by 58
38 more
  1. 2024ManiCM: Real-time 3D Diffusion Policy via Consistency Model for Robotic Manipulationcited by 57
  2. 2024Learning Multimodal Behaviors from Scratch with Diffusion Policy Gradientcited by 51
  3. 2024Diffusion-DICE: In-Sample Diffusion Guidance for Offline Reinforcement Learningcited by 42
  4. 2025Diffusion Models for Robotic Manipulation: A Surveycited by 41
  5. 2025Flow-Based Policy for Online Reinforcement Learningcited by 37
  6. 2026Q-learning with Adjoint Matchingcited by 37
  7. 2025EXPO: Stable Reinforcement Learning with Expressive Policiescited by 34
  8. 2024Diffusion World Model: Future Modeling Beyond Step-by-Step Rollout for Offline Reinforcement Learningcited by 31
  9. 2025DiWA: Diffusion Policy Adaptation with World Modelscited by 31
  10. 2025Scaling Offline RL via Efficient and Expressive Shortcut Modelscited by 27
  11. 2024Towards a mathematical theory for consistency training in diffusion modelscited by 26
  12. 2025One-Step Flow Policy Mirror Descentcited by 21
  13. 2024Deep Generative Models for Offline Policy Learning: Tutorial, Survey, and Perspectives on Future Directionscited by 20
  14. 2025One-Step Generative Policies with Q-Learning: A Reformulation of MeanFlowcited by 18
  15. 2024Sampling from Energy-based Policies using Diffusioncited by 17
  16. 2026Flow Policy Gradients for Robot Controlcited by 16
  17. 2025Wasserstein Convergence of Score-based Generative Models under Semiconvexity and Discontinuous Gradientscited by 15
  18. 2025Prior-Guided Diffusion Planning for Offline Reinforcement Learningcited by 14
  19. 2025Flow-Based Single-Step Completion for Efficient and Expressive Policy Learningcited by 14
  20. 2025Responsive Noise-Relaying Diffusion Policy: Responsive and Efficient Visuomotor Controlcited by 13
  21. 2024Generalizing Consistency Policy to Visual RL with Prioritized Proximal Experience Regularizationcited by 11
  22. 2025One-Step Flow Q-Learning: Addressing the Diffusion Policy Bottleneck in Offline Reinforcement Learningcited by 11
  23. 2025Multi-agent Coordination via Flow Matchingcited by 10
  24. 2025Guided Flow Policy: Learning from High-Value Actions in Offline Reinforcement Learningcited by 9
  25. 2025Global Search for Optimal Low Thrust Spacecraft Trajectories using Diffusion Models and the Indirect Methodcited by 8
  26. 2025Unleashing Flow Policies with Distributional Criticscited by 8
  27. 2025Predictive Planner for Autonomous Driving with Consistency Modelscited by 7
  28. 2024The Duality of Generative AI and Reinforcement Learning in Robotics: A Reviewcited by 6
  29. 2025Expressive Value Learning for Scalable Offline Reinforcement Learningcited by 6
  30. 2026Flow Actor-Critic for Offline Reinforcement Learningcited by 6
  31. 2026Q-Flow: Stable and Expressive Reinforcement Learning with Flow-Based Policycited by 6
  32. 2026Test-Time Gradient Guidance of Flow Policies in Reinforcement Learningcited by 6
  33. 2024Diffusion Transformer Captures Spatial-Temporal Dependencies: A Theory for Gaussian Process Datacited by 5
  34. 2026DeFlow: Decoupling Manifold Modeling and Value Maximization for Offline Policy Extractioncited by 5
  35. 2026Reinforcement Learning via Value Gradient Flowcited by 5
  36. 2026Scalable Maximum Entropy Reinforcement Learning for Diffusion Policies via Adjoint Matchingcited by 5
  37. 2025Offline Multi-agent Reinforcement Learning via Sequential Score Decompositioncited by 4
  38. 2026ReFORM: Reflected Flows for On-support Offline RL via Noise Manipulationcited by 4

50 most cited of 100 All 100 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Revisiting Generative Policies: A Simpler Reinforcement Learning Algorithmic Perspective
  2. 2024RL for Consistency Models: Faster Reward Guided Text-to-Image Generation
  3. 2026Stabilizing Consistency Training: A Flow Map Analysis and Self-Distillation
  4. 2024Scores as Actions: a framework of fine-tuning diffusion models by continuous-time reinforcement learning
  5. 2025Diffusion Models for Reinforcement Learning: Foundations, Taxonomy, and Development
  6. 2025ROCM: RLHF on consistency models
  7. 2023Boosting Continuous Control with Consistency Policy
  8. 2026Positive-Only Drifting Policy Optimization