mlsanityData to 7 Oct 2026

Boosting Continuous Control with Consistency Policy

arXiv 2310.06343 ↗First posted 10 Oct 2023cs.LGcited by 36

On the map: Diffusion models → Diffusion and flow policies for robotics

also in 1 more
Abstract (arXiv)

Due to its training stability and strong expression, the diffusion model has attracted considerable attention in offline reinforcement learning. However, several challenges have also come with it: 1) The demand for a large number of diffusion steps makes the diffusion-model-based methods time inefficient and limits their applications in real-time control; 2) How to achieve policy improvement with accurate guidance for diffusion model-based policy is still an open problem. Inspired by the consistency model, we propose a novel time-efficiency method named Consistency Policy with Q-Learning (CPQL), which derives action from noise by a single step. By establishing a mapping from the reverse diffusion trajectories to the desired policy, we simultaneously address the issues of time efficiency and inaccurate guidance when updating diffusion model-based policy with the learned Q-function. We demonstrate that CPQL can achieve policy improvement with accurate guidance for offline reinforcement learning, and can be seamlessly extended for online RL tasks. Experimental results indicate that CPQL achieves new state-of-the-art performance on 11 offline and 21 online tasks, significantly improving inference speed by nearly 45 times compared to Diffusion-QL. We will release our code later.

Key facts. Cited by 36 papers in the map (data through 7 Oct 2026). Placed in Diffusion and flow policies for robotics, one of 14 directions in Diffusion models. Builds on 47 of its 58 references that are in the map. First posted on arXiv 10 Oct 2023.

Builds on

47 of 58 references.

Closest ancestors · 15

Same direction, newest first

  1. 2023Crossway Diffusion: Improving Diffusion-based Visuomotor Policy via Self-supervised Learning4 Jul
  2. 2023Efficient Diffusion Policies for Offline Reinforcement Learning31 May
  3. 2023Policy Representation via Diffusion Probability Model for Reinforcement Learning22 May
  4. 2023Contrastive Energy Prediction for Exact Energy-Guided Diffusion Sampling in Offline Reinforcement Learning25 Apr
  5. 2023IDQL: Implicit Q-Learning as an Actor-Critic Method with Diffusion Policies20 Apr
  6. 2023Goal-Conditioned Imitation Learning using Score-based Diffusion Policies5 Apr
  7. 2023Trace and Pace: Controllable Pedestrian Animation via Guided Trajectory Diffusion4 Apr
  8. 2023EDGI: Equivariant Diffusion for Planning with Embodied Agents22 Mar
  9. 2023Diffusion Policy: Visuomotor Policy Learning via Action Diffusion7 Mar
  10. 2023Diffusion Model-Augmented Behavioral Cloning26 Feb
  11. 2023Imitating Human Behaviour with Diffusion Models25 Jan
  12. 2022Is Conditional Generative Modeling all you need for Decision-Making?28 Nov
3 more
  1. 2022Guided Conditional Diffusion for Controllable Traffic Simulation31 Oct
  2. 2022DALL-E-Bot: Introducing Web-Scale Diffusion Models to Robotics5 Oct
  3. 2022Diffusion Policies as an Expressive Policy Class for Offline Reinforcement Learning12 Aug

Other references · 32

  1. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 8,964
  4. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  5. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  6. 2015Continuous control with deep reinforcement learningcited by 3,279
  7. 2022Elucidating the Design Space of Diffusion-Based Generative Modelscited by 2,994
  8. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
  9. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,873
  10. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,777
  11. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,406
  12. 2021Offline Reinforcement Learning with Implicit Q-Learningcited by 1,245
20 more
  1. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
  2. 2023Mastering Diverse Domains through World Modelscited by 1,193
  3. 2022Planning with Diffusion for Flexible Behavior Synthesiscited by 1,023
  4. 2021A Minimalist Approach to Offline Reinforcement Learningcited by 882
  5. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 833
  6. 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 750
  7. 2017A Distributional Perspective on Reinforcement Learningcited by 743
  8. 2019Behavior Regularized Offline Reinforcement Learningcited by 654
  9. 2020dm_control: Software and Tasks for Continuous Controlcited by 368
  10. 2018Maximum a Posteriori Policy Optimisationcited by 318
  11. 2018Distributed Distributional Deterministic Policy Gradientscited by 195
  12. 2019AlgaeDICE: Policy Gradient from Arbitrary Experiencecited by 187
  13. 2022Offline Reinforcement Learning via High-Fidelity Generative Behavior Modelingcited by 177
  14. 2023Offline RL with No OOD Actions: In-Sample Learning via Implicit Value Regularizationcited by 114
  15. 2023Synthetic Experience Replaycited by 100
  16. 2020A Distributional View on Multi-Objective Policy Optimizationcited by 51
  17. 2021Offline Reinforcement Learning with Value-based Episodic Memorycited by 36
  18. 2023The In-Sample Softmax for Offline Reinforcement Learningcited by 33
  19. 2022Know Your Boundaries: The Necessity of Explicit Behavioral Cloning in Offline RLcited by 21
  20. 2023Can Pre-Trained Text-to-Image Models Generate Visual Goals for Reinforcement Learning?cited by 20

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 9; 2025: 14; 2026: 12.

36 papers.

Sort:

By area:

  1. 2024Diffusion Policy Policy Optimizationcited by 255
  2. 2024Consistency Policy: Accelerated Visuomotor Policies via Consistency Distillationcited by 159
  3. 2025ConRFT: A Reinforced Fine-tuning Method for VLA Models via Consistency Policycited by 139
  4. 2024Diffusion-based Reinforcement Learning via Q-weighted Variational Policy Optimizationcited by 99
  5. 2024FlowPolicy: Enabling Fast and Robust 3D Flow-based Policy via Consistency Flow Matching for Robot Manipulationcited by 99
  6. 2023Diffusion Models for Reinforcement Learning: A Surveycited by 84
  7. 2024Diffusion Actor-Critic with Entropy Regulatorcited by 81
  8. 2024ManiCM: Real-time 3D Diffusion Policy via Consistency Model for Robotic Manipulationcited by 57
  9. 2025Robo-Dopamine: General Process Reward Modeling for High-Precision Robotic Manipulationcited by 37
  10. 2025Maximum Entropy Reinforcement Learning with Diffusion Policycited by 29
  11. 2024Deep Generative Models for Offline Policy Learning: Tutorial, Survey, and Perspectives on Future Directionscited by 20
  12. 2025GenPO: Generative Diffusion Models Meet On-Policy Reinforcement Learningcited by 16
24 more
  1. 2025Enhanced DACER Algorithm with High Diffusion Efficiencycited by 13
  2. 2024Generalizing Consistency Policy to Visual RL with Prioritized Proximal Experience Regularizationcited by 11
  3. 2026Towards Long-Lived Robots: Continual Learning VLA Models via Reinforcement Fine-Tuningcited by 11
  4. 2025TeLoGraF: Temporal Logic Planning via Graph-encoded Flow Matchingcited by 9
  5. 2025TeViR: Text-to-Video Reward with Diffusion Models for Efficient Reinforcement Learningcited by 7
  6. 2025Survey of Vision-Language-Action Models for Embodied Manipulationcited by 7
  7. 2026One-Step Flow Policy: Self-Distillation for Fast Visuomotor Policiescited by 5
  8. 2026Scalable Maximum Entropy Reinforcement Learning for Diffusion Policies via Adjoint Matchingcited by 4
  9. 2025Real-Time Diffusion Policies for Games: Enhancing Consistency Policies with Q-Ensemblescited by 3
  10. 2025Accelerating Diffusion Planners in Offline RL via Reward-Aware Consistency Trajectory Distillationcited by 3
  11. 2025Learning Intractable Multimodal Policies with Reparameterization and Diversity Regularizationcited by 3
  12. 2025Dual-Actor Fine-Tuning of VLA Models: A Talk-and-Tweak Human-in-the-Loop Approachcited by 2
  13. 2026A Review of Online Diffusion Policy RL Algorithms for Scalable Robotic Controlcited by 2
  14. 2026Scaling Sim-to-Real VLA Reinforcement Learning with Generative 3D Worldscited by 2
  15. 2024Diffusion-based Episodes Augmentation for Offline Multi-Agent Reinforcement Learningcited by 1
  16. 2026Real-Time Generative Policy via Langevin-Guided Flow Matching for Autonomous Drivingcited by 1
  17. 2026Saliency-Guided Representation with Consistency Policy Learning for Visual Unsupervised Reinforcement Learningcited by 1
  18. 2026Mean Flow Policy Optimizationcited by 1
  19. 2026GenPO++: Generative Policy Optimization with Jacobian-free Likelihood Ratioscited by 1
  20. 2026StructRL: Structured Action-Space Exploration for Flow-Based VLAscited by 1
  21. 2025Mini Diffuser: Fast Multi-task Diffusion Policy Training Using Two-level Mini-batchescited by 0
  22. 2025D2 Actor Critic: Diffusion Actor Meets Distributional Criticcited by 0
  23. 2026ConsistencyPlanner: Real-time Planning with Fast-Sampling Consistency Modelscited by 0
  24. 2026Simple Actors and Deep Critics for Scalable Reinforcement Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Diffusion Policies creating a Trust Region for Offline Reinforcement Learning
  2. 2025Continuous Q-Score Matching: Diffusion Guided Reinforcement Learning for Continuous-Time Control
  3. 2025One-Step Flow Q-Learning: Addressing the Diffusion Policy Bottleneck in Offline Reinforcement Learning
  4. 2026Drift Q-Learning
  5. 2024ACL-QL: Adaptive Conservative Level in Q-Learning for Offline Reinforcement Learning
  6. 2026Noisy-Space Policy Gradient for Diffusion Policies in Offline Reinforcement Learning
  7. 2026Diffusion Policy with Behavioral Advantage Correction for Offline Reinforcement Learning
  8. 2024Diffusion Actor-Critic: Formulating Constrained Policy Iteration as Diffusion Noise Regression for Offline Reinforcement Learning