mlsanityData to 14 Oct 2026

Generalizing Consistency Policy to Visual RL with Prioritized Proximal Experience Regularization

arXiv 2410.00051 ↗First posted 28 Sep 2024cs.LG cs.AI cs.CVcited by 11

On the map: Diffusion models → Diffusion and flow policies for robotics

also in 1 more
Abstract (arXiv)

With high-dimensional state spaces, visual reinforcement learning (RL) faces significant challenges in exploitation and exploration, resulting in low sample efficiency and training stability. As a time-efficient diffusion model, although consistency models have been validated in online state-based RL, it is still an open question whether it can be extended to visual RL. In this paper, we investigate the impact of non-stationary distribution and the actor-critic framework on consistency policy in online RL, and find that consistency policy was unstable during the training, especially in visual RL with the high-dimensional state space. To this end, we suggest sample-based entropy regularization to stabilize the policy training, and propose a consistency policy with prioritized proximal experience regularization (CP3ER) to improve sample efficiency. CP3ER achieves new state-of-the-art (SOTA) performance in 21 tasks across DeepMind control suite and Meta-world. To our knowledge, CP3ER is the first method to apply diffusion/consistency models to visual RL and demonstrates the potential of consistency models in visual RL. More visualization results are available at https://jzndd.github.io/CP3ER-Page/.

Key facts. Cited by 11 papers in the map (data through 14 Oct 2026). Placed in Diffusion and flow policies for robotics, one of 14 directions in Diffusion models. Builds on 51 of its 62 references that are in the map. First posted on arXiv 28 Sep 2024.

Builds on

51 of 62 references.

Closest ancestors · 15

Same direction, newest first

  1. 2023Learning a Diffusion Model Policy from Rewards via Q-Score Matching18 Dec
  2. 2023World Models via Policy-Guided Trajectory Diffusion13 Dec
  3. 2023Diffusion Models for Reinforcement Learning: A Survey2 Nov
  4. 2023Boosting Continuous Control with Consistency Policy10 Oct
  5. 2023Consistency Models as a Rich and Efficient Policy Class for Reinforcement Learning29 Sep
  6. 2023Motion Planning Diffusion: Learning and Planning of Robot Motions with Diffusion Models3 Aug
  7. 2023Efficient Diffusion Policies for Offline Reinforcement Learning31 May
  8. 2023Diffusion Model is an Effective Planner and Data Synthesizer for Multi-Task Reinforcement Learning29 May
  9. 2023Policy Representation via Diffusion Probability Model for Reinforcement Learning22 May
  10. 2023Contrastive Energy Prediction for Exact Energy-Guided Diffusion Sampling in Offline Reinforcement Learning25 Apr
  11. 2023IDQL: Implicit Q-Learning as an Actor-Critic Method with Diffusion Policies20 Apr
  12. 2023Diffusion Policy: Visuomotor Policy Learning via Action Diffusion7 Mar
3 more
  1. 2023AdaptDiffuser: Diffusion Models as Adaptive Self-evolving Planners3 Feb
  2. 2023Imitating Human Behaviour with Diffusion Models25 Jan
  3. 2022Diffusion Policies as an Expressive Policy Class for Offline Reinforcement Learning12 Aug

Other references · 36

  1. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  2. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  3. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  4. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,585
  5. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,355
  6. 2023Stable Video Diffusion: Scaling Latent Video Diffusion Models to Large Datasetscited by 2,581
  7. 2022Video Diffusion Modelscited by 2,305
  8. 2023Consistency Modelscited by 1,822
  9. 2019Mastering Atari, Go, Chess and Shogi by Planning with a Learned Modelcited by 1,353
  10. 2019Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learningcited by 1,204
  11. 2022Planning with Diffusion for Flexible Behavior Synthesiscited by 1,061
  12. 2018DeepMind Control Suitecited by 956
24 more
  1. 2020CURL: Contrastive Unsupervised Representations for Reinforcement Learningcited by 760
  2. 2017Reinforcement Learning with Deep Energy-Based Policiescited by 669
  3. 2020Learning Quadrupedal Locomotion over Challenging Terraincited by 604
  4. 2020Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixelscited by 585
  5. 2019Learning agile and dynamic motor skills for legged robotscited by 504
  6. 2020Reinforcement Learning with Augmented Datacited by 468
  7. 2023Learning Interactive Real-World Simulatorscited by 458
  8. 2021Mastering Visual Continuous Control: Improved Data-Augmented Reinforcement Learningcited by 372
  9. 2020Data-Efficient Reinforcement Learning with Self-Predictive Representationscited by 318
  10. 2020Decoupling Representation Learning from Reinforcement Learningcited by 269
  11. 2022Offline Reinforcement Learning via High-Fidelity Generative Behavior Modelingcited by 182
  12. 2023The Dormant Neuron Phenomenon in Deep Reinforcement Learningcited by 174
  13. 2023Synthetic Experience Replaycited by 101
  14. 2020Multi-expert learning of adaptive legged locomotioncited by 67
  15. 2023TACO: Temporal Latent Action-Driven Contrastive Loss for Visual Reinforcement Learningcited by 57
  16. 2023DrM: Mastering Visual Reinforcement Learning through Dormant Ratio Minimizationcited by 47
  17. 2024In value-based deep reinforcement learning, a pruned network is a good networkcited by 39
  18. 2020Deep Reinforcement Learning based Automatic Exploration for Navigation in Unknown Environmentcited by 34
  19. 2022Stabilizing Off-Policy Deep Reinforcement Learning from Pixelscited by 32
  20. 2022Know Your Boundaries: The Necessity of Explicit Behavioral Cloning in Offline RLcited by 21
  21. 2023Reparameterized Policy Learning for Multimodal Trajectory Optimizationcited by 20
  22. 2022Revisiting Gaussian mixture critics in off-policy reinforcement learning: a sample-based approachcited by 6
  23. 2023NeuronsMAE: A Novel Multi-Agent Reinforcement Learning Environment for Cooperative and Competitive Multi-Robot Taskscited by 2
  24. 2024Pretrained Visual Representations in Reinforcement Learningcited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 7; 2026: 4.

11 papers.

  1. 2025ConRFT: A Reinforced Fine-tuning Method for VLA Models via Consistency Policycited by 154
  2. 2025RL-100: Performant Robotic Manipulation with Real-World Reinforcement Learningcited by 59
  3. 2025World4RL: Diffusion World Models for Policy Refinement with Reinforcement Learning for Robotic Manipulationcited by 35
  4. 2026Towards Long-Lived Robots: Continual Learning VLA Models via Reinforcement Fine-Tuningcited by 13
  5. 2025Survey of Vision-Language-Action Models for Embodied Manipulationcited by 7
  6. 2025Disentangled World Models: Learning to Transfer Semantic Knowledge from Distracting Videos for Reinforcement Learningcited by 6
  7. 2025Accelerating Diffusion Planners in Offline RL via Reward-Aware Consistency Trajectory Distillationcited by 3
  8. 2025CLAR: Learning 3D Representations for Robotic Manipulation by Fusing Masked Reconstruction with Multi-Level Contrastive Alignmentcited by 1
  9. 2026Task-Aware Exploration via a Predictive Bisimulation Metriccited by 1
  10. 2026Saliency-Guided Representation with Consistency Policy Learning for Visual Unsupervised Reinforcement Learningcited by 1
  11. 2026ConsistencyPlanner: Real-time Planning with Fast-Sampling Consistency Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026VIGOR: Zero-Shot Visual Generalization via Latent-Space Consistency in Model-Based Reinforcement Learning
  2. 2025Salience-Invariant Consistent Policy Learning for Generalization in Visual Reinforcement Learning
  3. 2022VRL3: A Data-Driven Framework for Visual Deep Reinforcement Learning
  4. 2024The Surprising Ineffectiveness of Pre-Trained Visual Representations for Model-Based Reinforcement Learning
  5. 2026V-Simba: Unleashing the Architectural Potential of RL in Visual Continuous Control
  6. 2023Continual Visual Reinforcement Learning with A Life-Long World Model
  7. 2024When Should We Prefer State-to-Visual DAgger Over Visual Reinforcement Learning?
  8. 2025One RL to See Them All: Visual Triple Unified Reinforcement Learning