mlsanityData to 7 Oct 2026

Contrastive Diffuser: Planning Towards High Return States via Contrastive Learning

arXiv 2402.02772 ↗First posted 5 Feb 2024cs.LGcited by 1

On the map: Offline RL and imitation learning from data → Diffusion and flow policies for offline RL

Abstract (arXiv)

The performance of offline reinforcement learning (RL) is sensitive to the proportion of high-return trajectories in the offline dataset. However, in many simulation environments and real-world scenarios, there are large ratios of low-return trajectories rather than high-return trajectories, which makes learning an efficient policy challenging. In this paper, we propose a method called Contrastive Diffuser (CDiffuser) to make full use of low-return trajectories and improve the performance of offline RL algorithms. Specifically, CDiffuser groups the states of trajectories in the offline dataset into high-return states and low-return states and treats them as positive and negative samples correspondingly. Then, it designs a contrastive mechanism to pull the trajectory of an agent toward high-return states and push them away from low-return states. Through the contrast mechanism, trajectories with low returns can serve as negative examples for policy learning, guiding the agent to avoid areas associated with low returns and achieve better performance. Experiments on 14 commonly used D4RL benchmarks demonstrate the effectiveness of our proposed method. Our code is publicly available at urlhttps://anonymous.4open.science/r/CDiffuser.

Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Placed in Diffusion and flow policies for offline RL, one of 13 directions in Offline RL and imitation learning from data. Builds on 37 of its 45 references that are in the map. First posted on arXiv 5 Feb 2024.

Builds on

37 of 45 references.

Closest ancestors · 5

Same direction, newest first

  1. 2023Score Regularized Policy Optimization through Diffusion Behavior11 Oct
  2. 2023Diffusion Policies for Out-of-Distribution Generalization in Offline Reinforcement Learning10 Jul
  3. 2022Is Conditional Generative Modeling all you need for Decision-Making?28 Nov
  4. 2022Offline Reinforcement Learning via High-Fidelity Generative Behavior Modeling29 Sep
  5. 2022Diffusion Policies as an Expressive Policy Class for Offline Reinforcement Learning12 Aug

Other references · 32

  1. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  2. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  3. 2020Denoising Diffusion Implicit Modelscited by 9,655
  4. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  5. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  6. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  7. 2015FaceNet: A Unified Embedding for Face Recognition and Clusteringcited by 2,833
  8. 2020Supervised Contrastive Learningcited by 2,803
  9. 2023Diffusion Policy: Visuomotor Policy Learning via Action Diffusioncited by 2,730
  10. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,873
  11. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,777
  12. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,528
20 more
  1. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,406
  2. 2022Planning with Diffusion for Flexible Behavior Synthesiscited by 1,023
  3. 2020What Makes for Good Views for Contrastive Learning?cited by 843
  4. 2020CURL: Contrastive Unsupervised Representations for Reinforcement Learningcited by 745
  5. 2021Offline Reinforcement Learning as One Big Sequence Modeling Problemcited by 636
  6. 2020MOPO: Model-based Offline Policy Optimizationcited by 630
  7. 2020Understanding the Behaviour of Contrastive Losscited by 397
  8. 2021Offline Reinforcement Learning with Fisher Divergence Critic Regularizationcited by 237
  9. 2022A Survey on Offline Reinforcement Learning: Taxonomy, Review, and Open Problemscited by 228
  10. 2020What Should Not Be Contrastive in Contrastive Learningcited by 224
  11. 2021Decoupled Contrastive Learningcited by 141
  12. 2021Contrastive Behavioral Similarity Embeddings for Generalization in Reinforcement Learningcited by 123
  13. 2021Contrastive Learning with Stronger Augmentationscited by 100
  14. 2020Human-centric Dialog Training via Offline Reinforcement Learningcited by 92
  15. 2022Robust Task Representations for Offline Meta-Reinforcement Learning via Contrastive Learningcited by 33
  16. 2021Offline Reinforcement Learning for Autonomous Driving with Safety and Exploration Enhancementcited by 28
  17. 2022Contrastive UCB: Provably Efficient Contrastive Self-Supervised Learning in Online Reinforcement Learningcited by 26
  18. 2022Boosting Offline Reinforcement Learning via Data Rebalancingcited by 22
  19. 2022Semi-Markov Offline Reinforcement Learning for Healthcarecited by 18
  20. 2023A General Offline Reinforcement Learning Framework for Interactive Recommendationcited by 18

Built on by

Cited by year

Citing papers per year: 2024: 1.

1 paper.

  1. 2024Diffusion Model for Planning: A Systematic Literature Reviewcited by 17

Similar, not cited

Close in wording, no citation either way

  1. 2024Diffusion Models as Optimizers for Efficient Planning in Offline RL
  2. 2024Preferred-Action-Optimized Diffusion Policies for Offline Reinforcement Learning
  3. 2024Contrastive Representation for Data Filtering in Cross-Domain Offline Reinforcement Learning
  4. 2024ATraDiff: Accelerating Online Reinforcement Learning with Imaginary Trajectories
  5. 2024DiffStitch: Boosting Offline Reinforcement Learning with Diffusion-based Trajectory Stitching
  6. 2023Using Offline Data to Speed Up Reinforcement Learning in Procedurally Generated Environments
  7. 2024Offline Reinforcement Learning from Datasets with Structured Non-Stationarity
  8. 2026RS-Diffuser: Risk-Sensitive Diffusion Planning with Distributional Value Guidance