mlsanityData to 7 Oct 2026

Contrastive Value Learning: Implicit Models for Simple Offline RL

arXiv 2211.02100 ↗First posted 3 Nov 2022cs.LG cs.AIcited by 15

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

Model-based reinforcement learning (RL) methods are appealing in the offline setting because they allow an agent to reason about the consequences of actions without interacting with the environment. Prior methods learn a 1-step dynamics model, which predicts the next state given the current state and action. These models do not immediately tell the agent which actions to take, but must be integrated into a larger RL framework. Can we model the environment dynamics in a different way, such that the learned model does directly indicate the value of each action? In this paper, we propose Contrastive Value Learning (CVL), which learns an implicit, multi-step model of the environment dynamics. This model can be learned without access to reward functions, but nonetheless can be used to directly estimate the value of each action, without requiring any TD learning. Because this model represents the multi-step transitions implicitly, it avoids having to predict high-dimensional observations and thus scales to high-dimensional tasks. Our experiments demonstrate that CVL outperforms prior offline RL methods on complex continuous control benchmarks.

Key facts. Cited by 15 papers in the map (data through 7 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 39 of its 50 references that are in the map. First posted on arXiv 3 Nov 2022.

Builds on

39 of 50 references.

Closest ancestors · 9

Same direction, newest first

  1. 2022Adaptive Behavior Cloning Regularization for Stable Offline-to-Online Reinforcement Learning25 Oct
  2. 2022How to Leverage Unlabeled Data in Offline Reinforcement Learning3 Feb
  3. 2021Offline Reinforcement Learning with Fisher Divergence Critic Regularization14 Mar
  4. 2021COMBO: Conservative Offline Model-Based Policy Optimization16 Feb
  5. 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
  6. 2020MOPO: Model-based Offline Policy Optimization27 May
  7. 2020MOReL : Model-Based Offline Reinforcement Learning12 May
  8. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learning15 Apr
  9. 2019Behavior Regularized Offline Reinforcement Learning26 Nov

Other references · 30

  1. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  2. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  3. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  4. 2016Densely Connected Convolutional Networkscited by 6,691
  5. 2016Layer Normalizationcited by 5,579
  6. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  7. 2013Efficient Estimation of Word Representations in Vector Spacecited by 4,346
  8. 2020Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hyperspherecited by 1,393
  9. 2018Learning deep representations by mutual information estimation and maximizationcited by 1,265
  10. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
  11. 2019Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learningcited by 1,168
  12. 2018IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architecturescited by 866
18 more
  1. 2021Deep Reinforcement Learning at the Edge of the Statistical Precipicecited by 781
  2. 2020CURL: Contrastive Unsupervised Representations for Reinforcement Learningcited by 745
  3. 2016Successor Features for Transfer in Reinforcement Learningcited by 331
  4. 2021Provable Guarantees for Self-Supervised Deep Learning with Spectral Contrastive Losscited by 311
  5. 2022Contrastive Learning as Goal-Conditioned Reinforcement Learningcited by 241
  6. 2020Model-Based Offline Planningcited by 126
  7. 2021Contrastive Behavioral Similarity Embeddings for Generalization in Reinforcement Learningcited by 123
  8. 2020Phasic Policy Gradientcited by 121
  9. 2021Learning One Representation to Optimize All Rewardscited by 110
  10. 2021Pretraining Representations for Data-Efficient Reinforcement Learningcited by 105
  11. 2018Noise Contrastive Estimation and Negative Sampling for Conditional Models: Consistency and Statistical Efficiencycited by 103
  12. 2020C-Learning: Learning to Achieve Goals via Recursive Classificationcited by 99
  13. 2020Deep Reinforcement and InfoMax Learningcited by 79
  14. 2022Pre-Training for Robots: Offline RL Enables Learning New Tasks from a Handful of Trialscited by 78
  15. 2021Learning Successor States and Goal-Dependent Values: A Mathematical Viewpointcited by 67
  16. 2020Generative Temporal Difference Learning for Infinite-Horizon Predictioncited by 43
  17. 2021Provable Representation Learning for Imitation with Contrastive Fourier Featurescited by 34
  18. 2021PsiPhi-Learning: Reinforcement Learning with Demonstrations using Successor Features and Inverse Temporal Difference Learningcited by 23

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 4; 2024: 3; 2025: 3; 2026: 5.

15 papers.

Sort:

By area:

  1. 2023Foundation Models for Decision Making: Problems, Methods, and Opportunitiescited by 160
  2. 2023Contrastive Difference Predictive Codingcited by 35
  3. 2024Inference via Interpolation: Contrastive Representations Provably Enable Planning and Inferencecited by 15
  4. 2024Listwise Reward Estimation for Offline Preference-based Reinforcement Learningcited by 14
  5. 2023Value function estimation using conditional diffusion models for controlcited by 10
  6. 2023Accelerating exploration and representation learning with offline pre-trainingcited by 7
  7. 2025Intention-Conditioned Flow Occupancy Modelscited by 7
  8. 2025Temporal Distance-aware Transition Augmentation for Offline Model-based Reinforcement Learningcited by 5
  9. 2024Off-Policy Maximum Entropy RL with Future State and Action Visitation Measurescited by 3
  10. 2026SVL: Goal-Conditioned Reinforcement Learning as Survival Learningcited by 3
  11. 2025Video-Enhanced Offline Reinforcement Learning: A Model-Based Approachcited by 1
  12. 2026Can We Really Learn One Representation to Optimize All Rewards?cited by 1
3 more
  1. 2026Temporal Representations for Exploration: Learning Complex Exploratory Behavior without Extrinsic Rewardscited by 1
  2. 2026Maximum-Entropy Exploration with Future State-Action Visitation Measurescited by 0
  3. 2026Maximizing Mutual Information Between Prompt and Response Improves LLM Performance With No Additional Datacited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Contrastive Example-Based Control
  2. 2021Offline Reinforcement Learning with Value-based Episodic Memory
  3. 2025Expressive Value Learning for Scalable Offline Reinforcement Learning
  4. 2025Scalable Offline Model-Based RL with Action Chunks
  5. 2020Overcoming Model Bias for Robust Offline Deep Reinforcement Learning
  6. 2024Contrastive Diffuser: Planning Towards High Return States via Contrastive Learning
  7. 2024Is Value Learning Really the Main Bottleneck in Offline RL?
  8. 2026Peng's Q(λ) for Conservative Value Estimation in Offline Reinforcement Learning