mlsanityData to 14 Oct 2026

Contrastive Difference Predictive Coding

arXiv 2310.20141 ↗First posted 31 Oct 2023cs.LG cs.AIcited by 36

Not on the map. Why?

Abstract (arXiv)

Predicting and reasoning about the future lie at the heart of many time-series questions. For example, goal-conditioned reinforcement learning can be viewed as learning representations to predict which states are likely to be visited in the future. While prior methods have used contrastive predictive coding to model time series data, learning representations that encode long-term dependencies usually requires large amounts of data. In this paper, we introduce a temporal difference version of contrastive predictive coding that stitches together pieces of different time series data to decrease the amount of data required to learn predictions of future events. We apply this representation learning method to derive an off-policy algorithm for goal-conditioned RL. Experiments demonstrate that, compared with prior RL methods, ours achieves 2 × median improvement in success rates and can better cope with stochastic environments. In tabular settings, we show that our method is about 20 × more sample efficient than the successor representation and 1500 × more sample efficient than the standard (Monte Carlo) version of contrastive predictive coding.

Key facts. Cited by 36 papers in the map (data through 14 Oct 2026). Builds on 73 of its 99 references that are in the map. First posted on arXiv 31 Oct 2023.

Builds on

73 of 99 references.

References · 73

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 12,049
  3. 2018Representation Learning with Contrastive Predictive Codingcited by 7,777
  4. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,403
  5. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,750
  6. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,864
  7. 2015FaceNet: A Unified Embedding for Face Recognition and Clusteringcited by 2,849
  8. 2021SimCSE: Simple Contrastive Learning of Sentence Embeddingscited by 2,173
  9. 2018Unsupervised Feature Learning via Non-Parametric Instance-level Discriminationcited by 1,845
  10. 2020Conservative Q-Learning for Offline Reinforcement Learningcited by 1,809
  11. 2016Generative Adversarial Imitation Learningcited by 1,597
  12. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,546
38 more
  1. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,427
  2. 2020Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hyperspherecited by 1,405
  3. 2019Contrastive Multiview Codingcited by 1,301
  4. 2021Offline Reinforcement Learning with Implicit Q-Learningcited by 1,293
  5. 2017Hindsight Experience Replaycited by 1,136
  6. 2021A Minimalist Approach to Offline Reinforcement Learningcited by 899
  7. 2019Stabilizing Off-Policy Q-Learning via Bootstrapping Error Reductioncited by 840
  8. 2020CURL: Contrastive Unsupervised Representations for Reinforcement Learningcited by 760
  9. 2018Diversity is All You Need: Learning Skills without a Reward Functioncited by 752
  10. 2022R3M: A Universal Visual Representation for Robot Manipulationcited by 694
  11. 2019Data-Efficient Image Recognition with Contrastive Predictive Codingcited by 673
  12. 2019On Variational Bounds of Mutual Informationcited by 559
  13. 2019A Theoretical Analysis of Contrastive Unsupervised Representation Learningcited by 530
  14. 2020Reinforcement Learning with Augmented Datacited by 468
  15. 2017Time-Contrastive Networks: Self-Supervised Learning from Videocited by 459
  16. 2018Data-Efficient Hierarchical Reinforcement Learningcited by 455
  17. 2015Deep Metric Learning via Lifted Structured Feature Embeddingcited by 449
  18. 2022VIP: Towards Universal Visual Reward and Representation via Value-Implicit Pre-Trainingcited by 395
  19. 2019Relay Policy Learning: Solving Long-Horizon Tasks via Imitation and Reinforcement Learningcited by 384
  20. 2016Successor Features for Transfer in Reinforcement Learningcited by 340
  21. 2019Learning Latent Plans from Playcited by 316
  22. 2019On Mutual Information Maximization for Representation Learningcited by 316
  23. 2016Variational Intrinsic Controlcited by 267
  24. 2018Multi-Goal Reinforcement Learning: Challenging Robotics Environments and Request for Researchcited by 260
  25. 2022Contrastive Learning as Goal-Conditioned Reinforcement Learningcited by 253
  26. 2018Visual Reinforcement Learning with Imagined Goalscited by 251
  27. 2018An efficient framework for learning sentence representationscited by 198
  28. 2019Search on the Replay Buffer: Bridging Planning and Reinforcement Learningcited by 194
  29. 2021RvS: What is Essential for Offline RL via Supervised Learning?cited by 185
  30. 2016Exploring the Limits of Language Modelingcited by 173
  31. 2018Learning by Playing - Solving Sparse Reward Tasks from Scratchcited by 167
  32. 2019Goal-conditioned Imitation Learningcited by 162
  33. 2018Self-Imitation Learningcited by 138
  34. 2021Goal-Conditioned Reinforcement Learning with Imagined Subgoalscited by 134
  35. 2020Explore, Discover and Learn: Unsupervised Discovery of State-Covering Skillscited by 122
  36. 2021Rapid Exploration for Open-World Navigation with Latent Goal Modelscited by 121
  37. 2023Optimal Goal-Reaching Reinforcement Learning via Quasimetric Learningcited by 121
  38. 2021Learning One Representation to Optimize All Rewardscited by 115

50 most cited of 73 All 73 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 8; 2025: 16; 2026: 12.

36 papers.

Sort:

By area:

  1. 2024OGBench: Benchmarking Offline Goal-Conditioned RLcited by 238
  2. 2024Learning Temporal Distances: Contrastive Successor Features Can Provide a Metric Structure for Decision-Makingcited by 56
  3. 20251000 Layer Networks for Self-Supervised RL: Scaling Depth Can Enable New Goal-Reaching Capabilitiescited by 45
  4. 2025Offline Goal-conditioned Reinforcement Learning with Quasimetric Representationscited by 24
  5. 2024A Single Goal is All You Need: Skills and Exploration Emerge from Contrastive RL without Rewards, Demonstrations, or Subgoalscited by 23
  6. 2024Can a MISL Fly? Analysis and Ingredients for Mutual Information Skill Learningcited by 21
  7. 2025Dual Goal Representationscited by 16
  8. 2025On a Connection Between Imitation Learning and RLHFcited by 13
  9. 2024Finer Behavioral Foundation Models via Auto-Regressive Features and Advantage Weightingcited by 12
  10. 2025Transitive RL: Value Learning via Divide and Conquercited by 12
  11. 2025Flattening Hierarchies with Policy Bootstrappingcited by 11
  12. 2025Intention-Conditioned Flow Occupancy Modelscited by 9
24 more
  1. 2024Align Your Intents: Offline Imitation Learning via Optimal Transportcited by 8
  2. 2025Test-time Offline Reinforcement Learning on Goal-related Experiencecited by 8
  3. 2025Temporal Distance-aware Transition Augmentation for Offline Model-based Reinforcement Learningcited by 6
  4. 2025Contrastive Representations for Temporal Reasoningcited by 6
  5. 2024Predictive representations: building blocks of intelligencecited by 4
  6. 2026Can We Really Learn One Representation to Optimize All Rewards?cited by 3
  7. 2026How VLAs (Really) Work In Open-World Environmentscited by 3
  8. 2024Exploration by Learning Diverse Skills through Successor State Measurescited by 2
  9. 2025Unifying Agent Interaction and World Information for Multi-agent Coordinationcited by 2
  10. 2025PvP: Data-Efficient Humanoid Robot Learning with Proprioceptive-Privileged Contrastive Representationscited by 2
  11. 2026Dreamer-CDP: Improving Reconstruction-free World Models Via Continuous Deterministic Representation Predictioncited by 2
  12. 2025Universal Value-Function Uncertaintiescited by 1
  13. 2025Periodic Skill Discoverycited by 1
  14. 2026Occupancy Reward Shaping: Improving Credit Assignment for Offline Goal-Conditioned Reinforcement Learningcited by 1
  15. 2026Compositional Transduction with Latent Analogies for Offline Goal-Conditioned Reinforcement Learningcited by 1
  16. 2026Recursive Value Learning for Long-Horizon Offline Goal-Conditioned RLcited by 1
  17. 2025Towards Achieving Perfect Multimodal Alignmentcited by 0
  18. 2025Reachability Weighted Offline Goal-conditioned Resamplingcited by 0
  19. 2026ViSA: Visited-State Augmentation for Generalized Goal-Space Contrastive Reinforcement Learningcited by 0
  20. 2026Self-Supervised On-Policy Reinforcement Learning via Contrastive Proximal Policy Optimisationcited by 0
  21. 2026Geometric Self-Supervised Pre-training for Neural Combinatorial Optimizationcited by 0
  22. 2026SUN: Reaching for Novelty in Reinforcement Learningcited by 0
  23. 2026GTRL: Grounding Divide-and-Conquer Value Learning with Temporal Differencescited by 0
  24. 2026ChronoSRL: Temporal Geometry for Self-Supervised Reinforcement Learningcited by 0