Data to 7 Oct 2026

Contrastive Learning as Goal-Conditioned Reinforcement Learning

arXiv 2206.07568 ↗First posted 15 Jun 2022cs.LG cs.AIcited by 241

Not on the map. Why?

Cited by 59 in Offline RL and imitation learning from data, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

In reinforcement learning (RL), it is easier to solve a task if given a good representation. While deep RL should automatically acquire such good representations, prior work often finds that learning representations in an end-to-end fashion is unstable and instead equip RL algorithms with additional representation learning parts (e.g., auxiliary losses, data augmentation). How can we design RL algorithms that directly acquire good representations? In this paper, instead of adding representation learning parts to an existing RL algorithm, we show (contrastive) representation learning methods can be cast as RL algorithms in their own right. To do this, we build upon prior work and apply contrastive representation learning to action-labeled trajectories, in such a way that the (inner product of) learned representations exactly corresponds to a goal-conditioned value function. We use this idea to reinterpret a prior RL method as performing contrastive learning, and then use the idea to propose a much simpler method that achieves similar performance. Across a range of goal-conditioned RL tasks, we demonstrate that contrastive RL methods achieve higher success rates than prior non-contrastive methods, including in the offline RL setting. We also show that contrastive RL outperforms prior methods on image-based tasks, without using data augmentation or auxiliary objectives.

Key facts. Cited by 241 papers in the map (data through 7 Oct 2026). Builds on 118 of its 151 references that are in the map. First posted on arXiv 15 Jun 2022.

Builds on

118 of 151 references.

References · 118

2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
2018Representation Learning with Contrastive Predictive Codingcited by 7,662
2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,712
2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
2017Deep reinforcement learning from human preferencescited by 3,846
2013Distributed Representations of Words and Phrases and their Compositionalitycited by 3,462
2013Playing Atari with Deep Reinforcement Learningcited by 3,211
2020Exploring Simple Siamese Representation Learningcited by 2,853
2015FaceNet: A Unified Embedding for Face Recognition and Clusteringcited by 2,833
2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
2018Unsupervised Feature Learning via Non-Parametric Instance-level Discriminationcited by 1,838

38 more

2019Dream to Control: Learning Behaviors by Latent Imaginationcited by 1,598
2016Generative Adversarial Imitation Learningcited by 1,576
2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,528
2018World Modelscited by 1,444
2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,406
2019Contrastive Multiview Codingcited by 1,297
2016Understanding intermediate layers using linear classifier probescited by 1,279
2018Learning deep representations by mutual information estimation and maximizationcited by 1,265
2021Offline Reinforcement Learning with Implicit Q-Learningcited by 1,245
2018Learning Latent Dynamics for Planning from Pixelscited by 1,221
2020Gradient Surgery for Multi-Task Learningcited by 1,210
2019Meta-World: A Benchmark and Evaluation for Multi-Task and Meta Reinforcement Learningcited by 1,168
2017Hindsight Experience Replaycited by 1,096
2021A Minimalist Approach to Offline Reinforcement Learningcited by 882
2020CURL: Contrastive Unsupervised Representations for Reinforcement Learningcited by 745
2018Diversity is All You Need: Learning Skills without a Reward Functioncited by 734
2020Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixelscited by 573
2019On Variational Bounds of Mutual Informationcited by 550
2016f-GAN: Training Generative Neural Samplers using Variational Divergence Minimizationcited by 504
2017Learning Robust Rewards with Adversarial Inverse Reinforcement Learningcited by 485
2017Time-Contrastive Networks: Self-Supervised Learning from Videocited by 453
2020Reinforcement Learning with Augmented Datacited by 452
2018Data-Efficient Hierarchical Reinforcement Learningcited by 436
2020Learning Invariant Representations for Reinforcement Learning without Reconstructioncited by 424
2014Deep metric learning using Triplet networkcited by 421
2021Mastering Visual Continuous Control: Improved Data-Augmented Reinforcement Learningcited by 359
2016Successor Features for Transfer in Reinforcement Learningcited by 331
2019On Mutual Information Maximization for Representation Learningcited by 315
2019Learning Latent Plans from Playcited by 310
2019Dynamics-Aware Unsupervised Discovery of Skillscited by 308
2019Improving Sample Efficiency in Model-Free Reinforcement Learning from Imagescited by 268
2015Embed to Control: A Locally Linear Latent Dynamics Model for Control from Raw Imagescited by 266
2020Decoupling Representation Learning from Reinforcement Learningcited by 263
2016Variational Intrinsic Controlcited by 262
2018Multi-Goal Reinforcement Learning: Challenging Robotics Environments and Request for Researchcited by 256
2019Extrapolating Beyond Suboptimal Demonstrations via Inverse Reinforcement Learning from Observationscited by 256
2017Automatic Goal Generation for Reinforcement Learning Agentscited by 252
2018Visual Reinforcement Learning with Imagined Goalscited by 248

50 most cited of 118 All 118 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 7; 2023: 43; 2024: 51; 2025: 65; 2026: 75.

241 papers.

Sort:

By area:

2023BridgeData V2: A Dataset for Robot Learning at Scalecited by 634
2023TD-MPC2: Scalable, Robust World Models for Continuous Controlcited by 482
2022VIP: Towards Universal Visual Reward and Representation via Value-Implicit Pre-Trainingcited by 384
2023Goal-Conditioned Imitation Learning using Score-based Diffusion Policiescited by 229
2024OGBench: Benchmarking Offline Goal-Conditioned RLcited by 216
2023Look Before You Leap: Unveiling the Power of GPT-4V in Robotic Vision-Language Planningcited by 161
2023HIQL: Offline Goal-Conditioned RL with Latent States as Actionscited by 133
2023Optimal Goal-Reaching Reinforcement Learning via Quasimetric Learningcited by 116
2022From Play to Policy: Conditional Behavior Generation from Uncurated Robot Datacited by 103
2024Foundation Policies with Hilbert Representationscited by 89
2024Is Value Learning Really the Main Bottleneck in Offline RL?cited by 82
2023Contrastive Preference Learning: Learning from Human Feedback without RLcited by 77

38 more

2025Learning from Reward-Free Offline Data: A Case for Planning with Latent Dynamics Modelscited by 77
2023Factorized Contrastive Learning: Going Beyond Multi-view Redundancycited by 73
2023Reinforcement Learning from Passive Data via Latent Intentionscited by 64
2023The Wisdom of Hindsight Makes Language Models Better Instruction Followerscited by 59
2023Behavior Retrieval: Few-Shot Imitation Learning by Querying Unlabeled Datasetscited by 56
2023TACO: Temporal Latent Action-Driven Contrastive Loss for Visual Reinforcement Learningcited by 56
2025Horizon Reduction Makes RL Scalablecited by 55
2024Learning Temporal Distances: Contrastive Successor Features Can Provide a Metric Structure for Decision-Makingcited by 53
2025Diffusion Guidance Is a Controllable Policy Improvement Operatorcited by 51
2023LLM-Powered Hierarchical Language Agent for Real-time Human-AI Coordinationcited by 48
2023RLCD: Reinforcement Learning from Contrastive Distillation for Language Model Alignmentcited by 42
20251000 Layer Networks for Self-Supervised RL: Scaling Depth Can Enable New Goal-Reaching Capabilitiescited by 39
2025Generative Trajectory Stitching through Diffusion Compositioncited by 38
2023Searching Large Neighborhoods for Integer Linear Programs with Contrastive Learningcited by 37
2025Meta-World+: An Improved, Standardized, RL Benchmarkcited by 37
2023Direct Preference-based Policy Optimization without Reward Modelingcited by 36
2023Contrastive Difference Predictive Codingcited by 35
2024Autonomous Improvement of Instruction Following Skills via Foundation Modelscited by 32
2024Unsupervised Zero-Shot Reinforcement Learning via Functional Reward Encodingscited by 31
2023Inverse Dynamics Pretraining Learns Good Representations for Multitask Imitationcited by 30
2025Self-Improving Embodied Foundation Modelscited by 28
2024Heterogeneous Contrastive Learning for Foundation Models and Beyondcited by 27
2023Behavior Contrastive Learning for Unsupervised Skill Discoverycited by 26
2023SPRINT: Scalable Policy Pre-Training via Language Instruction Relabelingcited by 24
2024Cross-Domain Policy Adaptation by Capturing Representation Mismatchcited by 24
2025Bigger, Regularized, Categorical: High-Capacity Value Functions are Efficient Multi-Task Learnerscited by 23
2025Option-aware Temporally Abstracted Value for Offline Goal-Conditioned Reinforcement Learningcited by 22
2023Zero-Shot Reinforcement Learning from Low Quality Datacited by 21
2023SMORE: Score Models for Offline Goal-Conditioned Reinforcement Learningcited by 21
2023GOPlan: Goal-conditioned Offline Reinforcement Learning by Planning with Learned Modelscited by 20
2024A Single Goal is All You Need: Skills and Exploration Emerge from Contrastive RL without Rewards, Demonstrations, or Subgoalscited by 20
2025floq: Training Critics via Flow-Matching for Scaling Compute in Value-Based RLcited by 20
2025Offline Goal-conditioned Reinforcement Learning with Quasimetric Representationscited by 20
2023Proximal Curriculum for Reinforcement Learning Agentscited by 19
2023Stabilizing Contrastive RL: Techniques for Robotic Goal Reaching from Offline Datacited by 19
2024Contrastive Representation for Data Filtering in Cross-Domain Offline Reinforcement Learningcited by 18
2024Can a MISL Fly? Analysis and Ingredients for Mutual Information Skill Learningcited by 18
2024Cross-Embodiment Robot Manipulation Skill Transfer using Latent Space Alignmentcited by 17

50 most cited of 241 All 241 (JSON)

Similar, not cited

Close in wording, no citation either way