# Stabilizing Contrastive RL: Techniques for Robotic Goal Reaching from Offline Data

arXiv: https://arxiv.org/abs/2306.03346 · HTML: https://mlsanity.com/paper/2306.03346/

## Key facts

- Cited by 23 papers in the map (data through 14 Oct 2026).
- Builds on 101 of its 113 references that are in the map.
- First posted on arXiv 6 Jun 2023.

## Builds on (most cited first, 30 of 101)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Masked Autoencoders Are Scalable Vision Learners](https://mlsanity.com/paper/2111.06377/) (2021)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [EfficientNet: Rethinking Model Scaling for Convolutional Neural Networks](https://mlsanity.com/paper/1905.11946/) (2019)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [Bootstrap your own latent: A new approach to self-supervised Learning](https://mlsanity.com/paper/2006.07733/) (2020)
- [Neural Discrete Representation Learning](https://mlsanity.com/paper/1711.00937/) (2017)
- [Dream to Control: Learning Behaviors by Latent Imagination](https://mlsanity.com/paper/1912.01603/) (2019)
- [An Empirical Study of Training Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.02057/) (2021)
- [Decision Transformer: Reinforcement Learning via Sequence Modeling](https://mlsanity.com/paper/2106.01345/) (2021)
- [World Models](https://mlsanity.com/paper/1803.10122/) (2018)
- [Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hypersphere](https://mlsanity.com/paper/2005.10242/) (2020)
- [Mastering Atari, Go, Chess and Shogi by Planning with a Learned Model](https://mlsanity.com/paper/1911.08265/) (2019)
- [Contrastive Multiview Coding](https://mlsanity.com/paper/1906.05849/) (2019)
- [Offline Reinforcement Learning with Implicit Q-Learning](https://mlsanity.com/paper/2110.06169/) (2021)
- [Learning deep representations by mutual information estimation and maximization](https://mlsanity.com/paper/1808.06670/) (2018)
- [Gradient Surgery for Multi-Task Learning](https://mlsanity.com/paper/2001.06782/) (2020)
- [Mastering Atari with Discrete World Models](https://mlsanity.com/paper/2010.02193/) (2020)
- [Scaling Vision Transformers](https://mlsanity.com/paper/2106.04560/) (2021)
- [A Minimalist Approach to Offline Reinforcement Learning](https://mlsanity.com/paper/2106.06860/) (2021)
- [IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architectures](https://mlsanity.com/paper/1802.01561/) (2018)

## Built on by (most cited first, 23 of 23)

- [GEVRM: Goal-Expressive Video Generation Model For Robust Visual Manipulation](https://mlsanity.com/paper/2502.09268/) (2025)
- [Inference via Interpolation: Contrastive Representations Provably Enable Planning and Inference](https://mlsanity.com/paper/2403.04082/) (2024)
- [Graph-Assisted Stitching for Offline Hierarchical Reinforcement Learning](https://mlsanity.com/paper/2506.07744/) (2025)
- [Intention-Conditioned Flow Occupancy Models](https://mlsanity.com/paper/2506.08902/) (2025)
- [Test-Time Graph Search for Goal-Conditioned Reinforcement Learning](https://mlsanity.com/paper/2510.07257/) (2025)
- [Multistep Quasimetric Learning for Scalable Goal-conditioned Reinforcement Learning](https://mlsanity.com/paper/2511.07730/) (2025)
- [Demystifying the Mechanisms Behind Emergent Exploration in Goal-conditioned RL](https://mlsanity.com/paper/2510.14129/) (2025)
- [Laplacian Representations for Decision-Time Planning](https://mlsanity.com/paper/2602.05031/) (2026)
- [Dynamic Contrastive Skill Learning with State-Transition Based Skill Clustering and Dynamic Length Adjustment](https://mlsanity.com/paper/2504.14805/) (2025)
- [Unpacking Softmax: How Temperature Drives Representation Collapse, Compression, and Generalization](https://mlsanity.com/paper/2506.01562/) (2025)
- [GCHR : Goal-Conditioned Hindsight Regularization for Sample-Efficient Reinforcement Learning](https://mlsanity.com/paper/2508.06108/) (2025)
- [Can We Really Learn One Representation to Optimize All Rewards?](https://mlsanity.com/paper/2602.11399/) (2026)
- [Three Steps at a Time: Learning Representations from Action Sequences in Contrastive RL](https://mlsanity.com/paper/2608.30640/) (2026)
- [FOCA: Future-Oriented Conditioning for Data-Efficient Vision-Language-Action Adaptation](https://mlsanity.com/paper/2606.20867/) (2026)
- [ACDC: Adaptive Curriculum Planning with Dynamic Contrastive Control for Goal-Conditioned Reinforcement Learning in Robotic Manipulation](https://mlsanity.com/paper/2603.02104/) (2026)
- [Scalable Dexterous Robot Learning with AR-based Remote Human-Robot Interactions](https://mlsanity.com/paper/2602.07341/) (2026)
- [MoMo: Conditioned Contrastive Representation Learning for Preference-Modulated Planning](https://mlsanity.com/paper/2605.08512/) (2026)
- [Self-Supervised On-Policy Reinforcement Learning via Contrastive Proximal Policy Optimisation](https://mlsanity.com/paper/2605.13554/) (2026)
- [Good Rankers, Bad Objectives: Bilinear Contrastive Critics under Expressive Policy Search](https://mlsanity.com/paper/2607.27422/) (2026)
- [ChronoSRL: Temporal Geometry for Self-Supervised Reinforcement Learning](https://mlsanity.com/paper/2609.36238/) (2026)
- [Direction-Conditioned Policies for Online Goal-Conditioned Reinforcement Learning](https://mlsanity.com/paper/2610.05087/) (2026)
- [Counterfactual Rollouts for Offline Contrastive Goal-Conditioned Reinforcement Learning](https://mlsanity.com/paper/2610.14953/) (2026)
- [Outperforming 1000 Euclidean Layers With Hyperbolic Goal-Conditioned RL](https://mlsanity.com/paper/2610.20009/) (2026)

## Abstract (arXiv)

Robotic systems that rely primarily on self-supervised learning have the potential to decrease the amount of human annotation and engineering effort required to learn control strategies. In the same way that prior robotic systems have leveraged self-supervised techniques from computer vision (CV) and natural language processing (NLP), our work builds on prior work showing that the reinforcement learning (RL) itself can be cast as a self-supervised problem: learning to reach any goal without human-specified rewards or labels. Despite the seeming appeal, little (if any) prior work has demonstrated how self-supervised RL methods can be practically deployed on robotic systems. By first studying a challenging simulated version of this task, we discover design decisions about architectures and hyperparameters that increase the success rate by 2 ×. These findings lay the groundwork for our main result: we demonstrate that a self-supervised RL algorithm based on contrastive learning can solve real-world, image-based robotic manipulation tasks, with tasks being specified by a single goal image provided after training.