mlsanityData to 7 Oct 2026

Stabilizing Contrastive RL: Techniques for Robotic Goal Reaching from Offline Data

arXiv 2306.03346 ↗First posted 6 Jun 2023cs.LG cs.AIcited by 19

Not on the map. Why?

Abstract (arXiv)

Robotic systems that rely primarily on self-supervised learning have the potential to decrease the amount of human annotation and engineering effort required to learn control strategies. In the same way that prior robotic systems have leveraged self-supervised techniques from computer vision (CV) and natural language processing (NLP), our work builds on prior work showing that the reinforcement learning (RL) itself can be cast as a self-supervised problem: learning to reach any goal without human-specified rewards or labels. Despite the seeming appeal, little (if any) prior work has demonstrated how self-supervised RL methods can be practically deployed on robotic systems. By first studying a challenging simulated version of this task, we discover design decisions about architectures and hyperparameters that increase the success rate by 2 ×. These findings lay the groundwork for our main result: we demonstrate that a self-supervised RL algorithm based on contrastive learning can solve real-world, image-based robotic manipulation tasks, with tasks being specified by a single goal image provided after training.

Key facts. Cited by 19 papers in the map (data through 7 Oct 2026). Builds on 101 of its 113 references that are in the map. First posted on arXiv 6 Jun 2023.

Builds on

101 of 113 references.

References · 101

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  4. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  5. 2020Language Models are Few-Shot Learnerscited by 32,199
  6. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  7. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  8. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  9. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  10. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  11. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  12. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,415
38 more
  1. 2020Scaling Laws for Neural Language Modelscited by 6,411
  2. 2016Layer Normalizationcited by 5,579
  3. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,712
  4. 2017Neural Discrete Representation Learningcited by 4,650
  5. 2021An Empirical Study of Training Self-Supervised Vision Transformerscited by 1,610
  6. 2019Dream to Control: Learning Behaviors by Latent Imaginationcited by 1,598
  7. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,528
  8. 2018World Modelscited by 1,444
  9. 2020Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hyperspherecited by 1,393
  10. 2019Mastering Atari, Go, Chess and Shogi by Planning with a Learned Modelcited by 1,327
  11. 2019Contrastive Multiview Codingcited by 1,297
  12. 2018Learning deep representations by mutual information estimation and maximizationcited by 1,265
  13. 2021Offline Reinforcement Learning with Implicit Q-Learningcited by 1,245
  14. 2020Gradient Surgery for Multi-Task Learningcited by 1,210
  15. 2021Scaling Vision Transformerscited by 993
  16. 2020Mastering Atari with Discrete World Modelscited by 990
  17. 2021A Minimalist Approach to Offline Reinforcement Learningcited by 882
  18. 2018IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architecturescited by 866
  19. 2019When to Trust Your Model: Model-Based Policy Optimizationcited by 753
  20. 2019Advantage-Weighted Regression: Simple and Scalable Off-Policy Reinforcement Learningcited by 750
  21. 2020CURL: Contrastive Unsupervised Representations for Reinforcement Learningcited by 745
  22. 2022R3M: A Universal Visual Representation for Robot Manipulationcited by 678
  23. 2020AWAC: Accelerating Online Reinforcement Learning with Offline Datasetscited by 647
  24. 2021Offline Reinforcement Learning as One Big Sequence Modeling Problemcited by 636
  25. 2020MOPO: Model-based Offline Policy Optimizationcited by 630
  26. 2020Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixelscited by 573
  27. 2022Perceiver-Actor: A Multi-Task Transformer for Robotic Manipulationcited by 545
  28. 2017Time-Contrastive Networks: Self-Supervised Learning from Videocited by 453
  29. 2020Reinforcement Learning with Augmented Datacited by 452
  30. 2020Learning Invariant Representations for Reinforcement Learning without Reconstructioncited by 424
  31. 2022VIP: Towards Universal Visual Reward and Representation via Value-Implicit Pre-Trainingcited by 384
  32. 2019Relay Policy Learning: Solving Long-Horizon Tasks via Imitation and Reinforcement Learningcited by 380
  33. 2021Mastering Visual Continuous Control: Improved Data-Augmented Reinforcement Learningcited by 359
  34. 2022Video PreTraining (VPT): Learning to Act by Watching Unlabeled Online Videoscited by 338
  35. 2019Learning Latent Plans from Playcited by 310
  36. 2020Agent57: Outperforming the Atari Human Benchmarkcited by 300
  37. 2015Deep multi-scale video prediction beyond mean square errorcited by 297
  38. 2016Deep Visual Foresight for Planning Robot Motioncited by 276

50 most cited of 101 All 101 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 9; 2026: 9.

19 papers.

Sort:

By area:

  1. 2025GEVRM: Goal-Expressive Video Generation Model For Robust Visual Manipulationcited by 26
  2. 2024Inference via Interpolation: Contrastive Representations Provably Enable Planning and Inferencecited by 15
  3. 2025Graph-Assisted Stitching for Offline Hierarchical Reinforcement Learningcited by 12
  4. 2025Test-Time Graph Search for Goal-Conditioned Reinforcement Learningcited by 8
  5. 2025Intention-Conditioned Flow Occupancy Modelscited by 7
  6. 2025Multistep Quasimetric Learning for Scalable Goal-conditioned Reinforcement Learningcited by 6
  7. 2026Laplacian Representations for Decision-Time Planningcited by 4
  8. 2025Dynamic Contrastive Skill Learning with State-Transition Based Skill Clustering and Dynamic Length Adjustmentcited by 3
  9. 2025Unpacking Softmax: How Temperature Drives Representation Collapse, Compression, and Generalizationcited by 3
  10. 2025Demystifying the Mechanisms Behind Emergent Exploration in Goal-conditioned RLcited by 3
  11. 2025GCHR : Goal-Conditioned Hindsight Regularization for Sample-Efficient Reinforcement Learningcited by 2
  12. 2026FOCA: Future-Oriented Conditioning for Data-Efficient Vision-Language-Action Adaptationcited by 2
7 more
  1. 2026Can We Really Learn One Representation to Optimize All Rewards?cited by 1
  2. 2026Three Steps at a Time: Learning Representations from Action Sequences in Contrastive RLcited by 1
  3. 2026Scalable Dexterous Robot Learning with AR-based Remote Human-Robot Interactionscited by 0
  4. 2026MoMo: Conditioned Contrastive Representation Learning for Preference-Modulated Planningcited by 0
  5. 2026Self-Supervised On-Policy Reinforcement Learning via Contrastive Proximal Policy Optimisationcited by 0
  6. 2026Good Rankers, Bad Objectives: Bilinear Contrastive Critics under Expressive Policy Searchcited by 0
  7. 2026ChronoSRL: Temporal Geometry for Self-Supervised Reinforcement Learningcited by 0