mlsanityData to 7 Oct 2026

Return-Based Contrastive Representation Learning for Reinforcement Learning

arXiv 2102.10960 ↗First posted 22 Feb 2021cs.LGcited by 36

Not on the map. Why?

Abstract (arXiv)

Recently, various auxiliary tasks have been proposed to accelerate representation learning and improve sample efficiency in deep reinforcement learning (RL). However, existing auxiliary tasks do not take the characteristics of RL problems into consideration and are unsupervised. By leveraging returns, the most important feedback signals in RL, we propose a novel auxiliary task that forces the learnt representations to discriminate state-action pairs with different returns. Our auxiliary loss is theoretically justified to learn representations that capture the structure of a new form of state-action abstraction, under which state-action pairs with similar return distributions are aggregated together. In low data regime, our algorithm outperforms strong baselines on complex tasks in Atari games and DeepMind Control suite, and achieves even better performance when combined with existing auxiliary tasks.

Key facts. Cited by 36 papers in the map (data through 7 Oct 2026). Builds on 36 of its 55 references that are in the map. First posted on arXiv 22 Feb 2021.

Builds on

36 of 55 references.

References · 36

  1. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  2. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  3. 2019Dream to Control: Learning Behaviors by Latent Imaginationcited by 1,598
  4. 2018Learning Latent Dynamics for Planning from Pixelscited by 1,221
  5. 2018DeepMind Control Suitecited by 935
  6. 2017Rainbow: Combining Improvements in Deep Reinforcement Learningcited by 774
  7. 2020CURL: Contrastive Unsupervised Representations for Reinforcement Learningcited by 745
  8. 2017A Distributional Perspective on Reinforcement Learningcited by 743
  9. 2019Model-Based Reinforcement Learning for Ataricited by 495
  10. 2017Time-Contrastive Networks: Self-Supervised Learning from Videocited by 453
  11. 2020Learning Invariant Representations for Reinforcement Learning without Reconstructioncited by 424
  12. 2016Reinforcement Learning with Unsupervised Auxiliary Taskscited by 400
24 more
  1. 2019Improving Sample Efficiency in Model-Free Reinforcement Learning from Imagescited by 268
  2. 2019Stochastic Latent Actor-Critic: Deep Reinforcement Learning with a Latent Variable Modelcited by 228
  3. 2019DeepMDP: Learning Continuous Latent Space Models for Representation Learningcited by 204
  4. 2016Learning to Navigate in Complex Environmentscited by 192
  5. 2019Unsupervised State Representation Learning in Ataricited by 144
  6. 2018Multi-task Deep Reinforcement Learning with PopArtcited by 138
  7. 2019Scalable methods for computing state similarity in deterministic Markov Decision Processescited by 133
  8. 2020MDP Homomorphic Networks: Group Symmetries in Reinforcement Learningcited by 133
  9. 2019When to use parametric models in reinforcement learning?cited by 118
  10. 2019Kinematic State Abstraction and Provably Efficient Rich-Observation Reinforcement Learningcited by 114
  11. 2018Universal Successor Features Approximatorscited by 102
  12. 2020Bootstrap Latent-Predictive Representations for Multitask Reinforcement Learningcited by 101
  13. 2018Playing hard exploration games by watching YouTubecited by 98
  14. 2020Deep Reinforcement and InfoMax Learningcited by 79
  15. 2016Loss is its own Reward: Self-Supervision for Reinforcement Learningcited by 72
  16. 2020The Value-Improvement Path: Towards Better Representations for Reinforcement Learningcited by 60
  17. 2020Predictive Information Accelerates Learning in RLcited by 56
  18. 2019A Geometric Perspective on Optimal Representations for Reinforcement Learningcited by 54
  19. 2019Discovery of Useful Questions as Auxiliary Taskscited by 45
  20. 2018Combined Reinforcement Learning via Abstract Representationscited by 37
  21. 2018Learning Actionable Representations from Visual Observationscited by 35
  22. 2018Many-Goals Reinforcement Learningcited by 22
  23. 2018Online Abstraction with MDP Homomorphisms for Deep Learningcited by 13
  24. 2020Auxiliary-task Based Deep Reinforcement Learning for Participant Selection Problem in Mobile Crowdsourcingcited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 7; 2022: 6; 2023: 10; 2024: 5; 2025: 4; 2026: 4.

36 papers.

Sort:

By area:

  1. 2021Deep Reinforcement Learning at the Edge of the Statistical Precipicecited by 781
  2. 2023For SALE: State-Action Representation Learning for Deep Reinforcement Learningcited by 73
  3. 2023RLCD: Reinforcement Learning from Contrastive Distillation for Language Model Alignmentcited by 42
  4. 2021PlayVirtual: Augmenting Cycle-Consistent Virtual Trajectories for Reinforcement Learningcited by 41
  5. 2022Mask-based Latent Reconstruction for Reinforcement Learningcited by 39
  6. 2022Deep Learning Methods for Abstract Visual Reasoning: A Survey on Raven's Progressive Matricescited by 37
  7. 2022Contrastive UCB: Provably Efficient Contrastive Self-Supervised Learning in Online Reinforcement Learningcited by 26
  8. 2023Behavior Contrastive Learning for Unsupervised Skill Discoverycited by 26
  9. 2024Cross-Domain Policy Adaptation by Capturing Representation Mismatchcited by 24
  10. 2021CoBERL: Contrastive BERT for Reinforcement Learningcited by 20
  11. 2021Towards Deeper Deep Reinforcement Learning with Spectral Normalizationcited by 19
  12. 2022Unsupervised Representation Learning in Deep Reinforcement Learning: A Reviewcited by 18
24 more
  1. 2021Learn Goal-Conditioned Policy with Intrinsic Motivation for Deep Reinforcement Learningcited by 15
  2. 2022Value-Consistent Representation Learning for Data-Efficient Reinforcement Learningcited by 15
  3. 2023Contrastive Learning for Enhancing Robust Scene Transfer in Vision-based Agile Flightcited by 11
  4. 2025MOBODY: Model Based Off-Dynamics Offline Reinforcement Learningcited by 8
  5. 2023RePreM: Representation Pre-training with Masked Model for Reinforcement Learningcited by 5
  6. 2023CR-VAE: Contrastive Regularization on Variational Autoencoders for Preventing Posterior Collapsecited by 4
  7. 2021Application of Deep Reinforcement Learning to Payment Fraudcited by 3
  8. 2022Learning List-wise Representation in Reinforcement Learning for Ads Allocation with Multiple Auxiliary Taskscited by 3
  9. 2023Sequential Action-Induced Invariant Representation for Reinforcement Learningcited by 3
  10. 2024Feasibility Consistent Representation Learning for Safe Reinforcement Learningcited by 3
  11. 2024MOOSS: Mask-Enhanced Temporal Contrastive Learning for Smooth State Evolution in Visual Reinforcement Learningcited by 3
  12. 2021Pretrained Encoders are All You Needcited by 2
  13. 2024Learning Goal-Conditioned Representations for Language Reward Modelscited by 2
  14. 2023Revisiting Estimation Bias in Policy Gradients for Deep Reinforcement Learningcited by 1
  15. 2023Domain Adaptation of Reinforcement Learning Agents based on Network Service Proximitycited by 1
  16. 2025Stackelberg Coupling of Online Representation Learning and Reinforcement Learningcited by 1
  17. 2023Policy-Induced Self-Supervision Improves Representation Finetuning in Visual RLcited by 0
  18. 2024Highway Graph to Accelerate Reinforcement Learningcited by 0
  19. 2025Enhance Exploration in Safe Reinforcement Learning with Contrastive Representation Learningcited by 0
  20. 2025Gaze on the Prize: Shaping Visual Attention with Return-Guided Contrastive Learningcited by 0
  21. 2026Scalable Dexterous Robot Learning with AR-based Remote Human-Robot Interactionscited by 0
  22. 2026SAVGO: Learning State-Action Value Geometry with Cosine Similarity for Continuous Controlcited by 0
  23. 2026Robust Scene Transfer for PointGoal Navigation via Privileged Sensor Guided Contrastive Learningcited by 0
  24. 2026Mask-based Predictive Representations for Reinforcement Learningcited by 0