mlsanityData to 7 Oct 2026

Decoupling Representation Learning from Reinforcement Learning

arXiv 2009.08319 ↗First posted 14 Sep 2020cs.LG cs.AI cs.CVcited by 263

Not on the map. Why?

Cited by 30 in Offline RL and imitation learning from data, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

In an effort to overcome limitations of reward-driven feature learning in deep reinforcement learning (RL) from images, we propose decoupling representation learning from policy learning. To this end, we introduce a new unsupervised learning (UL) task, called Augmented Temporal Contrast (ATC), which trains a convolutional encoder to associate pairs of observations separated by a short time difference, under image augmentations and using a contrastive loss. In online RL experiments, we show that training the encoder exclusively using ATC matches or outperforms end-to-end RL in most environments. Additionally, we benchmark several leading UL algorithms by pre-training encoders on expert demonstrations and using them, with weights frozen, in RL agents; we find that agents using ATC-trained encoders outperform all others. We also train multi-task encoders on data from multiple environments and show generalization to different downstream RL tasks. Finally, we ablate components of ATC, and introduce a new data augmentation to enable replay of (compressed) latent images from pre-trained encoders when RL requires augmentation. Our experiments span visually diverse RL benchmarks in DeepMind Control, DeepMind Lab, and Atari, and our complete code is available at https://github.com/astooke/rlpyt/tree/master/rlpyt/ul.

Key facts. Cited by 263 papers in the map (data through 7 Oct 2026). First posted on arXiv 14 Sep 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 62; 2022: 63; 2023: 52; 2024: 46; 2025: 26; 2026: 14.

263 papers.

Sort:

By area:

  1. 2021Mastering Visual Continuous Control: Improved Data-Augmented Reinforcement Learningcited by 359
  2. 2022Contrastive Learning as Goal-Conditioned Reinforcement Learningcited by 241
  3. 2021Reinforcement Learning with Prototypical Representationscited by 179
  4. 2021Behavior From the Void: Unsupervised Active Pre-Trainingcited by 173
  5. 2022The Unsurprising Effectiveness of Pre-Trained Vision Models for Controlcited by 155
  6. 2021The Surprising Effectiveness of Representation Learning for Visual Imitationcited by 149
  7. 2021URLB: Unsupervised Reinforcement Learning Benchmarkcited by 138
  8. 2021Representation Learning for Online and Offline RL in Low-rank MDPscited by 128
  9. 2021Contrastive Behavioral Similarity Embeddings for Generalization in Reinforcement Learningcited by 123
  10. 2024Survey on Large Language Model-Enhanced Reinforcement Learning: Concept, Taxonomy, and Methodscited by 114
  11. 2021State Entropy Maximization with Random Encoders for Efficient Explorationcited by 107
  12. 2022Reinforcement Learning with Action-Free Pre-Training from Videoscited by 107
38 more
  1. 2021B-Pref: Benchmarking Preference-Based Reinforcement Learningcited by 106
  2. 2021Pretraining Representations for Data-Efficient Reinforcement Learningcited by 105
  3. 2021Why Generalization in RL is Difficult: Epistemic POMDPs and Implicit Partial Observabilitycited by 102
  4. 2021Stabilizing Deep Q-Learning with ConvNets and Vision Transformers under Data Augmentationcited by 101
  5. 2023UniDexGrasp++: Improving Dexterous Grasping Policy Learning via Geometry-aware Curriculum and Iterative Generalist-Specialist Learningcited by 101
  6. 2021Representation Matters: Offline Pretraining for Sequential Decision Makingcited by 90
  7. 2023DREAMWALKER: Mental Planning for Continuous Vision-Language Navigationcited by 82
  8. 2021Decoupling Value and Policy for Generalization in Reinforcement Learningcited by 81
  9. 2023Multi-View Masked World Models for Visual Robotic Manipulationcited by 78
  10. 2022Offline Visual Representation Learning for Embodied Navigationcited by 74
  11. 2021DreamerPro: Reconstruction-Free Model-Based Reinforcement Learning with Prototypical Representationscited by 73
  12. 2023For SALE: State-Action Representation Learning for Deep Reinforcement Learningcited by 73
  13. 2021Learning Vision-Guided Quadrupedal Locomotion End-to-End with Cross-Modal Transformerscited by 71
  14. 2022CIC: Contrastive Intrinsic Control for Unsupervised Skill Discoverycited by 66
  15. 2021A Survey of Deep RL and IL for Autonomous Driving Policy Learningcited by 59
  16. 2021Learning Task Informed Abstractionscited by 58
  17. 2022Pre-Trained Image Encoder for Generalizable Visual Reinforcement Learningcited by 58
  18. 2021On The Effect of Auxiliary Tasks on Representation Dynamicscited by 56
  19. 2023TACO: Temporal Latent Action-Driven Contrastive Loss for Visual Reinforcement Learningcited by 56
  20. 2022End-to-End Affordance Learning for Robotic Manipulationcited by 54
  21. 2022VRL3: A Data-Driven Framework for Visual Deep Reinforcement Learningcited by 51
  22. 2021Towards Robust Bisimulation Metric Learningcited by 49
  23. 2021Towards More Generalizable One-shot Visual Imitation Learningcited by 48
  24. 2021Improving Transferability of Representations via Augmentation-Aware Self-Supervisioncited by 48
  25. 2022Learning Generalizable Dexterous Manipulation from Human Grasp Affordancecited by 46
  26. 2022Visual Reinforcement Learning with Self-Supervised 3D Representationscited by 45
  27. 2023DrM: Mastering Visual Reinforcement Learning through Dormant Ratio Minimizationcited by 45
  28. 2022Masked Autoencoding for Scalable and Generalizable Decision Makingcited by 44
  29. 2023SMART: Self-supervised Multi-task pretrAining with contRol Transformerscited by 44
  30. 2021Curious Representation Learning for Embodied Intelligencecited by 43
  31. 2022Reinforcement Learning with Neural Radiance Fieldscited by 43
  32. 2022Visuo-Tactile Transformers for Manipulationcited by 43
  33. 2021Learning Markov State Abstractions for Deep Reinforcement Learningcited by 42
  34. 2021SECANT: Self-Expert Cloning for Zero-Shot Generalization of Visual Policiescited by 42
  35. 2023PLASTIC: Improving Input and Label Plasticity for Sample Efficient Reinforcement Learningcited by 42
  36. 2021PlayVirtual: Augmenting Cycle-Consistent Virtual Trajectories for Reinforcement Learningcited by 41
  37. 2023Proto-Value Networks: Scaling Representation Learning with Auxiliary Taskscited by 41
  38. 2023PartManip: Learning Cross-Category Generalizable Part Manipulation Policy from Point Cloud Observationscited by 40

50 most cited of 263 All 263 (JSON)