mlsanityData to 7 Oct 2026

Reinforcement Learning with Prototypical Representations

arXiv 2102.11271 ↗First posted 22 Feb 2021cs.LG cs.AIcited by 179

Not on the map. Why?

Abstract (arXiv)

Learning effective representations in image-based environments is crucial for sample efficient Reinforcement Learning (RL). Unfortunately, in RL, representation learning is confounded with the exploratory experience of the agent -- learning a useful representation requires diverse data, while effective exploration is only possible with coherent representations. Furthermore, we would like to learn representations that not only generalize across tasks but also accelerate downstream exploration for efficient task-specific training. To address these challenges we propose Proto-RL, a self-supervised framework that ties representation learning with exploration through prototypical representations. These prototypes simultaneously serve as a summarization of the exploratory experience of an agent as well as a basis for representing observations. We pre-train these task-agnostic representations and prototypes on environments without downstream task information. This enables state-of-the-art downstream policy learning on a set of difficult continuous control tasks.

Key facts. Cited by 179 papers in the map (data through 7 Oct 2026). Builds on 46 of its 61 references that are in the map. First posted on arXiv 22 Feb 2021.

Builds on

46 of 61 references.

References · 46

  1. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  2. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  3. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  4. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  5. 2016Layer Normalizationcited by 5,579
  6. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,712
  7. 2015Continuous control with deep reinforcement learningcited by 3,279
  8. 2013Playing Atari with Deep Reinforcement Learningcited by 3,211
  9. 2020Unsupervised Learning of Visual Features by Contrasting Cluster Assignmentscited by 2,845
  10. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
  11. 2013Sinkhorn Distances: Lightspeed Computation of Optimal Transportation Distancescited by 2,277
  12. 2018Unsupervised Feature Learning via Non-Parametric Instance-level Discriminationcited by 1,838
34 more
  1. 2015Deep Reinforcement Learning with Double Q-learningcited by 1,704
  2. 2019Dream to Control: Learning Behaviors by Latent Imaginationcited by 1,598
  3. 2018Unsupervised Representation Learning by Predicting Image Rotationscited by 1,586
  4. 2016Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzlescited by 1,379
  5. 2018Learning Latent Dynamics for Planning from Pixelscited by 1,221
  6. 2018Soft Actor-Critic Algorithms and Applicationscited by 1,179
  7. 2017Curiosity-driven Exploration by Self-supervised Predictioncited by 1,173
  8. 2017Hindsight Experience Replaycited by 1,096
  9. 2015Unsupervised Visual Representation Learning by Context Predictioncited by 1,062
  10. 2018Exploration by Random Network Distillationcited by 966
  11. 2018DeepMind Control Suitecited by 935
  12. 2015End-to-End Training of Deep Visuomotor Policiescited by 847
  13. 2020CURL: Contrastive Unsupervised Representations for Reinforcement Learningcited by 745
  14. 2019Data-Efficient Image Recognition with Contrastive Predictive Codingcited by 672
  15. 2016Unifying Count-Based Exploration and Intrinsic Motivationcited by 632
  16. 2020Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixelscited by 573
  17. 2019Self-labelling via simultaneous clustering and representation learningcited by 525
  18. 2020Reinforcement Learning with Augmented Datacited by 452
  19. 2016Reinforcement Learning with Unsupervised Auxiliary Taskscited by 400
  20. 2020Planning to Explore via Self-Supervised World Modelscited by 365
  21. 2019Learning Correspondence from the Cycle-Consistency of Timecited by 269
  22. 2019Improving Sample Efficiency in Model-Free Reinforcement Learning from Imagescited by 268
  23. 2020Decoupling Representation Learning from Reinforcement Learningcited by 263
  24. 2017Count-Based Exploration with Neural Density Modelscited by 255
  25. 2018Provably Efficient Maximum Entropy Explorationcited by 240
  26. 2019Stochastic Latent Actor-Critic: Deep Reinforcement Learning with a Latent Variable Modelcited by 228
  27. 2016Split-Brain Autoencoders: Unsupervised Learning by Cross-Channel Predictioncited by 217
  28. 2019Efficient Exploration via State Marginal Matchingcited by 186
  29. 2016Learning to Poke by Poking: Experiential Learning of Intuitive Physicscited by 152
  30. 2020Visual Imitation Made Easycited by 111
  31. 2020Learning Predictive Representations for Deformable Objects Using Contrastive Estimationcited by 72
  32. 2016The Curious Robot: Learning Visual Representations via Physical Interactionscited by 37
  33. 2020Robust Policies via Mid-Level Visual Representations: An Experimental Study in Manipulation and Navigationcited by 31
  34. 2020Task-Agnostic Exploration via Policy Gradient of a Non-Parametric State Entropy Estimatecited by 21

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 28; 2022: 50; 2023: 43; 2024: 22; 2025: 21; 2026: 15.

179 papers.

Sort:

By area:

  1. 2021Mastering Visual Continuous Control: Improved Data-Augmented Reinforcement Learningcited by 359
  2. 2023Guiding Pretraining in Reinforcement Learning with Large Language Modelscited by 185
  3. 2021Behavior From the Void: Unsupervised Active Pre-Trainingcited by 173
  4. 2022Masked World Models for Visual Controlcited by 169
  5. 2022The Unsurprising Effectiveness of Pre-Trained Vision Models for Controlcited by 155
  6. 2021The Surprising Effectiveness of Representation Learning for Visual Imitationcited by 149
  7. 2021URLB: Unsupervised Reinforcement Learning Benchmarkcited by 138
  8. 2021APS: Active Pretraining with Successor Featurescited by 109
  9. 2022Reinforcement Learning with Action-Free Pre-Training from Videoscited by 107
  10. 2021Pretraining Representations for Data-Efficient Reinforcement Learningcited by 105
  11. 2021Stabilizing Deep Q-Learning with ConvNets and Vision Transformers under Data Augmentationcited by 101
  12. 2025Maximizing Confidence Alone Improves Reasoningcited by 97
38 more
  1. 2022Don't Change the Algorithm, Change the Data: Exploratory Data for Offline Reinforcement Learningcited by 93
  2. 2024Foundation Policies with Hilbert Representationscited by 89
  3. 2023METRA: Scalable Unsupervised RL with Metric-Aware Abstractioncited by 80
  4. 2022Does Zero-Shot Reinforcement Learning Exist?cited by 79
  5. 2023XSkill: Cross Embodiment Skill Discoverycited by 74
  6. 2021DreamerPro: Reconstruction-Free Model-Based Reinforcement Learning with Prototypical Representationscited by 73
  7. 2022CIC: Contrastive Intrinsic Control for Unsupervised Skill Discoverycited by 66
  8. 2023Dexterity from Touch: Self-Supervised Pre-Training of Tactile Representations with Robotic Playcited by 60
  9. 2022Pre-Trained Image Encoder for Generalizable Visual Reinforcement Learningcited by 58
  10. 2021Exploration in Deep Reinforcement Learning: From Single-Agent to Multiagent Domaincited by 57
  11. 2023TACO: Temporal Latent Action-Driven Contrastive Loss for Visual Reinforcement Learningcited by 56
  12. 2025Zero-Shot Whole-Body Humanoid Control via Behavioral Foundation Modelscited by 51
  13. 2023Controllability-Aware Unsupervised Skill Discoverycited by 49
  14. 2022Masked Autoencoding for Scalable and Generalizable Decision Makingcited by 44
  15. 2021Learning Markov State Abstractions for Deep Reinforcement Learningcited by 42
  16. 2022Contextualize Me -- The Case for Context in Reinforcement Learningcited by 41
  17. 2021MADE: Exploration via Maximizing Deviation from Explored Regionscited by 40
  18. 2023Masked Trajectory Models for Prediction, Representation, and Controlcited by 40
  19. 2022Mask-based Latent Reconstruction for Reinforcement Learningcited by 39
  20. 2021Interesting Object, Curious Agent: Learning Task-Agnostic Explorationcited by 38
  21. 2022The Importance of Non-Markovianity in Maximum State Entropy Explorationcited by 35
  22. 2022Spectral Decomposition Representation for Reinforcement Learningcited by 35
  23. 2021HyAR: Addressing Discrete-Continuous Action Reinforcement Learning via Hybrid Action Representationcited by 33
  24. 2023Emergent Agentic Transformer from Chain of Hindsight Experiencecited by 33
  25. 2022Choreographer: Learning and Adapting Skills in Imaginationcited by 31
  26. 2021Procedural Generalization by Planning with Self-Supervised World Modelscited by 30
  27. 2022Challenging Common Assumptions in Convex Reinforcement Learningcited by 29
  28. 2022The Challenges of Exploration for Offline Reinforcement Learningcited by 26
  29. 2023Behavior Contrastive Learning for Unsupervised Skill Discoverycited by 26
  30. 2022An information-theoretic perspective on intrinsic motivation in reinforcement learning: a surveycited by 25
  31. 2023Learning Goal-Conditioned Policies Offline with Self-Supervised Reward Shapingcited by 25
  32. 2021Learning Representations for Pixel-based Control: What Matters and Why?cited by 24
  33. 2022A Comprehensive Survey of Data Augmentation in Visual Reinforcement Learningcited by 23
  34. 2021Curiosity-Driven Exploration via Latent Bayesian Surprisecited by 22
  35. 2023Accelerating Reinforcement Learning with Value-Conditional State Entropy Explorationcited by 22
  36. 2021Beyond Fine-Tuning: Transferring Behavior in Reinforcement Learningcited by 21
  37. 2021CARL: A Benchmark for Contextual and Adaptive Reinforcement Learningcited by 21
  38. 2021Dynamic Bottleneck for Robust Self-Supervised Explorationcited by 21

50 most cited of 179 All 179 (JSON)