mlsanityData to 7 Oct 2026

Learning Representations for Pixel-based Control: What Matters and Why?

arXiv 2111.07775 ↗First posted 15 Nov 2021cs.LG cs.AI cs.CVcited by 24

Not on the map. Why?

Abstract (arXiv)

Learning representations for pixel-based control has garnered significant attention recently in reinforcement learning. A wide range of methods have been proposed to enable efficient learning, leading to sample complexities similar to those in the full state setting. However, moving beyond carefully curated pixel data sets (centered crop, appropriate lighting, clear background, etc.) remains challenging. In this paper, we adopt a more difficult setting, incorporating background distractors, as a first step towards addressing this challenge. We present a simple baseline approach that can learn meaningful representations with no metric-based learning, no data augmentations, no world-model learning, and no contrastive learning. We then analyze when and why previously proposed methods are likely to fail or reduce to the same performance as the baseline in this harder setting and why we should think carefully about extending such methods beyond the well curated environments. Our results show that finer categorization of benchmarks on the basis of characteristics like density of reward, planning horizon of the problem, presence of task-irrelevant components, etc., is crucial in evaluating algorithms. Based on these observations, we propose different metrics to consider when evaluating an algorithm on benchmark tasks. We hope such a data-centric view can motivate researchers to rethink representation learning when investigating how to best apply RL to real-world tasks.

Key facts. Cited by 24 papers in the map (data through 7 Oct 2026). Builds on 51 of its 67 references that are in the map. First posted on arXiv 15 Nov 2021.

Builds on

51 of 67 references.

References · 51

  1. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  2. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  3. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,712
  4. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  5. 2020Exploring Simple Siamese Representation Learningcited by 2,853
  6. 2017The Kinetics Human Action Video Datasetcited by 1,894
  7. 2021Barlow Twins: Self-Supervised Learning via Redundancy Reductioncited by 1,818
  8. 2019Dream to Control: Learning Behaviors by Latent Imaginationcited by 1,598
  9. 2018World Modelscited by 1,444
  10. 2018Learning Latent Dynamics for Planning from Pixelscited by 1,221
  11. 2020Mastering Atari with Discrete World Modelscited by 990
  12. 2018DeepMind Control Suitecited by 935
38 more
  1. 2021Deep Reinforcement Learning at the Edge of the Statistical Precipicecited by 781
  2. 2017Rainbow: Combining Improvements in Deep Reinforcement Learningcited by 774
  3. 2020CURL: Contrastive Unsupervised Representations for Reinforcement Learningcited by 745
  4. 2017Deep Reinforcement Learning that Matterscited by 727
  5. 2016Unifying Count-Based Exploration and Intrinsic Motivationcited by 632
  6. 2020Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixelscited by 573
  7. 2019On Variational Bounds of Mutual Informationcited by 550
  8. 2019Model-Based Reinforcement Learning for Ataricited by 495
  9. 2020Reinforcement Learning with Augmented Datacited by 452
  10. 2020Learning Invariant Representations for Reinforcement Learning without Reconstructioncited by 424
  11. 2016Reinforcement Learning with Unsupervised Auxiliary Taskscited by 400
  12. 2020Data-Efficient Reinforcement Learning with Self-Predictive Representationscited by 309
  13. 2019Improving Sample Efficiency in Model-Free Reinforcement Learning from Imagescited by 268
  14. 2015Embed to Control: A Locally Linear Latent Dynamics Model for Control from Raw Imagescited by 266
  15. 2020Decoupling Representation Learning from Reinforcement Learningcited by 263
  16. 2020Model-Based Reinforcement Learning with Value-Targeted Regressioncited by 233
  17. 2019Stochastic Latent Actor-Critic: Deep Reinforcement Learning with a Latent Variable Modelcited by 228
  18. 2019DeepMDP: Learning Continuous Latent Space Models for Representation Learningcited by 204
  19. 2021Reinforcement Learning with Prototypical Representationscited by 179
  20. 2015Action-Conditional Video Prediction using Deep Networks in Atari Gamescited by 171
  21. 2019Provably efficient RL with Rich Observations via Latent State Decodingcited by 169
  22. 2020What Matters In On-Policy Reinforcement Learning? A Large-Scale Empirical Studycited by 157
  23. 2019Scalable methods for computing state similarity in deterministic Markov Decision Processescited by 133
  24. 2021Contrastive Behavioral Similarity Embeddings for Generalization in Reinforcement Learningcited by 123
  25. 2019When to use parametric models in reinforcement learning?cited by 118
  26. 2020Invariant Causal Prediction for Block MDPscited by 104
  27. 2021The Distracting Control Suite -- A Challenging Benchmark for Reinforcement Learning from Pixelscited by 97
  28. 2020Automatic Data Augmentation for Generalization in Deep Reinforcement Learningcited by 71
  29. 2021Learning Task Informed Abstractionscited by 58
  30. 2020Predictive Information Accelerates Learning in RLcited by 56
  31. 2021Temporal Predictive Coding For Model-Based Planning In Latent Spacecited by 54
  32. 2015From Pixels to Torques: Policy Learning with Deep Dynamical Modelscited by 44
  33. 2018Natural Environment Benchmarks for Reinforcement Learningcited by 32
  34. 2021DRIBO: Robust Deep Reinforcement Learning via Multi-View Information Bottleneckcited by 31
  35. 2020Measuring Visual Generalization in Continuous Control from Pixelscited by 20
  36. 2021Model-Invariant State Abstractions for Model-Based Reinforcement Learningcited by 18
  37. 2020Ecological Reinforcement Learningcited by 13
  38. 2020Learning Discrete State Abstractions With Deep Variational Inferencecited by 10

50 most cited of 51 All 51 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 4; 2023: 6; 2024: 9; 2025: 4; 2026: 1.

24 papers.

Sort:

By area:

  1. 2024Bridging State and History Representations: Understanding Self-Predictive RLcited by 52
  2. 2025Latent Action Learning Requires Supervision in the Presence of Distractorscited by 40
  3. 2023Revisiting Plasticity in Visual Reinforcement Learning: Data, Modules and Training Stagescited by 27
  4. 2022A Comprehensive Survey of Data Augmentation in Visual Reinforcement Learningcited by 23
  5. 2025A Survey of State Representation Learning for Deep Reinforcement Learningcited by 17
  6. 2022Light-weight probing of unsupervised representations for Reinforcement Learningcited by 15
  7. 2022Agent-Controller Representations: Principled Offline RL with Rich Exogenous Informationcited by 15
  8. 2024When does Self-Prediction help? Understanding Auxiliary Tasks in Reinforcement Learningcited by 13
  9. 2023Distributionally Robust Statistical Verification with Imprecise Neural Networkscited by 12
  10. 2023Ignorance is Bliss: Robust Control via Information Gatingcited by 10
  11. 2022A Mixture of Surprises for Unsupervised Reinforcement Learningcited by 9
  12. 2023Combining Reconstruction and Contrastive Methods for Multimodal Representations in RLcited by 9
12 more
  1. 2024iQRL -- Implicitly Quantized Representations for Sample-efficient Reinforcement Learningcited by 9
  2. 2024KalMamba: Towards Efficient Probabilistic State Space Models for RL under Uncertaintycited by 9
  3. 2023How Safe Am I Given What I See? Calibrated Prediction of Safety Chances for Image-Controlled Autonomycited by 8
  4. 2024The Surprising Ineffectiveness of Pre-Trained Visual Representations for Model-Based Reinforcement Learningcited by 7
  5. 2024Revisiting Data Augmentation in Deep Reinforcement Learningcited by 5
  6. 2024Sliding Puzzles Gym: A Scalable Benchmark for State Representation in Visual Reinforcement Learningcited by 3
  7. 2025AFFORD2ACT: Affordance-Guided Automatic Keypoint Selection for Generalizable and Lightweight Robotic Manipulationcited by 3
  8. 2023Differentially Encoded Observation Spaces for Perceptive Reinforcement Learningcited by 1
  9. 2024Identifiable Representation and Model Learning for Latent Dynamic Systemscited by 1
  10. 2024Uncovering RL Integration in SSL Loss: Objective-Specific Implications for Data-Efficient RLcited by 0
  11. 2025Self-Consistent Model-based Adaptation for Visual Reinforcement Learningcited by 0
  12. 2026Squeezing More from the Stream : Learning Representation Online for Streaming Reinforcement Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024DMC-VB: A Benchmark for Representation Learning for Control with Visual Distractors
  2. 2021Robust Robotic Control from Pixels using Contrastive Recurrent State-Space Models
  3. 2022Task-Induced Representation Learning
  4. 2024Learning Latent Dynamic Robust Representations for World Models
  5. 2022Behavior Prior Representation learning for Offline Reinforcement Learning
  6. 2022Reinforcement Learning with Action-Free Pre-Training from Videos
  7. 2025Pixel Motion as Universal Representation for Robot Control
  8. 2023Harnessing Discrete Representations For Continual Reinforcement Learning