mlsanityData to 7 Oct 2026

Improving Sample Efficiency in Model-Free Reinforcement Learning from Images

arXiv 1910.01741 ↗First posted 2 Oct 2019cs.LG cs.AI cs.ROcited by 268

Not on the map. Why?

Cited by 23 in Offline RL and imitation learning from data, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Training an agent to solve control tasks directly from high-dimensional images with model-free reinforcement learning (RL) has proven difficult. A promising approach is to learn a latent representation together with the control policy. However, fitting a high-capacity encoder using a scarce reward signal is sample inefficient and leads to poor performance. Prior work has shown that auxiliary losses, such as image reconstruction, can aid efficient representation learning. However, incorporating reconstruction loss into an off-policy learning algorithm often leads to training instability. We explore the underlying reasons and identify variational autoencoders, used by previous investigations, as the cause of the divergence. Following these findings, we propose effective techniques to improve training stability. This results in a simple approach capable of matching state-of-the-art model-free and model-based algorithms on MuJoCo control tasks. Furthermore, our approach demonstrates robustness to observational noise, surpassing existing approaches in this setting. Code, results, and videos are anonymously available at https://sites.google.com/view/sac-ae/home.

Key facts. Cited by 268 papers in the map (data through 7 Oct 2026). First posted on arXiv 2 Oct 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2020: 2; 2021: 56; 2022: 61; 2023: 57; 2024: 41; 2025: 37; 2026: 14.

268 papers.

Sort:

By area:

  1. 2023Mastering Diverse Domains through World Modelscited by 1,193
  2. 2021Deep Reinforcement Learning at the Edge of the Statistical Precipicecited by 781
  3. 2021Domain Generalization: A Surveycited by 760
  4. 2021Mastering Visual Continuous Control: Improved Data-Augmented Reinforcement Learningcited by 359
  5. 2022Masked Visual Pre-training for Motor Controlcited by 246
  6. 2022The Primacy Bias in Deep Reinforcement Learningcited by 242
  7. 2022Contrastive Learning as Goal-Conditioned Reinforcement Learningcited by 241
  8. 2021Mastering Atari Games with Limited Datacited by 235
  9. 2021Reinforcement Learning with Prototypical Representationscited by 179
  10. 2022Masked World Models for Visual Controlcited by 169
  11. 2021URLB: Unsupervised Reinforcement Learning Benchmarkcited by 138
  12. 2021Coarse-to-Fine Q-attention: Efficient Learning for Visual Robotic Manipulation via Discretisationcited by 120
38 more
  1. 2021State Entropy Maximization with Random Encoders for Efficient Explorationcited by 107
  2. 2022Reinforcement Learning with Action-Free Pre-Training from Videoscited by 107
  3. 2021Stabilizing Deep Q-Learning with ConvNets and Vision Transformers under Data Augmentationcited by 101
  4. 2023UniDexGrasp++: Improving Dexterous Grasping Policy Learning via Geometry-aware Curriculum and Iterative Generalist-Specialist Learningcited by 101
  5. 2021Learning by Watching: Physical Imitation of Manipulation Skills from Human Videoscited by 87
  6. 2021S4RL: Surprisingly Simple Self-Supervision for Offline Reinforcement Learningcited by 85
  7. 2021Q-attention: Enabling Efficient Learning for Vision-based Robotic Manipulationcited by 85
  8. 2021Decoupling Value and Policy for Generalization in Reinforcement Learningcited by 81
  9. 2023For SALE: State-Action Representation Learning for Deep Reinforcement Learningcited by 73
  10. 2021Learning Vision-Guided Quadrupedal Locomotion End-to-End with Cross-Modal Transformerscited by 71
  11. 2022CIC: Contrastive Intrinsic Control for Unsupervised Skill Discoverycited by 66
  12. 2023When Do Transformers Shine in RL? Decoupling Memory from Credit Assignmentcited by 62
  13. 2021Learning Task Informed Abstractionscited by 58
  14. 2023TACO: Temporal Latent Action-Driven Contrastive Loss for Visual Reinforcement Learningcited by 56
  15. 2022On Pre-Training for Visuo-Motor Control: Revisiting a Learning-from-Scratch Baselinecited by 55
  16. 2022Iso-Dream: Isolating and Leveraging Noncontrollable Visual Dynamics in World Modelscited by 53
  17. 2024Bridging State and History Representations: Understanding Self-Predictive RLcited by 52
  18. 2024LLaRA: Supercharging Robot Learning Data for Vision-Language Policycited by 51
  19. 2022Look Closer: Bridging Egocentric and Third-Person Views with Transformers for Robotic Manipulationcited by 49
  20. 2021DRIVE: Deep Reinforced Accident Anticipation with Visual Explanationcited by 46
  21. 2022Learning Generalizable Dexterous Manipulation from Human Grasp Affordancecited by 46
  22. 2022Visual Reinforcement Learning with Self-Supervised 3D Representationscited by 45
  23. 2021MICo: Improved representations via sampling-based state similarity for Markov decision processescited by 43
  24. 2022Reinforcement Learning with Neural Radiance Fieldscited by 43
  25. 2022Visuo-Tactile Transformers for Manipulationcited by 43
  26. 2021Learning Markov State Abstractions for Deep Reinforcement Learningcited by 42
  27. 2021SECANT: Self-Expert Cloning for Zero-Shot Generalization of Visual Policiescited by 42
  28. 2023PLASTIC: Improving Input and Label Plasticity for Sample Efficient Reinforcement Learningcited by 42
  29. 2021PlayVirtual: Augmenting Cycle-Consistent Virtual Trajectories for Reinforcement Learningcited by 41
  30. 2021MADE: Exploration via Maximizing Deviation from Explored Regionscited by 40
  31. 2021AutoFL: Enabling Heterogeneity-Aware Energy Efficient Federated Learningcited by 39
  32. 2022Mask-based Latent Reconstruction for Reinforcement Learningcited by 39
  33. 2022Bisimulation Makes Analogies in Goal-Conditioned Reinforcement Learningcited by 38
  34. 2022Learning to Drive by Watching YouTube Videos: Action-Conditioned Contrastive Policy Pretrainingcited by 37
  35. 2023Crossway Diffusion: Improving Diffusion-based Visuomotor Policy via Self-supervised Learningcited by 37
  36. 2021Return-Based Contrastive Representation Learning for Reinforcement Learningcited by 36
  37. 2023AMAGO: Scalable In-Context Reinforcement Learning for Adaptive Agentscited by 36
  38. 2022On Transforming Reinforcement Learning by Transformer: The Development Trajectorycited by 35

50 most cited of 268 All 268 (JSON)