mlsanityData to 7 Oct 2026

Measuring Progress in Deep Reinforcement Learning Sample Efficiency

arXiv 2102.04881 ↗First posted 9 Feb 2021cs.LG cs.CYcited by 8

Not on the map. Why?

Abstract (arXiv)

Sampled environment transitions are a critical input to deep reinforcement learning (DRL) algorithms. Current DRL benchmarks often allow for the cheap and easy generation of large amounts of samples such that perceived progress in DRL does not necessarily correspond to improved sample efficiency. As simulating real world processes is often prohibitively hard and collecting real world experience is costly, sample efficiency is an important indicator for economically relevant applications of DRL. We investigate progress in sample efficiency on Atari games and continuous control tasks by comparing the number of samples that a variety of algorithms need to reach a given performance level according to training curves in the corresponding publications. We find exponential progress in sample efficiency with estimated doubling times of around 10 to 18 months on Atari, 5 to 24 months on state-based continuous control and of around 4 to 9 months on pixel-based continuous control depending on the specific task and performance level.

Key facts. Cited by 8 papers in the map (data through 7 Oct 2026). Builds on 93 of its 123 references that are in the map. First posted on arXiv 9 Feb 2021.

Builds on

93 of 123 references.

References · 93

  1. 2017Attention Is All You Needcited by 59,932
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2020Scaling Laws for Neural Language Modelscited by 6,411
  4. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  5. 2015Continuous control with deep reinforcement learningcited by 3,279
  6. 2013Playing Atari with Deep Reinforcement Learningcited by 3,211
  7. 2015Trust Region Policy Optimizationcited by 2,503
  8. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,302
  9. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,218
  10. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problemscited by 1,873
  11. 2015Deep Reinforcement Learning with Double Q-learningcited by 1,704
  12. 2016OpenAI Gymcited by 1,684
38 more
  1. 2019Dream to Control: Learning Behaviors by Latent Imaginationcited by 1,598
  2. 2020D4RL: Datasets for Deep Data-Driven Reinforcement Learningcited by 1,406
  3. 2019Mastering Atari, Go, Chess and Shogi by Planning with a Learned Modelcited by 1,327
  4. 2018Learning Latent Dynamics for Planning from Pixelscited by 1,221
  5. 2019Dota 2 with Large Scale Deep Reinforcement Learningcited by 1,084
  6. 2015Prioritized Experience Replaycited by 1,070
  7. 2018DeepMind Control Suitecited by 935
  8. 2017Revisiting Unreasonable Effectiveness of Data in Deep Learning Eracited by 916
  9. 2018IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architecturescited by 866
  10. 2017Rainbow: Combining Improvements in Deep Reinforcement Learningcited by 774
  11. 2015Dueling Network Architectures for Deep Reinforcement Learningcited by 769
  12. 2019When to Trust Your Model: Model-Based Policy Optimizationcited by 753
  13. 2020CURL: Contrastive Unsupervised Representations for Reinforcement Learningcited by 745
  14. 2017A Distributional Perspective on Reinforcement Learningcited by 743
  15. 2017Deep Reinforcement Learning that Matterscited by 727
  16. 2017Reinforcement Learning with Deep Energy-Based Policiescited by 659
  17. 2018Learning Dexterous In-Hand Manipulationcited by 643
  18. 2017Sim-to-Real Transfer of Robotic Control with Dynamics Randomizationcited by 616
  19. 2020Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixelscited by 573
  20. 2019Model-Based Reinforcement Learning for Ataricited by 495
  21. 2017Distributional Reinforcement Learning with Quantile Regressioncited by 464
  22. 2015Deep Recurrent Q-Learning for Partially Observable MDPscited by 457
  23. 2020Reinforcement Learning with Augmented Datacited by 452
  24. 2020Planning to Explore via Self-Supervised World Modelscited by 365
  25. 2018Implicit Quantile Networks for Distributional Reinforcement Learningcited by 362
  26. 2016Benchmarking Deep Reinforcement Learning for Continuous Controlcited by 361
  27. 2018Maximum a Posteriori Policy Optimisationcited by 318
  28. 2020Agent57: Outperforming the Atari Human Benchmarkcited by 300
  29. 2019Challenges of Real-World Reinforcement Learningcited by 282
  30. 2019Improving Sample Efficiency in Model-Free Reinforcement Learning from Imagescited by 268
  31. 2017Revisiting the Arcade Learning Environment: Evaluation Protocols and Open Problems for General Agentscited by 267
  32. 2017Noisy Networks for Explorationcited by 262
  33. 2018Distributed Prioritized Experience Replaycited by 260
  34. 2019Stochastic Latent Actor-Critic: Deep Reinforcement Learning with a Latent Variable Modelcited by 228
  35. 2019Fixing the train-test resolution discrepancycited by 209
  36. 2019Go-Explore: a New Approach for Hard-Exploration Problemscited by 207
  37. 2018Distributed Distributional Deterministic Policy Gradientscited by 195
  38. 2016Sample Efficient Actor-Critic with Experience Replaycited by 185

50 most cited of 93 All 93 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 2; 2022: 2; 2023: 0; 2024: 2; 2025: 1; 2026: 1.

8 papers.

  1. 2022The Alignment Problem from a Deep Learning Perspectivecited by 246
  2. 2022Will we run out of data? Limits of LLM scaling based on human-generated datacited by 187
  3. 2024Algorithmic progress in language modelscited by 31
  4. 2025A Survey of Reinforcement Learning for Optimization in Automationcited by 4
  5. 2021Algorithmic collusion: A critical reviewcited by 3
  6. 2024The Graph's Apprentice: Teaching an LLM Low Level Knowledge for Circuit Quality Estimationcited by 3
  7. 2021Balancing Value Underestimation and Overestimation with Realistic Actor-Criticcited by 1
  8. 2026Integrating Novelty and Surprise for Experience Prioritization and Exploration in Image-Based Reinforcement Learningcited by 0