mlsanityData to 14 Oct 2026

Is Exploration or Optimization the Problem for Deep Reinforcement Learning?

arXiv 2508.01329 ↗First posted 2 Aug 2025cs.LG cs.AIcited by 1

On the map: Deep reinforcement learning methods → Deep RL algorithms and representations

Abstract (arXiv)

In the era of deep reinforcement learning, making progress is more complex, as the collected experience must be compressed into a deep model for future exploitation and sampling. Many papers have shown that training a deep learning policy under the changing state and action distribution leads to sub-optimal performance, or even collapse. This naturally leads to the concern that even if the community creates improved exploration algorithms or reward objectives, will those improvements fall on the textitdeaf ears of optimization difficulties. This work proposes a new textitpractical sub-optimality estimator to determine optimization limitations of deep reinforcement learning algorithms. Through experiments across environments and RL algorithms, it is shown that the difference between the best experience generated is 2-3× better than the policies' learned performance. This large difference indicates that deep RL methods only exploit half of the good experience they generate.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Deep RL algorithms and representations, one of 15 directions in Deep reinforcement learning methods. Builds on 49 of its 60 references that are in the map. First posted on arXiv 2 Aug 2025.

Builds on

49 of 60 references.

Closest ancestors · 5

Same direction, newest first

  1. 2024Improving Deep Reinforcement Learning by Reducing the Chain Effect of Value and Policy Churn7 Sep
  2. 2024Normalization and effective learning rates in reinforcement learning1 Jul
  3. 2024Bigger, Regularized, Optimistic: scaling for compute and sample-efficient continuous control25 May
  4. 2024Stop Regressing: Training Value Functions via Classification for Scalable Deep RL6 Mar
  5. 2024Mixtures of Experts Unlock Parameter Scaling for Deep RL13 Feb

Other references · 44

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  3. 2015Continuous control with deep reinforcement learningcited by 3,407
  4. 2015Trust Region Policy Optimizationcited by 2,580
  5. 2018Addressing Function Approximation Error in Actor-Critic Methodscited by 2,417
  6. 2015Deep Reinforcement Learning with Double Q-learningcited by 1,745
  7. 2017Curiosity-driven Exploration by Self-supervised Predictioncited by 1,194
  8. 2019Dota 2 with Large Scale Deep Reinforcement Learningcited by 1,103
  9. 2015Prioritized Experience Replaycited by 1,099
  10. 2018Exploration by Random Network Distillationcited by 993
  11. 2021Deep Reinforcement Learning at the Edge of the Statistical Precipicecited by 807
  12. 2017Rainbow: Combining Improvements in Deep Reinforcement Learningcited by 786
32 more
  1. 2017A Distributional Perspective on Reinforcement Learningcited by 765
  2. 2016Unifying Count-Based Exploration and Intrinsic Motivationcited by 639
  3. 2020Learning Invariant Representations for Reinforcement Learning without Reconstructioncited by 436
  4. 2022Temporal Difference Learning for Model Predictive Controlcited by 397
  5. 2016#Exploration: A Study of Count-Based Exploration for Deep Reinforcement Learningcited by 309
  6. 2021Randomized Ensembled Double Q-Learning: Learning Fast Without a Modelcited by 288
  7. 2017Count-Based Exploration with Neural Density Modelscited by 257
  8. 2022The Primacy Bias in Deep Reinforcement Learningcited by 247
  9. 2019AlgaeDICE: Policy Gradient from Arbitrary Experiencecited by 188
  10. 2018Deep Reinforcement Learning and the Deadly Triadcited by 174
  11. 2023Understanding plasticity in neural networkscited by 174
  12. 2020Controlling Overestimation Bias with Truncated Mixture of Continuous Distributional Quantile Criticscited by 165
  13. 2023Bigger, Better, Faster: Human-level Atari with human-level efficiencycited by 147
  14. 2020Phasic Policy Gradientcited by 123
  15. 2020GenDICE: Generalized Offline Estimation of Stationary Valuescited by 122
  16. 2021OptiDICE: Offline Policy Optimization via Stationary Distribution Correction Estimationcited by 121
  17. 2020Maxmin Q-learning: Controlling the Estimation Bias of Q-learningcited by 117
  18. 2021Decoupling Value and Policy for Generalization in Reinforcement Learningcited by 82
  19. 2020DisCor: Corrective Feedback in Reinforcement Learning via Distribution Correctioncited by 71
  20. 2021The Difficulty of Passive Learning in Deep Reinforcement Learningcited by 50
  21. 2021Breaking the Deadly Triad with a Target Networkcited by 47
  22. 2022Atari-5: Distilling the Arcade Learning Environment down to Five Gamescited by 44
  23. 2019Towards Characterizing Divergence in Deep Q-Learningcited by 43
  24. 2020Interference and Generalization in Temporal Difference Learningcited by 42
  25. 2020Can Increasing Input Dimensionality Improve Deep Reinforcement Learning?cited by 37
  26. 2021HyAR: Addressing Discrete-Continuous Action Reinforcement Learning via Hybrid Action Representationcited by 35
  27. 2020BeBold: Exploration Beyond the Boundary of Explored Regionscited by 33
  28. 2022Target Network and Truncation Overcome The Deadly Triad in Q-Learningcited by 31
  29. 2022Overcoming the Spectral Bias of Neural Value Approximationcited by 19
  30. 2020Improving Performance in Reinforcement Learning by Breaking Generalization in Neural Networkscited by 17
  31. 2024Non-Stationary Learning of Neural Networks with Automatic Soft Parameter Resetcited by 17
  32. 2021Bridging the Gap Between Target Networks and Functional Regularizationcited by 7

Built on by

Cited by year

Citing papers per year: 2025: 1.

1 paper.

  1. 2025Let it Calm: Exploratory Annealed Decoding for Verifiable Reinforcement Learningcited by 10

Similar, not cited

Close in wording, no citation either way

  1. 2024Efficient Exploration in Deep Reinforcement Learning: A Novel Bayesian Actor-Critic Algorithm
  2. 2026Parameter Exploration for RLVR via Variational Learning
  3. 2024A Comprehensive Survey of Reinforcement Learning: From Algorithms to Practical Challenges
  4. 2021On Multi-objective Policy Optimization as a Tool for Reinforcement Learning: Case Studies in Offline RL and Finetuning
  5. 2022Exploration in Deep Reinforcement Learning: A Survey
  6. 2026Principled Analysis of Deep Reinforcement Learning Evaluation and Design Paradigms
  7. 2026Emergence of Exploration in Policy Gradient Reinforcement Learning via Retrying
  8. 2024Behind the Myth of Exploration in Policy Gradients