mlsanityData to 14 Oct 2026

Performance Variation in Deep Reinforcement Learning

arXiv 2606.06746 ↗First posted 4 Jun 2026cs.LGcited by 2

On the map: Deep reinforcement learning methods → Deep RL algorithms and representations

Abstract (arXiv)

Deep reinforcement learning (RL) algorithms often suffer from low run-to-run robustness, manifesting as significant performance variation across independent runs of identically configured agents. Although this issue poses a spectrum of challenges across research and practice, relatively few studies develop methods to evaluate it; RL research instead often reports uncertainty in the estimated mean performance. In this paper, we outline the limitations of conventional uncertainty and variation estimates, particularly their misalignment with purpose and the risk of underreporting. We then propose an alternative percentile-based statistic and visualization method, min-max IPR and run-wise percentile highlighting, respectively. These percentile-based tools are easy to interpret and rely on standard properties of sample percentiles, providing rich information about run-to-run performance variation. We demonstrate this through three case studies. First, we show that LayerNorm and penultimate-layer normalizations narrow performance variation in PPO, whereas the variation is mostly unchanged in SAC. Second, we compare PPO, SAC, TD-MPC, and TD-MPC2, and show TD-MPC exhibits the least variation while being the most data efficient among the four. Finally, in a comparison of DQN and Rainbow on five Atari environments, we show that both algorithms exhibit similar levels of performance variation.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in Deep RL algorithms and representations, one of 15 directions in Deep reinforcement learning methods. Builds on 27 of its 43 references that are in the map. First posted on arXiv 4 Jun 2026.

Builds on

27 of 43 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024SimBa: Simplicity Bias for Scaling Up Parameters in Deep Reinforcement Learning13 Oct
  2. 2024Improving Deep Reinforcement Learning by Reducing the Chain Effect of Value and Policy Churn7 Sep

Other references · 25

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2016Layer Normalizationcited by 5,669
  3. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,885
  4. 2018DeepMind Control Suitecited by 956
  5. 2021Deep Reinforcement Learning at the Edge of the Statistical Precipicecited by 807
  6. 2017Rainbow: Combining Improvements in Deep Reinforcement Learningcited by 786
  7. 2017Deep Reinforcement Learning that Matterscited by 754
  8. 2023TD-MPC2: Scalable, Robust World Models for Continuous Controlcited by 512
  9. 2015Deep Recurrent Q-Learning for Partially Observable MDPscited by 465
  10. 2022Temporal Difference Learning for Model Predictive Controlcited by 397
  11. 2017Revisiting the Arcade Learning Environment: Evaluation Protocols and Open Problems for General Agentscited by 269
  12. 2023Learning Agile Soccer Skills for a Bipedal Robot with Deep Reinforcement Learningcited by 133
13 more
  1. 2017Reproducibility of Benchmarked Deep Reinforcement Learning Tasks for Continuous Controlcited by 93
  2. 2018How Many Random Seeds? Statistical Power Analysis in Deep Reinforcement Learning Experimentscited by 60
  3. 2023Hyperparameters in Reinforcement Learning and How To Tune Themcited by 59
  4. 2023Empirical Design in Reinforcement Learningcited by 55
  5. 2022Atari-5: Distilling the Arcade Learning Environment down to Five Gamescited by 44
  6. 2018Accelerated Methods for Deep Reinforcement Learningcited by 40
  7. 2018Setting up a Reinforcement Learning Task with a Real-World Robotcited by 33
  8. 2021Is High Variance Unavoidable in RL? A Case Study in Continuous Controlcited by 28
  9. 2017Pitfalls and Best Practices in Algorithm Configurationcited by 25
  10. 2015State of the Art Control of Atari Games Using Shallow Reinforcement Learningcited by 23
  11. 2019Let's Play Again: Variability of Deep Reinforcement Learning Agents in Atari Environmentscited by 11
  12. 2020Quantity vs. Quality: On Hyperparameter Optimization for Deep Reinforcement Learningcited by 5
  13. 2023AdaStop: adaptive statistical testing for sound comparisons of Deep RL agentscited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026Benchmarking Action Spaces in Reinforcement Learning for Vision-based Robotic Manipulationcited by 1
  2. 2026Rethinking the Suitability of Reinforcement Learning Algorithms Under Practical Transfer Constraintscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Behavior-Consistent Deep Reinforcement Learning
  2. 2026Robust Policy Optimization via Adversarial Importance Sampling
  3. 2025Rethinking RL Evaluation: Can Benchmarks Truly Reveal Failures of RL Methods?
  4. 2026The Low-Rank Structure of VLA Reinforcement Learning
  5. 2024Open RL Benchmark: Comprehensive Tracked Experiments for Reinforcement Learning
  6. 2025Measures of Variability for Risk-averse Policy Gradient
  7. 2024Improving Generalization on the ProcGen Benchmark with Simple Architectural Changes and Scale
  8. 2026DataFlex-RL: An Evaluation Platform for RLVR Data Policies