mlsanityData to 7 Oct 2026

On the consistency of hyper-parameter selection in value-based deep reinforcement learning

arXiv 2406.17523 ↗First posted 25 Jun 2024cs.LG cs.AIcited by 21

Not on the map. Why?

Abstract (arXiv)

Deep reinforcement learning (deep RL) has achieved tremendous success on various domains through a combination of algorithmic design and careful selection of hyper-parameters. Algorithmic improvements are often the result of iterative enhancements built upon prior approaches, while hyper-parameter choices are typically inherited from previous methods or fine-tuned specifically for the proposed technique. Despite their crucial impact on performance, hyper-parameter choices are frequently overshadowed by algorithmic advancements. This paper conducts an extensive empirical study focusing on the reliability of hyper-parameter selection for value-based deep reinforcement learning agents, including the introduction of a new score to quantify the consistency and reliability of various hyper-parameters. Our findings not only help establish which hyper-parameters are most critical to tune, but also help clarify which tunings remain consistent across different training regimes.

Key facts. Cited by 21 papers in the map (data through 7 Oct 2026). Builds on 51 of its 80 references that are in the map. First posted on arXiv 25 Jun 2024.

Builds on

51 of 80 references.

References · 51

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
  3. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,415
  4. 2016Layer Normalizationcited by 5,579
  5. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  6. 2016Aggregated Residual Transformations for Deep Neural Networkscited by 2,540
  7. 2015Deep Reinforcement Learning with Double Q-learningcited by 1,704
  8. 2015Prioritized Experience Replaycited by 1,070
  9. 2017Sigmoid-Weighted Linear Units for Neural Network Function Approximation in Reinforcement Learningcited by 935
  10. 2021Deep Reinforcement Learning at the Edge of the Statistical Precipicecited by 781
  11. 2017Rainbow: Combining Improvements in Deep Reinforcement Learningcited by 774
  12. 2015Dueling Network Architectures for Deep Reinforcement Learningcited by 769
38 more
  1. 2017A Distributional Perspective on Reinforcement Learningcited by 743
  2. 2017Deep Reinforcement Learning that Matterscited by 727
  3. 2016Language Modeling with Gated Convolutional Networkscited by 688
  4. 2020Image Augmentation Is All You Need: Regularizing Deep Reinforcement Learning from Pixelscited by 573
  5. 2019Model-Based Reinforcement Learning for Ataricited by 495
  6. 2016Benchmarking Deep Reinforcement Learning for Continuous Controlcited by 361
  7. 2017Revisiting the Arcade Learning Environment: Evaluation Protocols and Open Problems for General Agentscited by 267
  8. 2017Noisy Networks for Explorationcited by 262
  9. 2020Implementation Matters in Deep Policy Gradients: A Case Study on PPO and TRPOcited by 184
  10. 2023The Dormant Neuron Phenomenon in Deep Reinforcement Learningcited by 171
  11. 2023Bigger, Better, Faster: Human-level Atari with human-level efficiencycited by 145
  12. 2024Stop Regressing: Training Value Functions via Classification for Scalable Deep RLcited by 136
  13. 2018Dopamine: A Research Framework for Deep Reinforcement Learningcited by 121
  14. 2019When to use parametric models in reinforcement learning?cited by 118
  15. 2020Hyperparameter Selection for Offline Reinforcement Learningcited by 113
  16. 2016Averaged-DQN: Variance Reduction and Stabilization for Deep Reinforcement Learningcited by 106
  17. 2022Discovered Policy Optimisationcited by 103
  18. 2017Reproducibility of Benchmarked Deep Reinforcement Learning Tasks for Continuous Controlcited by 92
  19. 2020Revisiting Rainbow: Promoting more Insightful and Inclusive Deep Reinforcement Learning Researchcited by 78
  20. 2024Mixtures of Experts Unlock Parameter Scaling for Deep RLcited by 75
  21. 2022Automated Reinforcement Learning (AutoRL): A Survey and Open Problemscited by 69
  22. 2021Spectral Normalisation for Deep Reinforcement Learning: an Optimisation Perspectivecited by 64
  23. 2022The State of Sparse Training in Deep Reinforcement Learningcited by 60
  24. 2023Hyperparameters in Reinforcement Learning and How To Tune Themcited by 58
  25. 2021Revisiting Design Choices in Offline Model-Based Reinforcement Learningcited by 50
  26. 2021The Difficulty of Passive Learning in Deep Reinforcement Learningcited by 50
  27. 2019Measuring the Reliability of Reinforcement Learning Algorithmscited by 49
  28. 2021Evolving Reinforcement Learning Algorithmscited by 48
  29. 2019Off-Policy Deep Reinforcement Learning by Bootstrapping the Covariate Shiftcited by 47
  30. 2020Discount Factor as a Regularizer in Reinforcement Learningcited by 44
  31. 2023Proto-Value Networks: Scaling Representation Learning with Auxiliary Taskscited by 41
  32. 2024In value-based deep reinforcement learning, a pruned network is a good networkcited by 38
  33. 2019How to Initialize your Network? Robust Initialization for WeightNorm & ResNetscited by 36
  34. 2023Small batch deep reinforcement learningcited by 35
  35. 2015How to Discount Deep Reinforcement Learning: Towards New Dynamic Strategiescited by 26
  36. 2019On Inductive Biases in Deep Reinforcement Learningcited by 22
  37. 2021Hyperparameter Selection for Imitation Learningcited by 17
  38. 2019Using a Logarithmic Mapping to Enable Lower Discount Factors in Reinforcement Learningcited by 16

50 most cited of 51 All 51 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 5; 2025: 11; 2026: 5.

21 papers.

Sort:

By area:

  1. 2025Stable Gradients for Stable Learning at Scale in Deep Reinforcement Learningcited by 22
  2. 2025Scaling Off-Policy Reinforcement Learning with Batch and Weight Normalizationcited by 19
  3. 2024MAD-TD: Model-Augmented Data stabilizes High Update Ratio RLcited by 16
  4. 2025Measure gradients, not activations! Enhancing neuronal activity in deep reinforcement learningcited by 14
  5. 2025The Impact of On-Policy Parallelized Data Collection on Deep Reinforcement Learning Networkscited by 12
  6. 2025Simplicial Embeddings Improve Sample Efficiency in Actor-Critic Agentscited by 12
  7. 2025Multi-Task Reinforcement Learning Enables Parameter Scalingcited by 8
  8. 2024Beyond The Rainbow: High Performance Deep Reinforcement Learning on a Desktop PCcited by 7
  9. 2025The Courage to Stop: Overcoming Sunk Cost Fallacy in Deep Reinforcement Learningcited by 6
  10. 2026Stable Deep Reinforcement Learning via Isotropic Gaussian Representationscited by 5
  11. 2025Optimistic critics can empower small actorscited by 4
  12. 2024ARLBench: Flexible and Efficient Benchmarking for Hyperparameter Optimization in Reinforcement Learningcited by 3
9 more
  1. 2025The Formalism-Implementation Gap in Reinforcement Learning Researchcited by 3
  2. 2024Uncovering RL Integration in SSL Loss: Objective-Specific Implications for Data-Efficient RLcited by 0
  3. 2024A Research Agenda for Usability and Generalisation in Reinforcement Learningcited by 0
  4. 2025Is Bellman Equation Enough for Learning Control?cited by 0
  5. 2025Cavity Duplexer Tuning with 1d Resnet-like Neural Networkscited by 0
  6. 2026Beyond Success Rates: Trainability and Extractability for Offline GCRLcited by 0
  7. 2026Representation Learning Enables Scalable Multitask Deep Reinforcement Learningcited by 0
  8. 2026Efficient Heteroscedastic Bayesian Optimization for Risk-Aware AutoRLcited by 0
  9. 2026V-Simba: Unleashing the Architectural Potential of RL in Visual Continuous Controlcited by 0