mlsanityData to 14 Oct 2026

A Method for Evaluating Hyperparameter Sensitivity in Reinforcement Learning

arXiv 2412.07165 ↗First posted 10 Dec 2024cs.LG cs.AIcited by 15

On the map: Deep reinforcement learning methods → Deep RL algorithms and representations

Abstract (arXiv)

The performance of modern reinforcement learning algorithms critically relies on tuning ever-increasing numbers of hyperparameters. Often, small changes in a hyperparameter can lead to drastic changes in performance, and different environments require very different hyperparameter settings to achieve state-of-the-art performance reported in the literature. We currently lack a scalable and widely accepted approach to characterizing these complex interactions. This work proposes a new empirical methodology for studying, comparing, and quantifying the sensitivity of an algorithm's performance to hyperparameter tuning for a given set of environments. We then demonstrate the utility of this methodology by assessing the hyperparameter sensitivity of several commonly used normalization variants of PPO. The results suggest that several algorithmic performance improvements may, in fact, be a result of an increased reliance on hyperparameter tuning.

Key facts. Cited by 15 papers in the map (data through 14 Oct 2026). Placed in Deep RL algorithms and representations, one of 15 directions in Deep reinforcement learning methods. Builds on 13 of its 21 references that are in the map. First posted on arXiv 10 Dec 2024.

Builds on

13 of 21 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024The Cross-environment Hyperparameter Setting Benchmark for Reinforcement Learning26 Jul

Other references · 12

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2023Mastering Diverse Domains through World Modelscited by 1,253
  3. 2017Rainbow: Combining Improvements in Deep Reinforcement Learningcited by 786
  4. 2021Brax -- A Differentiable Physics Engine for Large Scale Rigid Body Simulationcited by 407
  5. 2020What Matters In On-Policy Reinforcement Learning? A Large-Scale Empirical Studycited by 162
  6. 2022Discovered Policy Optimisationcited by 104
  7. 2016Learning values across many orders of magnitudecited by 85
  8. 2023Hyperparameters in Reinforcement Learning and How To Tune Themcited by 59
  9. 2020Evaluating the Performance of Reinforcement Learning Algorithmscited by 36
  10. 2020Sample-Efficient Automated Deep Reinforcement Learningcited by 32
  11. 2023Reward Scale Robustness for Proximal Policy Optimization via DreamerV3 Trickscited by 7
  12. 2022Hyperparameters in Contextual RL are Highly Situationalcited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 7; 2026: 8.

15 papers.

Sort:

By area:

  1. 2025Iterative Distillation for Reward-Guided Fine-Tuning of Diffusion Models in Biomolecular Designcited by 11
  2. 2025Improving Value Estimation Critically Enhances Vanilla Policy Gradientcited by 5
  3. 2025How Should We Meta-Learn Reinforcement Learning Algorithms?cited by 5
  4. 2026SoK: The Pitfalls of Deep Reinforcement Learning for Cybersecuritycited by 4
  5. 2025The Formalism-Implementation Gap in Reinforcement Learning Researchcited by 3
  6. 2025Deep Reinforcement Learning for Dynamic Algorithm Configuration: A Case Study on Optimizing OneMax with the (1+(λ,λ))-GAcited by 1
  7. 2026Position: RL Researchers Need to Distinguish Between Solving Simulators and Using Simulators as a Proxycited by 1
  8. 2025Multiple Weaks Win Single Strong: Large Language Models Ensemble Weak Reinforcement Learning Agents into a Supreme Onecited by 0
  9. 2025Sig2Model: A Boosting-Driven Model for Updatable Learned Indexescited by 0
  10. 2026Beyond Success Rates: Trainability and Extractability for Offline GCRLcited by 0
  11. 2026Object-Oriented Transition Modeling with Inductive Logic Programmingcited by 0
  12. 2026NASimJax: A GPU-Accelerated Policy Learning Framework for Penetration Testingcited by 0
3 more
  1. 2026Feedback-Normalized Developer Memory for Reinforcement-Learning Coding Agents: A Safety-Gated MCP Architecturecited by 0
  2. 2026Plan2Cleanse: Test-Time Backdoor Defense via Monte-Carlo Planning in Deep Reinforcement Learningcited by 0
  3. 2026Learning-Based Collaborative MEC for LLM Inference with Soft-Deadline Awareness via Transformer-Enhanced PPOcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024ARLBench: Flexible and Efficient Benchmarking for Hyperparameter Optimization in Reinforcement Learning
  2. 2024On the consistency of hyper-parameter selection in value-based deep reinforcement learning
  3. 2024Combining Automated Optimisation of Hyperparameters and Reward Shape
  4. 2024Hyper: Hyperparameter Robust Efficient Exploration in Reinforcement Learning
  5. 2025Hyperparameter Optimisation with Practical Interpretability and Explanation Methods in Probabilistic Curriculum Learning
  6. 2024Position: Benchmarking is Limited in Reinforcement Learning Research
  7. 2025HyperController: A Hyperparameter Controller for Fast and Stable Training of Reinforcement Learning Neural Networks
  8. 2025A Clean Slate for Offline Reinforcement Learning