# A Method for Evaluating Hyperparameter Sensitivity in Reinforcement Learning

arXiv: https://arxiv.org/abs/2412.07165 · HTML: https://mlsanity.com/paper/2412.07165/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- Placed in Deep RL algorithms and representations, one of 15 directions in Deep reinforcement learning methods.
- Builds on 13 of its 21 references that are in the map.
- First posted on arXiv 10 Dec 2024.

## Placed in

- Deep reinforcement learning methods / [Deep RL algorithms and representations](https://mlsanity.com/deeprl/deep-rl-algorithms-and-representations/)

## Builds on (most cited first, 13 of 13)

- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Mastering Diverse Domains through World Models](https://mlsanity.com/paper/2301.04104/) (2023)
- [Rainbow: Combining Improvements in Deep Reinforcement Learning](https://mlsanity.com/paper/1710.02298/) (2017)
- [Brax -- A Differentiable Physics Engine for Large Scale Rigid Body Simulation](https://mlsanity.com/paper/2106.13281/) (2021)
- [What Matters In On-Policy Reinforcement Learning? A Large-Scale Empirical Study](https://mlsanity.com/paper/2006.05990/) (2020)
- [Discovered Policy Optimisation](https://mlsanity.com/paper/2210.05639/) (2022)
- [Learning values across many orders of magnitude](https://mlsanity.com/paper/1602.07714/) (2016)
- [Hyperparameters in Reinforcement Learning and How To Tune Them](https://mlsanity.com/paper/2306.01324/) (2023)
- [Evaluating the Performance of Reinforcement Learning Algorithms](https://mlsanity.com/paper/2006.16958/) (2020)
- [Sample-Efficient Automated Deep Reinforcement Learning](https://mlsanity.com/paper/2009.01555/) (2020)
- [Reward Scale Robustness for Proximal Policy Optimization via DreamerV3 Tricks](https://mlsanity.com/paper/2310.17805/) (2023)
- [Hyperparameters in Contextual RL are Highly Situational](https://mlsanity.com/paper/2212.10876/) (2022)
- [The Cross-environment Hyperparameter Setting Benchmark for Reinforcement Learning](https://mlsanity.com/paper/2407.18840/) (2024)

## Built on by (most cited first, 15 of 15)

- [Iterative Distillation for Reward-Guided Fine-Tuning of Diffusion Models in Biomolecular Design](https://mlsanity.com/paper/2507.00445/) (2025)
- [Improving Value Estimation Critically Enhances Vanilla Policy Gradient](https://mlsanity.com/paper/2505.19247/) (2025)
- [How Should We Meta-Learn Reinforcement Learning Algorithms?](https://mlsanity.com/paper/2507.17668/) (2025)
- [SoK: The Pitfalls of Deep Reinforcement Learning for Cybersecurity](https://mlsanity.com/paper/2602.08690/) (2026)
- [The Formalism-Implementation Gap in Reinforcement Learning Research](https://mlsanity.com/paper/2510.16175/) (2025)
- [Deep Reinforcement Learning for Dynamic Algorithm Configuration: A Case Study on Optimizing OneMax with the (1+(λ,λ))-GA](https://mlsanity.com/paper/2512.03805/) (2025)
- [Position: RL Researchers Need to Distinguish Between Solving Simulators and Using Simulators as a Proxy](https://mlsanity.com/paper/2606.28433/) (2026)
- [Multiple Weaks Win Single Strong: Large Language Models Ensemble Weak Reinforcement Learning Agents into a Supreme One](https://mlsanity.com/paper/2505.15306/) (2025)
- [Sig2Model: A Boosting-Driven Model for Updatable Learned Indexes](https://mlsanity.com/paper/2509.20781/) (2025)
- [Beyond Success Rates: Trainability and Extractability for Offline GCRL](https://mlsanity.com/paper/2602.05459/) (2026)
- [Object-Oriented Transition Modeling with Inductive Logic Programming](https://mlsanity.com/paper/2602.07602/) (2026)
- [NASimJax: A GPU-Accelerated Policy Learning Framework for Penetration Testing](https://mlsanity.com/paper/2603.19864/) (2026)
- [Feedback-Normalized Developer Memory for Reinforcement-Learning Coding Agents: A Safety-Gated MCP Architecture](https://mlsanity.com/paper/2605.01567/) (2026)
- [Plan2Cleanse: Test-Time Backdoor Defense via Monte-Carlo Planning in Deep Reinforcement Learning](https://mlsanity.com/paper/2605.09638/) (2026)
- [Learning-Based Collaborative MEC for LLM Inference with Soft-Deadline Awareness via Transformer-Enhanced PPO](https://mlsanity.com/paper/2608.02031/) (2026)

## Abstract (arXiv)

The performance of modern reinforcement learning algorithms critically relies on tuning ever-increasing numbers of hyperparameters. Often, small changes in a hyperparameter can lead to drastic changes in performance, and different environments require very different hyperparameter settings to achieve state-of-the-art performance reported in the literature. We currently lack a scalable and widely accepted approach to characterizing these complex interactions. This work proposes a new empirical methodology for studying, comparing, and quantifying the sensitivity of an algorithm's performance to hyperparameter tuning for a given set of environments. We then demonstrate the utility of this methodology by assessing the hyperparameter sensitivity of several commonly used normalization variants of PPO. The results suggest that several algorithmic performance improvements may, in fact, be a result of an increased reliance on hyperparameter tuning.