# Measuring the Reliability of Reinforcement Learning Algorithms

arXiv: https://arxiv.org/abs/1912.05663 · HTML: https://mlsanity.com/paper/1912.05663/

## Key facts

- Cited by 51 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 10 Dec 2019.

## Built on by (most cited first, 30 of 51)

- [Deep Reinforcement Learning at the Edge of the Statistical Precipice](https://mlsanity.com/paper/2108.13264/) (2021)
- [A Sober Look at Progress in Language Model Reasoning: Pitfalls and Paths to Reproducibility](https://mlsanity.com/paper/2504.07086/) (2025)
- [Revisiting Design Choices in Offline Model-Based Reinforcement Learning](https://mlsanity.com/paper/2110.04135/) (2021)
- [Towards a Standardised Performance Evaluation Protocol for Cooperative MARL](https://mlsanity.com/paper/2209.10485/) (2022)
- [Is High Variance Unavoidable in RL? A Case Study in Continuous Control](https://mlsanity.com/paper/2110.11222/) (2021)
- [Evaluate & Evaluation on the Hub: Better Best Practices for Data and Model Measurements](https://mlsanity.com/paper/2210.01970/) (2022)
- [For Pre-Trained Vision Models in Motor Control, Not All Policy Learning Methods are Created Equal](https://mlsanity.com/paper/2304.04591/) (2023)
- [On the consistency of hyper-parameter selection in value-based deep reinforcement learning](https://mlsanity.com/paper/2406.17523/) (2024)
- [LightZero: A Unified Benchmark for Monte Carlo Tree Search in General Sequential Decision Scenarios](https://mlsanity.com/paper/2310.08348/) (2023)
- [Stackelberg Actor-Critic: Game-Theoretic Reinforcement Learning Algorithms](https://mlsanity.com/paper/2109.12286/) (2021)
- [Zipfian environments for Reinforcement Learning](https://mlsanity.com/paper/2203.08222/) (2022)
- [A Domain-Agnostic Approach for Characterization of Lifelong Learning Systems](https://mlsanity.com/paper/2301.07799/) (2023)
- [Robust Gymnasium: A Unified Modular Benchmark for Robust Reinforcement Learning](https://mlsanity.com/paper/2502.19652/) (2025)
- [Policy Optimization in a Noisy Neighborhood: On Return Landscapes in Continuous Control](https://mlsanity.com/paper/2309.14597/) (2023)
- [The Impact of Task Underspecification in Evaluating Deep Reinforcement Learning](https://mlsanity.com/paper/2210.08607/) (2022)
- [Continuous Control With Ensemble Deep Deterministic Policy Gradients](https://mlsanity.com/paper/2111.15382/) (2021)
- [Assessing the Impact of Distribution Shift on Reinforcement Learning Performance](https://mlsanity.com/paper/2402.03590/) (2024)
- [Real-time system optimal traffic routing under uncertainties -- Can physics models boost reinforcement learning?](https://mlsanity.com/paper/2407.07364/) (2024)
- [Relative Entropy Pathwise Policy Optimization](https://mlsanity.com/paper/2507.11019/) (2025)
- [Replicable Reinforcement Learning with Linear Function Approximation](https://mlsanity.com/paper/2509.08660/) (2025)
- [L2Explorer: A Lifelong Reinforcement Learning Assessment Environment](https://mlsanity.com/paper/2203.07454/) (2022)
- [REValueD: Regularised Ensemble Value-Decomposition for Factorisable Markov Decision Processes](https://mlsanity.com/paper/2401.08850/) (2024)
- [A Distance-based Anomaly Detection Framework for Deep Reinforcement Learning](https://mlsanity.com/paper/2109.09889/) (2021)
- [Deep Reinforcement Learning for Cryptocurrency Trading: Practical Approach to Address Backtest Overfitting](https://mlsanity.com/paper/2209.05559/) (2022)
- [PRUDEX-Compass: Towards Systematic Evaluation of Reinforcement Learning in Financial Markets](https://mlsanity.com/paper/2302.00586/) (2023)
- [Ensemble Value Functions for Efficient Exploration in Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2302.03439/) (2023)
- [Measuring and Mitigating Interference in Reinforcement Learning](https://mlsanity.com/paper/2307.04887/) (2023)
- [SoK: The Pitfalls of Deep Reinforcement Learning for Cybersecurity](https://mlsanity.com/paper/2602.08690/) (2026)
- [Beyond Rewards in Reinforcement Learning for Cyber Defence](https://mlsanity.com/paper/2602.04809/) (2026)
- [A2Perf: Real-World Autonomous Agents Benchmark](https://mlsanity.com/paper/2503.03056/) (2025)

## Abstract (arXiv)

Lack of reliability is a well-known issue for reinforcement learning (RL) algorithms. This problem has gained increasing attention in recent years, and efforts to improve it have grown substantially. To aid RL researchers and production users with the evaluation and improvement of reliability, we propose a set of metrics that quantitatively measure different aspects of reliability. In this work, we focus on variability and risk, both during training and after learning (on a fixed policy). We designed these metrics to be general-purpose, and we also designed complementary statistical tests to enable rigorous comparisons on these metrics. In this paper, we first describe the desired properties of the metrics and their design, the aspects of reliability that they measure, and their applicability to different scenarios. We then describe the statistical tests and make additional practical recommendations for reporting results. The metrics and accompanying statistical tools have been made available as an open-source library at https://github.com/google-research/rl-reliability-metrics. We apply our metrics to a set of common RL algorithms and environments, compare them, and analyze the results.