Data to 7 Oct 2026

How Many Random Seeds? Statistical Power Analysis in Deep Reinforcement Learning Experiments

arXiv 1806.08295 ↗First posted 21 Jun 2018cs.LG stat.MLcited by 56

Not on the map. Why?

Abstract (arXiv)

Consistently checking the statistical significance of experimental results is one of the mandatory methodological steps to address the so-called "reproducibility crisis" in deep reinforcement learning. In this tutorial paper, we explain how the number of random seeds relates to the probabilities of statistical errors. For both the t-test and the bootstrap confidence interval test, we recall theoretical guidelines to determine the number of random seeds one should use to provide a statistically significant comparison of the performance of two algorithms. Finally, we discuss the influence of deviations from the assumptions usually made by statistical tests. We show that they can lead to inaccurate evaluations of statistical errors and provide guidelines to counter these negative effects. We make our code available to perform the tests.

Key facts. Cited by 56 papers in the map (data through 7 Oct 2026). First posted on arXiv 21 Jun 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 6; 2022: 5; 2023: 7; 2024: 7; 2025: 14; 2026: 17.

56 papers.

Sort:

By area:

2021Deep Reinforcement Learning at the Edge of the Statistical Precipicecited by 781
2024AndroidWorld: A Dynamic Benchmarking Environment for Autonomous Agentscited by 450
2025A Sober Look at Progress in Language Model Reasoning: Pitfalls and Paths to Reproducibilitycited by 89
2022Automated Reinforcement Learning (AutoRL): A Survey and Open Problemscited by 69
2023Empirical Design in Reinforcement Learningcited by 54
2022Towards a Standardised Performance Evaluation Protocol for Cooperative MARLcited by 48
2021Lifelong Robotic Reinforcement Learning by Retaining Experiencescited by 34
2024Reproducibility in Machine Learning-based Research: Overview, Barriers and Driverscited by 34
2022How stable are Transferability Metrics evaluations?cited by 30
2021Is High Variance Unavoidable in RL? A Case Study in Continuous Controlcited by 26
2022Towards Training Reproducible Deep Learning Modelscited by 23
2022Lifelong Reinforcement Learning with Modulating Maskscited by 20

38 more

2021TeachMyAgent: a Benchmark for Automatic Curriculum Learning in Deep RLcited by 18
2024Position: Benchmarking is Limited in Reinforcement Learning Researchcited by 15
2025Rethinking RL Scaling for Vision Language Models: A Transparent, From-Scratch Framework and Comprehensive Evaluation Schemecited by 14
2023A Domain-Agnostic Approach for Characterization of Lifelong Learning Systemscited by 13
2025Robust Gymnasium: A Unified Modular Benchmark for Robust Reinforcement Learningcited by 12
2023Sharing Lifelong Reinforcement Learning Knowledge via Modulating Maskscited by 10
2025Benchmarking Quantum Reinforcement Learningcited by 9
2026EvoTrainer: Co-Evolving LLM Policies and Training Harnesses for Autonomous Agentic Reinforcement Learningcited by 9
2021Optimistic Reinforcement Learning by Forward Kullback-Leibler Divergence Optimizationcited by 6
2025Octax: Accelerated CHIP-8 Arcade Environments for Reinforcement Learning in JAXcited by 5
2021DESTA: A Framework for Safe Reinforcement Learning with Markov Games of Interventioncited by 4
2023Deep Q-Network Based Decision Making for Autonomous Drivingcited by 4
2023Consciousness-Inspired Spatio-Temporal Abstractions for Better Generalization in Reinforcement Learningcited by 4
2024Assessing the Impact of Distribution Shift on Reinforcement Learning Performancecited by 4
2026SoK: The Pitfalls of Deep Reinforcement Learning for Cybersecuritycited by 4
2023AdaStop: adaptive statistical testing for sound comparisons of Deep RL agentscited by 3
2024Representation Learning For Efficient Deep Multi-Agent Reinforcement Learningcited by 3
2025Evidence on the Regularisation Properties of Maximum-Entropy Reinforcement Learningcited by 3
2025PTEB: Towards Robust Text Embedding Evaluation via Stochastic Paraphrasing at Evaluation Time with LLMscited by 3
2024The Cross-environment Hyperparameter Setting Benchmark for Reinforcement Learningcited by 2
2024Double Actor-Critic with TD Error-Driven Regularization in Reinforcement Learningcited by 2
2025A2Perf: Real-World Autonomous Agents Benchmarkcited by 2
2025gfnx: Fast and Scalable Library for Generative Flow Networks in JAXcited by 2
2026On Effectiveness and Efficiency of Agentic Tool-calling and RL Trainingcited by 2
2026Performance Variation in Deep Reinforcement Learningcited by 2
2023How much can change in a year? Revisiting Evaluation in Multi-Agent Reinforcement Learningcited by 1
2025SOCIA-nabla: Textual Gradient Meets Multi-Agent Orchestration for Automated Simulator Generationcited by 1
2026Action Shapley: A Training Data Selection Metric for World Model in Reinforcement Learningcited by 1
2026Behavior-Consistent Deep Reinforcement Learningcited by 1
2026Engineering Reliable Coding Agents: Evaluating and Operating the System Around the Modelcited by 1
2025Multiple Weaks Win Single Strong: Large Language Models Ensemble Weak Reinforcement Learning Agents into a Supreme Onecited by 0
2025Policy Search, Retrieval, and Composition via Task Similarity in Collaborative Agentic Systemscited by 0
2025SOCIA-Nabla: Textual Gradient Meets Multi-Agent Orchestration for Automated Simulator Generationcited by 0
2025When Does Pairing Seeds Reduce Variance? Evidence from a Multi-Agent Economic Simulationcited by 0
2026SOCIA-EVO: Automated Simulator Construction via Dual-Anchored Bi-Level Optimizationcited by 0
2026Stochastic MeanFlow Policies: One-Step Generative Control with Entropic Mirror Descentcited by 0
2026Auto-DSM Under the Lens: A Black-Box Evaluation Framework for LLM-Based DSM Generationcited by 0
2026When Do Options Help? Policy Necrosis and Redundant Coverage in Option-Criticcited by 0

50 most cited of 56 All 56 (JSON)