# Guaranteeing Reproducibility in Deep Learning Competitions

arXiv: https://arxiv.org/abs/2005.06041 · HTML: https://mlsanity.com/paper/2005.06041/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 12 May 2020.

## Built on by (most cited first, 4 of 4)

- [The MineRL 2020 Competition on Sample Efficient Reinforcement Learning using Human Priors](https://mlsanity.com/paper/2101.11071/) (2021)
- [MineRL Diamond 2021 Competition: Overview, Results, and Lessons Learned](https://mlsanity.com/paper/2202.10583/) (2022)
- [Emotion-Cause Pair Extraction as Question Answering](https://mlsanity.com/paper/2301.01982/) (2023)
- [Towards robust and domain agnostic reinforcement learning competitions](https://mlsanity.com/paper/2106.03748/) (2021)

## Abstract (arXiv)

To encourage the development of methods with reproducible and robust training behavior, we propose a challenge paradigm where competitors are evaluated directly on the performance of their learning procedures rather than pre-trained agents. Since competition organizers re-train proposed methods in a controlled setting they can guarantee reproducibility, and -- by retraining submissions using a held-out test set -- help ensure generalization past the environments on which they were trained.