# Quantifying Reproducibility in NLP and ML

arXiv: https://arxiv.org/abs/2109.01211 · HTML: https://mlsanity.com/paper/2109.01211/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 4 of its 30 references that are in the map.
- First posted on arXiv 2 Sep 2021.

## Builds on (most cited first, 4 of 4)

- [A Systematic Review of Reproducibility Research in Natural Language Processing](https://mlsanity.com/paper/2103.07929/) (2021)
- [The Human Evaluation Datasheet 1.0: A Template for Recording Details of Human Evaluation Experiments in NLP](https://mlsanity.com/paper/2103.09710/) (2021)
- [Experiments with Universal CEFR Classification](https://mlsanity.com/paper/1804.06636/) (2018)
- [Sustainable computational science: the ReScience initiative](https://mlsanity.com/paper/1707.04393/) (2017)

## Built on by (most cited first, 1 of 1)

- [You Are What You Write: Preserving Privacy in the Era of Large Language Models](https://mlsanity.com/paper/2204.09391/) (2022)

## Abstract (arXiv)

Reproducibility has become an intensely debated topic in NLP and ML over recent years, but no commonly accepted way of assessing reproducibility, let alone quantifying it, has so far emerged. The assumption has been that wider scientific reproducibility terminology and definitions are not applicable to NLP/ML, with the result that many different terms and definitions have been proposed, some diametrically opposed. In this paper, we test this assumption, by taking the standard terminology and definitions from metrology and applying them directly to NLP/ML. We find that we are able to straightforwardly derive a practical framework for assessing reproducibility which has the desirable property of yielding a quantified degree of reproducibility that is comparable across different reproduction studies.