# Quantified Reproducibility Assessment of NLP Results

arXiv: https://arxiv.org/abs/2204.05961 · HTML: https://mlsanity.com/paper/2204.05961/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 5 of its 28 references that are in the map.
- First posted on arXiv 12 Apr 2022.

## Builds on (most cited first, 5 of 5)

- [A Call for Clarity in Reporting BLEU Scores](https://mlsanity.com/paper/1804.08771/) (2018)
- [A Systematic Review of Reproducibility Research in Natural Language Processing](https://mlsanity.com/paper/2103.07929/) (2021)
- [The Human Evaluation Datasheet 1.0: A Template for Recording Details of Human Evaluation Experiments in NLP](https://mlsanity.com/paper/2103.09710/) (2021)
- [Experiments with Universal CEFR Classification](https://mlsanity.com/paper/1804.06636/) (2018)
- [Sustainable computational science: the ReScience initiative](https://mlsanity.com/paper/1707.04393/) (2017)

## Built on by (most cited first, 8 of 8)

- [KoLA: Carefully Benchmarking World Knowledge of Large Language Models](https://mlsanity.com/paper/2306.09296/) (2023)
- [Missing Information, Unresponsive Authors, Experimental Flaws: The Impossibility of Assessing the Reproducibility of Previous Human Evaluations in NLP](https://mlsanity.com/paper/2305.01633/) (2023)
- [Position: Key Claims in LLM Research Have a Long Tail of Footnotes](https://mlsanity.com/paper/2308.07120/) (2023)
- [When Good and Reproducible Results are a Giant with Feet of Clay: The Importance of Software Quality in NLP](https://mlsanity.com/paper/2303.16166/) (2023)
- [Collect, Measure, Repeat: Reliability Factors for Responsible AI Data Collection](https://mlsanity.com/paper/2308.12885/) (2023)
- [Principles from Clinical Research for NLP Model Generalization](https://mlsanity.com/paper/2311.03663/) (2023)
- [Investigating Reproducibility at Interspeech Conferences: A Longitudinal and Comparative Perspective](https://mlsanity.com/paper/2306.10033/) (2023)
- [With a Little Help from the Authors: Reproducing Human Evaluation of an MT Error Detector](https://mlsanity.com/paper/2308.06527/) (2023)

## Abstract (arXiv)

This paper describes and tests a method for carrying out quantified reproducibility assessment (QRA) that is based on concepts and definitions from metrology. QRA produces a single score estimating the degree of reproducibility of a given system and evaluation measure, on the basis of the scores from, and differences between, different reproductions. We test QRA on 18 system and evaluation measure combinations (involving diverse NLP tasks and types of evaluation), for each of which we have the original results and one to seven reproduction results. The proposed QRA method produces degree-of-reproducibility scores that are comparable across multiple reproductions not only of the same, but of different original studies. We find that the proposed method facilitates insights into causes of variation between reproductions, and allows conclusions to be drawn about what changes to system and/or evaluation design might lead to improved reproducibility.