# QRA++: Quantified Reproducibility Assessment for Common Types of Results in Natural Language Processing

arXiv: https://arxiv.org/abs/2505.17043 · HTML: https://mlsanity.com/paper/2505.17043/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 2 of its 58 references that are in the map.
- First posted on arXiv 13 May 2025.

## Builds on (most cited first, 2 of 2)

- [Misinfo Reaction Frames: Reasoning about Readers' Reactions to News Headlines](https://mlsanity.com/paper/2104.08790/) (2021)
- [Experiments with Universal CEFR Classification](https://mlsanity.com/paper/1804.06636/) (2018)

## Built on by (most cited first, 1 of 1)

- [RecourseBench: A Modular Framework for Reproducible Algorithmic Recourse Evaluation](https://mlsanity.com/paper/2606.16113/) (2026)

## Abstract (arXiv)

Reproduction studies reported in NLP provide individual data points which in combination indicate worryingly low levels of reproducibility in the field. Because each reproduction study reports quantitative conclusions based on its own, often not explicitly stated, criteria for reproduction success/failure, the conclusions drawn are hard to interpret, compare, and learn from. In this paper, we present QRA++, a quantitative approach to reproducibility assessment that (i) produces continuous-valued degree of reproducibility assessments at three levels of granularity; (ii) utilises reproducibility measures that are directly comparable across different studies; and (iii) grounds expectations about degree of reproducibility in degree of similarity between experiments. QRA++ enables more informative reproducibility assessments to be conducted, and conclusions to be drawn about what causes reproducibility to be better/poorer. We illustrate this by applying QRA++ to three example sets of comparable experiments, revealing clear evidence that degree of reproducibility depends on similarity of experiment properties, but also system type and evaluation method.