# Please, Don't Forget the Difference and the Confidence Interval when Seeking for the State-of-the-Art Status

arXiv: https://arxiv.org/abs/2205.11134 · HTML: https://mlsanity.com/paper/2205.11134/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 11 of its 49 references that are in the map.
- First posted on arXiv 23 May 2022.

## Builds on (most cited first, 11 of 11)

- [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018)
- [XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalization](https://mlsanity.com/paper/2003.11080/) (2020)
- [Datasets: A Community Library for Natural Language Processing](https://mlsanity.com/paper/2109.02846/) (2021)
- [Are GANs Created Equal? A Large-Scale Study](https://mlsanity.com/paper/1711.10337/) (2017)
- [Show Your Work: Improved Reporting of Experimental Results](https://mlsanity.com/paper/1909.03004/) (2019)
- [The Benchmark Lottery](https://mlsanity.com/paper/2107.07002/) (2021)
- [Effective LSTMs for Target-Dependent Sentiment Classification](https://mlsanity.com/paper/1512.01100/) (2015)
- [Aspect Level Sentiment Classification with Deep Memory Network](https://mlsanity.com/paper/1605.08900/) (2016)
- [Attentional Encoder Network for Targeted Sentiment Classification](https://mlsanity.com/paper/1902.09314/) (2019)
- [WASSA-2017 Shared Task on Emotion Intensity](https://mlsanity.com/paper/1708.03700/) (2017)
- [Statistical Testing on ASR Performance via Blockwise Bootstrap](https://mlsanity.com/paper/1912.09508/) (2019)

## Built on by (most cited first, 3 of 3)

- [Do Language Models Encode Semantic Relations? Probing and Sparse Feature Analysis](https://mlsanity.com/paper/2603.17624/) (2026)
- [AmpleHate: Amplifying the Attention for Versatile Implicit Hate Detection](https://mlsanity.com/paper/2505.19528/) (2025)
- [Class Imbalance and Batch Effects in LLM-Based Screening for Systematic Reviews](https://mlsanity.com/paper/2608.14737/) (2026)

## Abstract (arXiv)

This paper argues for the widest possible use of bootstrap confidence intervals for comparing NLP system performances instead of the state-of-the-art status (SOTA) and statistical significance testing. Their main benefits are to draw attention to the difference in performance between two systems and to help assessing the degree of superiority of one system over another. Two cases studies, one comparing several systems and the other based on a K-fold cross-validation procedure, illustrate these benefits. A python module for obtaining these confidence intervals as well as a second function implementing the Fisher-Pitman test for paired samples are freely available on PyPi.