# Significant Improvements over the State of the Art? A Case Study of the MS MARCO Document Ranking Leaderboard

arXiv: https://arxiv.org/abs/2102.12887 · HTML: https://mlsanity.com/paper/2102.12887/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 25 Feb 2021.

## Built on by (most cited first, 12 of 12)

- [Deep Reinforcement Learning at the Edge of the Statistical Precipice](https://mlsanity.com/paper/2108.13264/) (2021)
- [The Benchmark Lottery](https://mlsanity.com/paper/2107.07002/) (2021)
- [Overview of the TREC 2022 deep learning track](https://mlsanity.com/paper/2507.10865/) (2025)
- [Overview of the TREC 2021 deep learning track](https://mlsanity.com/paper/2507.08191/) (2025)
- [The worst of both worlds: A comparative analysis of errors in learning from data in psychology and machine learning](https://mlsanity.com/paper/2203.06498/) (2022)
- [MGeo: Multi-Modal Geographic Pre-Training Method](https://mlsanity.com/paper/2301.04283/) (2023)
- [Benchmark Data Repositories for Better Benchmarking](https://mlsanity.com/paper/2410.24100/) (2024)
- [Overview of the TREC 2023 deep learning track](https://mlsanity.com/paper/2507.08890/) (2025)
- [Deep Reinforcement Learning for Cryptocurrency Trading: Practical Approach to Address Backtest Overfitting](https://mlsanity.com/paper/2209.05559/) (2022)
- [Atomized Search Length: Beyond User Models](https://mlsanity.com/paper/2201.01745/) (2022)
- [Reproducibility and Artifact Consistency of the SIGIR 2022 Recommender Systems Papers Based on Message Passing](https://mlsanity.com/paper/2503.07823/) (2025)
- [Challenges in Deep Learning-Based Small Organ Segmentation: A Benchmarking Perspective for Medical Research with Limited Datasets](https://mlsanity.com/paper/2509.05892/) (2025)

## Abstract (arXiv)

Leaderboards are a ubiquitous part of modern research in applied machine learning. By design, they sort entries into some linear order, where the top-scoring entry is recognized as the "state of the art" (SOTA). Due to the rapid progress being made in information retrieval today, particularly with neural models, the top entry in a leaderboard is replaced with some regularity. These are touted as improvements in the state of the art. Such pronouncements, however, are almost never qualified with significance testing. In the context of the MS MARCO document ranking leaderboard, we pose a specific question: How do we know if a run is significantly better than the current SOTA? We ask this question against the backdrop of recent IR debates on scale types: in particular, whether commonly used significance tests are even mathematically permissible. Recognizing these potential pitfalls in evaluation methodology, our study proposes an evaluation framework that explicitly treats certain outcomes as distinct and avoids aggregating them into a single-point metric. Empirical analysis of SOTA runs from the MS MARCO document ranking leaderboard reveals insights about how one run can be "significantly better" than another that are obscured by the current official evaluation metric (MRR@100).