# MS MARCO: Benchmarking Ranking Models in the Large-Data Regime

arXiv: https://arxiv.org/abs/2105.04021 · HTML: https://mlsanity.com/paper/2105.04021/

## Key facts

- Cited by 28 papers in the map (data through 14 Oct 2026).
- Builds on 34 of its 92 references that are in the map.
- First posted on arXiv 9 May 2021.

## Builds on (most cited first, 30 of 34)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [MS MARCO: A Human Generated MAchine Reading COmprehension Dataset](https://mlsanity.com/paper/1611.09268/) (2016)
- [Passage Re-ranking with BERT](https://mlsanity.com/paper/1901.04085/) (2019)
- [Document Ranking with a Pretrained Sequence-to-Sequence Model](https://mlsanity.com/paper/2003.06713/) (2020)
- [Pretrained Transformers for Text Ranking: BERT and Beyond](https://mlsanity.com/paper/2010.06467/) (2020)
- [Overview of the TREC 2020 deep learning track](https://mlsanity.com/paper/2102.07662/) (2021)
- [CORD-19: The COVID-19 Open Research Dataset](https://mlsanity.com/paper/2004.10706/) (2020)
- [End-to-End Neural Ad-hoc Ranking with Kernel Pooling](https://mlsanity.com/paper/1706.06613/) (2017)
- [Few-Shot Generative Conversational Query Rewriting](https://mlsanity.com/paper/2006.05009/) (2020)
- [Learning to Match Using Local and Distributed Representations of Text for Web Search](https://mlsanity.com/paper/1610.08136/) (2016)
- [TREC CAsT 2019: The Conversational Assistance Track Overview](https://mlsanity.com/paper/2003.13624/) (2020)
- [Neural Ranking Models with Weak Supervision](https://mlsanity.com/paper/1704.08803/) (2017)
- [Query Resolution for Conversational Search with Limited Supervision](https://mlsanity.com/paper/2005.11723/) (2020)
- [BERT-QE: Contextualized Query Expansion for Document Re-ranking](https://mlsanity.com/paper/2009.07258/) (2020)
- [PROP: Pre-training with Representative Words Prediction for Ad-hoc Retrieval](https://mlsanity.com/paper/2010.10137/) (2020)
- [TREC-COVID: Constructing a Pandemic Information Retrieval Test Collection](https://mlsanity.com/paper/2005.04474/) (2020)
- [TREC Deep Learning Track: Reusable Test Collections in the Large Data Regime](https://mlsanity.com/paper/2104.09399/) (2021)
- [Covidex: Neural Ranking Models and Keyword Search Infrastructure for the COVID-19 Open Research Dataset](https://mlsanity.com/paper/2007.07846/) (2020)
- [Learning Latent Vector Spaces for Product Search](https://mlsanity.com/paper/1608.07253/) (2016)
- [Selective Weak Supervision for Neural Information Retrieval](https://mlsanity.com/paper/2001.10382/) (2020)
- [Rapidly Deploying a Neural Search Engine for the COVID-19 Open Research Dataset: Preliminary Thoughts and Lessons Learned](https://mlsanity.com/paper/2004.05125/) (2020)
- [Topic Propagation in Conversational Search](https://mlsanity.com/paper/2004.14054/) (2020)
- [Optimizing Query Evaluations using Reinforcement Learning for Web Search](https://mlsanity.com/paper/1804.04410/) (2018)
- [Neural Ranking Models with Multiple Document Fields](https://mlsanity.com/paper/1711.09174/) (2017)
- [Benchmark for Complex Answer Retrieval](https://mlsanity.com/paper/1705.04803/) (2017)
- [RRF102: Meeting the TREC-COVID Challenge with a 100+ Runs Ensemble](https://mlsanity.com/paper/2010.00200/) (2020)
- [SLEDGE-Z: A Zero-Shot Baseline for COVID-19 Literature Search](https://mlsanity.com/paper/2010.05987/) (2020)
- [Long Document Ranking with Query-Directed Sparse Transformer](https://mlsanity.com/paper/2010.12683/) (2020)

## Built on by (most cited first, 28 of 28)

- [GEO: Generative Engine Optimization](https://mlsanity.com/paper/2311.09735/) (2023)
- [Making a MIRACL: Multilingual Information Retrieval Across a Continuum of Languages](https://mlsanity.com/paper/2210.09984/) (2022)
- [On Cross-Lingual Retrieval with Multilingual Text Encoders](https://mlsanity.com/paper/2112.11031/) (2021)
- [Overview of the TREC 2022 deep learning track](https://mlsanity.com/paper/2507.10865/) (2025)
- [Overview of the TREC 2021 deep learning track](https://mlsanity.com/paper/2507.08191/) (2025)
- [Parameter-Efficient Neural Reranking for Cross-Lingual and Multilingual Retrieval](https://mlsanity.com/paper/2204.02292/) (2022)
- [MGeo: Multi-Modal Geographic Pre-Training Method](https://mlsanity.com/paper/2301.04283/) (2023)
- [Overview of the TREC 2023 deep learning track](https://mlsanity.com/paper/2507.08890/) (2025)
- [A Reproducibility Study of PLAID](https://mlsanity.com/paper/2404.14989/) (2024)
- [Revealing the Blind Spot of Sentence Encoder Evaluation by HEROS](https://mlsanity.com/paper/2306.05083/) (2023)
- [Generative Product Recommendations for Implicit Superlative Queries](https://mlsanity.com/paper/2504.18748/) (2025)
- [Disentangled Modeling of Domain and Relevance for Adaptable Dense Retrieval](https://mlsanity.com/paper/2208.05753/) (2022)
- [A Massive Scale Semantic Similarity Dataset of Historical English](https://mlsanity.com/paper/2306.17810/) (2023)
- [LEANN: A Low-Storage Vector Index](https://mlsanity.com/paper/2506.08276/) (2025)
- [The Role of Relevance in Fair Ranking](https://mlsanity.com/paper/2305.05608/) (2023)
- [Neural Machine Unranking](https://mlsanity.com/paper/2408.05330/) (2024)
- [Aligning Extraction and Generation for Robust Retrieval-Augmented Generation](https://mlsanity.com/paper/2503.04789/) (2025)
- [Isolated but Exposed: Persistence-Based Memory Extraction Attack on LLM Agents](https://mlsanity.com/paper/2607.23444/) (2026)
- [ZusammenQA: Data Augmentation with Specialized Models for Cross-lingual Open-retrieval Question Answering System](https://mlsanity.com/paper/2205.14981/) (2022)
- [Are We There Yet? A Decision Framework for Replacing Term Based Retrieval with Dense Retrieval Systems](https://mlsanity.com/paper/2206.12993/) (2022)
- [Annotating Data for Fine-Tuning a Neural Ranker? Current Active Learning Strategies are not Better than Random Selection](https://mlsanity.com/paper/2309.06131/) (2023)
- [Verif.ai: Towards an Open-Source Scientific Generative Question-Answering System with Referenced and Verifiable Answers](https://mlsanity.com/paper/2402.18589/) (2024)
- [Scientific QA System with Verifiable Answers](https://mlsanity.com/paper/2407.11485/) (2024)
- [Neural Corrective Machine Unranking](https://mlsanity.com/paper/2411.08562/) (2024)
- [AgentSearchBench: A Benchmark for AI Agent Search in the Wild](https://mlsanity.com/paper/2604.22436/) (2026)
- [Towards Consistency Filtering-Free Unsupervised Learning for Dense Retrieval](https://mlsanity.com/paper/2308.02926/) (2023)
- [Beyond Caption-Based Queries for Video Moment Retrieval](https://mlsanity.com/paper/2603.02363/) (2026)
- [STORM: Stepwise Token Optimization with Reward-Guided Beam Search](https://mlsanity.com/paper/2606.10621/) (2026)

## Abstract (arXiv)

Evaluation efforts such as TREC, CLEF, NTCIR and FIRE, alongside public leaderboard such as MS MARCO, are intended to encourage research and track our progress, addressing big questions in our field. However, the goal is not simply to identify which run is "best", achieving the top score. The goal is to move the field forward by developing new robust techniques, that work in many different settings, and are adopted in research and practice. This paper uses the MS MARCO and TREC Deep Learning Track as our case study, comparing it to the case of TREC ad hoc ranking in the 1990s. We show how the design of the evaluation effort can encourage or discourage certain outcomes, and raising questions about internal and external validity of results. We provide some analysis of certain pitfalls, and a statement of best practices for avoiding such pitfalls. We summarize the progress of the effort so far, and describe our desired end state of "robust usefulness", along with steps that might be required to get us there.