mlsanityData to 7 Oct 2026

MS MARCO: Benchmarking Ranking Models in the Large-Data Regime

arXiv 2105.04021 ↗First posted 9 May 2021cs.IR cs.AI cs.LGcited by 28

Not on the map. Why?

Abstract (arXiv)

Evaluation efforts such as TREC, CLEF, NTCIR and FIRE, alongside public leaderboard such as MS MARCO, are intended to encourage research and track our progress, addressing big questions in our field. However, the goal is not simply to identify which run is "best", achieving the top score. The goal is to move the field forward by developing new robust techniques, that work in many different settings, and are adopted in research and practice. This paper uses the MS MARCO and TREC Deep Learning Track as our case study, comparing it to the case of TREC ad hoc ranking in the 1990s. We show how the design of the evaluation effort can encourage or discourage certain outcomes, and raising questions about internal and external validity of results. We provide some analysis of certain pitfalls, and a statement of best practices for avoiding such pitfalls. We summarize the progress of the effort so far, and describe our desired end state of "robust usefulness", along with steps that might be required to get us there.

Key facts. Cited by 28 papers in the map (data through 7 Oct 2026). Builds on 34 of its 92 references that are in the map. First posted on arXiv 9 May 2021.

Builds on

34 of 92 references.

References · 34

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  3. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  4. 2016MS MARCO: A Human Generated MAchine Reading COmprehension Datasetcited by 1,591
  5. 2019Passage Re-ranking with BERTcited by 629
  6. 2020Document Ranking with a Pretrained Sequence-to-Sequence Modelcited by 373
  7. 2020Pretrained Transformers for Text Ranking: BERT and Beyondcited by 266
  8. 2021Overview of the TREC 2020 deep learning trackcited by 193
  9. 2020CORD-19: The COVID-19 Open Research Datasetcited by 179
  10. 2017End-to-End Neural Ad-hoc Ranking with Kernel Poolingcited by 85
  11. 2020Few-Shot Generative Conversational Query Rewritingcited by 85
  12. 2016Learning to Match Using Local and Distributed Representations of Text for Web Searchcited by 68
22 more
  1. 2020TREC CAsT 2019: The Conversational Assistance Track Overviewcited by 64
  2. 2017Neural Ranking Models with Weak Supervisioncited by 61
  3. 2020Query Resolution for Conversational Search with Limited Supervisioncited by 52
  4. 2020BERT-QE: Contextualized Query Expansion for Document Re-rankingcited by 40
  5. 2020PROP: Pre-training with Representative Words Prediction for Ad-hoc Retrievalcited by 40
  6. 2020TREC-COVID: Constructing a Pandemic Information Retrieval Test Collectioncited by 23
  7. 2021TREC Deep Learning Track: Reusable Test Collections in the Large Data Regimecited by 23
  8. 2020Covidex: Neural Ranking Models and Keyword Search Infrastructure for the COVID-19 Open Research Datasetcited by 15
  9. 2016Learning Latent Vector Spaces for Product Searchcited by 14
  10. 2020Selective Weak Supervision for Neural Information Retrievalcited by 10
  11. 2020Rapidly Deploying a Neural Search Engine for the COVID-19 Open Research Dataset: Preliminary Thoughts and Lessons Learnedcited by 10
  12. 2020Topic Propagation in Conversational Searchcited by 10
  13. 2018Optimizing Query Evaluations using Reinforcement Learning for Web Searchcited by 9
  14. 2017Neural Ranking Models with Multiple Document Fieldscited by 7
  15. 2017Benchmark for Complex Answer Retrievalcited by 6
  16. 2020RRF102: Meeting the TREC-COVID Challenge with a 100+ Runs Ensemblecited by 6
  17. 2020SLEDGE-Z: A Zero-Shot Baseline for COVID-19 Literature Searchcited by 6
  18. 2020Long Document Ranking with Query-Directed Sparse Transformercited by 4
  19. 2020CMT in TREC-COVID Round 2: Mitigating the Generalization Gaps from Web to Special Domain Searchcited by 4
  20. 2021Towards Meaningful Statements in IR Evaluation. Mapping Evaluation Measures to Interval Scalescited by 4
  21. 2020SLEDGE: A Simple Yet Effective Baseline for COVID-19 Scientific Knowledge Searchcited by 2
  22. 2021Open-Domain Conversational Search Assistant with Transformerscited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 1; 2022: 5; 2023: 7; 2024: 5; 2025: 6; 2026: 4.

28 papers.

Sort:

By area:

  1. 2023GEO: Generative Engine Optimizationcited by 53
  2. 2022Making a MIRACL: Multilingual Information Retrieval Across a Continuum of Languagescited by 47
  3. 2021On Cross-Lingual Retrieval with Multilingual Text Encoderscited by 22
  4. 2025Overview of the TREC 2022 deep learning trackcited by 22
  5. 2025Overview of the TREC 2021 deep learning trackcited by 18
  6. 2022Parameter-Efficient Neural Reranking for Cross-Lingual and Multilingual Retrievalcited by 17
  7. 2023MGeo: Multi-Modal Geographic Pre-Training Methodcited by 12
  8. 2025Overview of the TREC 2023 deep learning trackcited by 9
  9. 2024A Reproducibility Study of PLAIDcited by 8
  10. 2023Revealing the Blind Spot of Sentence Encoder Evaluation by HEROScited by 5
  11. 2025Generative Product Recommendations for Implicit Superlative Queriescited by 5
  12. 2022Disentangled Modeling of Domain and Relevance for Adaptable Dense Retrievalcited by 4
16 more
  1. 2023A Massive Scale Semantic Similarity Dataset of Historical Englishcited by 4
  2. 2025LEANN: A Low-Storage Vector Indexcited by 4
  3. 2023The Role of Relevance in Fair Rankingcited by 3
  4. 2024Neural Machine Unrankingcited by 2
  5. 2025Aligning Extraction and Generation for Robust Retrieval-Augmented Generationcited by 2
  6. 2026Isolated but Exposed: Persistence-Based Memory Extraction Attack on LLM Agentscited by 2
  7. 2022ZusammenQA: Data Augmentation with Specialized Models for Cross-lingual Open-retrieval Question Answering Systemcited by 1
  8. 2022Are We There Yet? A Decision Framework for Replacing Term Based Retrieval with Dense Retrieval Systemscited by 1
  9. 2023Annotating Data for Fine-Tuning a Neural Ranker? Current Active Learning Strategies are not Better than Random Selectioncited by 1
  10. 2024Verif.ai: Towards an Open-Source Scientific Generative Question-Answering System with Referenced and Verifiable Answerscited by 1
  11. 2024Scientific QA System with Verifiable Answerscited by 1
  12. 2024Neural Corrective Machine Unrankingcited by 1
  13. 2026AgentSearchBench: A Benchmark for AI Agent Search in the Wildcited by 1
  14. 2023Towards Consistency Filtering-Free Unsupervised Learning for Dense Retrievalcited by 0
  15. 2026Beyond Caption-Based Queries for Video Moment Retrievalcited by 0
  16. 2026STORM: Stepwise Token Optimization with Reward-Guided Beam Searchcited by 0