# Overview of the TREC 2021 deep learning track

arXiv: https://arxiv.org/abs/2507.08191 · HTML: https://mlsanity.com/paper/2507.08191/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- Builds on 9 of its 19 references that are in the map.
- First posted on arXiv 10 Jul 2025.

## Builds on (most cited first, 9 of 9)

- [MS MARCO: A Human Generated MAchine Reading COmprehension Dataset](https://mlsanity.com/paper/1611.09268/) (2016)
- [Latent Retrieval for Weakly Supervised Open Domain Question Answering](https://mlsanity.com/paper/1906.00300/) (2019)
- [Overview of the TREC 2019 deep learning track](https://mlsanity.com/paper/2003.07820/) (2020)
- [Overview of the TREC 2020 deep learning track](https://mlsanity.com/paper/2102.07662/) (2021)
- [MS MARCO: Benchmarking Ranking Models in the Large-Data Regime](https://mlsanity.com/paper/2105.04021/) (2021)
- [TREC Deep Learning Track: Reusable Test Collections in the Large Data Regime](https://mlsanity.com/paper/2104.09399/) (2021)
- [Significant Improvements over the State of the Art? A Case Study of the MS MARCO Document Ranking Leaderboard](https://mlsanity.com/paper/2102.12887/) (2021)
- [Optimizing Query Evaluations using Reinforcement Learning for Web Search](https://mlsanity.com/paper/1804.04410/) (2018)
- [Incorporating Query Term Independence Assumption for Efficient Retrieval and Ranking using Deep Neural Networks](https://mlsanity.com/paper/1907.03693/) (2019)

## Built on by (most cited first, 18 of 18)

- [RAVine: Reality-Aligned Evaluation for Agentic Search](https://mlsanity.com/paper/2507.16725/) (2025)
- [An Efficient Rubric-based Generative Verifier for Search-Augmented LLMs](https://mlsanity.com/paper/2510.14660/) (2025)
- [Learning from Emptiness: De-biasing Listwise Rerankers with Content-Agnostic Probability Calibration](https://mlsanity.com/paper/2604.10150/) (2026)
- [Test-time Corpus Feedback: From Retrieval to RAG](https://mlsanity.com/paper/2508.15437/) (2025)
- [Mitigating the Threshold Priming Effect in Large Language Model-Based Relevance Judgments via Personality Infusing](https://mlsanity.com/paper/2512.00390/) (2025)
- [Scaling Laws for Embedding Dimension in Information Retrieval](https://mlsanity.com/paper/2602.05062/) (2026)
- [Understanding Matching Mechanisms in Cross-Encoders](https://mlsanity.com/paper/2507.14604/) (2025)
- [Lighting the Way for BRIGHT: Reproducible Baselines with Anserini, Pyserini, and RankLLM](https://mlsanity.com/paper/2509.02558/) (2025)
- [Judging with Personality and Confidence: A Study on Personality-Conditioned LLM Relevance Assessment](https://mlsanity.com/paper/2601.01862/) (2026)
- [Still Fresh? Evaluating Temporal Drift in Retrieval Benchmarks](https://mlsanity.com/paper/2603.04532/) (2026)
- [Blending Learning to Rank and Dense Representations for Efficient and Effective Cascades](https://mlsanity.com/paper/2510.16393/) (2025)
- [MACA: A Framework for Distilling Trustworthy LLMs into Efficient Retrievers](https://mlsanity.com/paper/2601.00926/) (2026)
- [Re-Rankers as Relevance Judges](https://mlsanity.com/paper/2601.04455/) (2026)
- [BlitzRank: Principled Zero-shot Ranking Agents with Tournament Graphs](https://mlsanity.com/paper/2602.05448/) (2026)
- [RAQG-QPP: Query Performance Prediction with Retrieved Query Variants and Retrieval Augmented Query Generation](https://mlsanity.com/paper/2604.27244/) (2026)
- [Search for Coverage: Learning Coverage-Aware Retrieval with Augmented Sub-Question Answerability](https://mlsanity.com/paper/2605.28522/) (2026)
- [Q2D-Web: A Large-Scale Benchmark for Retrieval in Agentic RAG Systems](https://mlsanity.com/paper/2609.08887/) (2026)
- [No More Free Lunch: Corpus Task Complexity Matters as Corpora Grow](https://mlsanity.com/paper/2609.29245/) (2026)

## Abstract (arXiv)

This is the third year of the TREC Deep Learning track. As in previous years, we leverage the MS MARCO datasets that made hundreds of thousands of human annotated training labels available for both passage and document ranking tasks. In addition, this year we refreshed both the document and the passage collections which also led to a nearly four times increase in the document collection size and nearly 16 times increase in the size of the passage collection. Deep neural ranking models that employ large scale pretraininig continued to outperform traditional retrieval methods this year. We also found that single stage retrieval can achieve good performance on both tasks although they still do not perform at par with multistage retrieval pipelines. Finally, the increase in the collection size and the general data refresh raised some questions about completeness of NIST judgments and the quality of the training labels that were mapped to the new collections from the old ones which we discuss in this report.