# Overview of the TREC 2023 deep learning track

arXiv: https://arxiv.org/abs/2507.08890 · HTML: https://mlsanity.com/paper/2507.08890/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Builds on 5 of its 16 references that are in the map.
- First posted on arXiv 10 Jul 2025.

## Builds on (most cited first, 5 of 5)

- [MS MARCO: A Human Generated MAchine Reading COmprehension Dataset](https://mlsanity.com/paper/1611.09268/) (2016)
- [Overview of the TREC 2019 deep learning track](https://mlsanity.com/paper/2003.07820/) (2020)
- [Overview of the TREC 2020 deep learning track](https://mlsanity.com/paper/2102.07662/) (2021)
- [MS MARCO: Benchmarking Ranking Models in the Large-Data Regime](https://mlsanity.com/paper/2105.04021/) (2021)
- [Significant Improvements over the State of the Art? A Case Study of the MS MARCO Document Ranking Leaderboard](https://mlsanity.com/paper/2102.12887/) (2021)

## Built on by (most cited first, 9 of 9)

- [An Efficient Rubric-based Generative Verifier for Search-Augmented LLMs](https://mlsanity.com/paper/2510.14660/) (2025)
- [Learning from Emptiness: De-biasing Listwise Rerankers with Content-Agnostic Probability Calibration](https://mlsanity.com/paper/2604.10150/) (2026)
- [BiXSE: Improving Dense Retrieval via Probabilistic Graded Relevance Distillation](https://mlsanity.com/paper/2508.06781/) (2025)
- [Lighting the Way for BRIGHT: Reproducible Baselines with Anserini, Pyserini, and RankLLM](https://mlsanity.com/paper/2509.02558/) (2025)
- [AI Prior Art Search: Semantic Clusters and Evaluation Infrastructure](https://mlsanity.com/paper/2512.18384/) (2025)
- [Re-Rankers as Relevance Judges](https://mlsanity.com/paper/2601.04455/) (2026)
- [BlitzRank: Principled Zero-shot Ranking Agents with Tournament Graphs](https://mlsanity.com/paper/2602.05448/) (2026)
- [Plans for Evaluating Structured Generative Search Summaries](https://mlsanity.com/paper/2605.26400/) (2026)
- [Q2D-Web: A Large-Scale Benchmark for Retrieval in Agentic RAG Systems](https://mlsanity.com/paper/2609.08887/) (2026)

## Abstract (arXiv)

This is the fifth year of the TREC Deep Learning track. As in previous years, we leverage the MS MARCO datasets that made hundreds of thousands of human-annotated training labels available for both passage and document ranking tasks. We mostly repeated last year's design, to get another matching test set, based on the larger, cleaner, less-biased v2 passage and document set, with passage ranking as primary and document ranking as a secondary task (using labels inferred from passage). As we did last year, we sample from MS MARCO queries that were completely held out, unused in corpus construction, unlike the test queries in the first three years. This approach yields a more difficult test with more headroom for improvement. Alongside the usual MS MARCO (human) queries from MS MARCO, this year we generated synthetic queries using a fine-tuned T5 model and using a GPT-4 prompt. The new headline result this year is that runs using Large Language Model (LLM) prompting in some way outperformed runs that use the "nnlm" approach, which was the best approach in the previous four years. Since this is the last year of the track, future iterations of prompt-based ranking can happen in other tracks. Human relevance assessments were applied to all query types, not just human MS MARCO queries. Evaluation using synthetic queries gave similar results to human queries, with system ordering agreement of τ=0.8487. However, human effort was needed to select a subset of the synthetic queries that were usable. We did not see clear evidence of bias, where runs using GPT-4 were favored when evaluated using synthetic GPT-4 queries, or where runs using T5 were favored when evaluated on synthetic T5 queries.