mlsanityData to 14 Oct 2026

TREC Deep Learning Track: Reusable Test Collections in the Large Data Regime

arXiv 2104.09399 ↗First posted 19 Apr 2021cs.IR cs.AI cs.LGcited by 23

Not on the map. Why?

Abstract (arXiv)

The TREC Deep Learning (DL) Track studies ad hoc search in the large data regime, meaning that a large set of human-labeled training data is available. Results so far indicate that the best models with large data may be deep neural networks. This paper supports the reuse of the TREC DL test collections in three ways. First we describe the data sets in detail, documenting clearly and in one place some details that are otherwise scattered in track guidelines, overview papers and in our associated MS MARCO leaderboard pages. We intend this description to make it easy for newcomers to use the TREC DL data. Second, because there is some risk of iteration and selection bias when reusing a data set, we describe the best practices for writing a paper using TREC DL data, without overfitting. We provide some illustrative analysis. Finally we address a number of issues around the TREC DL data, including an analysis of reusability.

Key facts. Cited by 23 papers in the map (data through 14 Oct 2026). Builds on 12 of its 33 references that are in the map. First posted on arXiv 19 Apr 2021.

Builds on

12 of 33 references.

References · 12

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,811
  3. 2016MS MARCO: A Human Generated MAchine Reading COmprehension Datasetcited by 1,605
  4. 2016Bag of Tricks for Efficient Text Classificationcited by 876
  5. 2020Pretrained Transformers for Text Ranking: BERT and Beyondcited by 267
  6. 2021Overview of the TREC 2020 deep learning trackcited by 196
  7. 2016Learning to Match Using Local and Distributed Representations of Text for Web Searchcited by 68
  8. 2020ORCAS: 18 Million Clicked Query-Document Pairs for Analyzing Searchcited by 34
  9. 2019An Updated Duet Model for Passage Re-rankingcited by 12
  10. 2018Optimizing Query Evaluations using Reinforcement Learning for Web Searchcited by 9
  11. 2020Conformer-Kernel with Query Term Independence for Document Retrievalcited by 3
  12. 2020Conformer-Kernel with Query Term Independence at TREC 2020 Deep Learning Trackcited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 1; 2022: 3; 2023: 5; 2024: 2; 2025: 9; 2026: 3.

23 papers.

Sort:

By area:

  1. 2022Shopping Queries Dataset: A Large-Scale ESCI Benchmark for Improving Product Searchcited by 65
  2. 2021MS MARCO: Benchmarking Ranking Models in the Large-Data Regimecited by 28
  3. 2025Overview of the TREC 2022 deep learning trackcited by 22
  4. 2024Synthetic Test Collections for Retrieval Evaluationcited by 19
  5. 2025Overview of the TREC 2021 deep learning trackcited by 18
  6. 2025FreshStack: Building Realistic Benchmarks for Evaluating Retrieval on Technical Documentscited by 15
  7. 2022Multi-Task Retrieval-Augmented Text Generation with Relevance Samplingcited by 10
  8. 2024Reliable Confidence Intervals for Information Retrieval Evaluation Using Generative A.Icited by 9
  9. 2023CAME: Competitively Learning a Mixture-of-Experts Model for First-stage Retrievalcited by 6
  10. 2025From Standalone LLMs to Integrated Intelligence: A Survey of Compound Al Systemscited by 6
  11. 2023Explain like I am BM25: Interpreting a Dense Model's Ranked-List with a Sparse Approximationcited by 5
  12. 2025Towards Natural Language-Based Document Image Retrieval: New Dataset and Benchmarkcited by 5
11 more
  1. 2022mRobust04: A Multilingual Version of the TREC Robust 2004 Benchmarkcited by 3
  2. 2023Parameter-Efficient Sparse Retrievers and Rerankers using Adapterscited by 3
  3. 2023Overview of the TREC 2023 Product Product Search Trackcited by 2
  4. 2023Annotating Data for Fine-Tuning a Neural Ranker? Current Active Learning Strategies are not Better than Random Selectioncited by 1
  5. 2025Optimizing Compound Retrieval Systemscited by 1
  6. 2025Access Paths for Efficient Ordering with Large Language Modelscited by 1
  7. 2025Reproducibility and Artifact Consistency of the SIGIR 2022 Recommender Systems Papers Based on Message Passingcited by 0
  8. 2025REGENT: Relevance-Guided Attention for Entity-Aware Multi-Vector Neural Re-Rankingcited by 0
  9. 2026RMIT-ADM+S at the MMU-RAG NeurIPS 2025 Competitioncited by 0
  10. 2026FRESCO: Benchmarking and Optimizing Re-rankers for Evolving Semantic Conflict in Retrieval-Augmented Generationcited by 0
  11. 2026Reproducing Complex Set-Compositional Information Retrievalcited by 0