# Synthetic Test Collections for Retrieval Evaluation

arXiv: https://arxiv.org/abs/2405.07767 · HTML: https://mlsanity.com/paper/2405.07767/

## Key facts

- Cited by 19 papers in the map (data through 14 Oct 2026).
- Builds on 7 of its 21 references that are in the map.
- First posted on arXiv 13 May 2024.

## Builds on (most cited first, 7 of 7)

- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Models](https://mlsanity.com/paper/2104.08663/) (2021)
- [Can Large Language Models Be an Alternative to Human Evaluations?](https://mlsanity.com/paper/2305.01937/) (2023)
- [Large language models can accurately predict searcher preferences](https://mlsanity.com/paper/2309.10621/) (2023)
- [Perspectives on Large Language Models for Relevance Judgment](https://mlsanity.com/paper/2304.09161/) (2023)
- [Synthetic data generation for end-to-end thermal infrared tracking](https://mlsanity.com/paper/1806.01013/) (2018)
- [TREC Deep Learning Track: Reusable Test Collections in the Large Data Regime](https://mlsanity.com/paper/2104.09399/) (2021)

## Built on by (most cited first, 19 of 19)

- [Retrieve, Annotate, Evaluate, Repeat: Leveraging Multimodal LLMs for Large-Scale Product Retrieval Evaluation](https://mlsanity.com/paper/2409.11860/) (2024)
- [Towards Understanding Bias in Synthetic Data for Evaluation](https://mlsanity.com/paper/2506.10301/) (2025)
- [Total Recall QA: A Verifiable Evaluation Suite for Deep Research Agents](https://mlsanity.com/paper/2603.18516/) (2026)
- [Enhancing Tourism Recommender Systems for Sustainable City Trips Using Retrieval-Augmented Generation](https://mlsanity.com/paper/2409.18003/) (2024)
- [Tip of the Tongue Query Elicitation for Simulated Evaluation](https://mlsanity.com/paper/2502.17776/) (2025)
- [DisastIR: A Comprehensive Information Retrieval Benchmark for Disaster Management](https://mlsanity.com/paper/2505.15856/) (2025)
- [The Decoy Dilemma in Online Medical Information Evaluation: A Comparative Study of Credibility Assessments by LLM and Human Judges](https://mlsanity.com/paper/2411.15396/) (2024)
- [HotelMatch-LLM: Joint Multi-Task Training of Small and Large Language Models for Efficient Multimodal Hotel Retrieval](https://mlsanity.com/paper/2506.07296/) (2025)
- [PosIR: Position-Aware Heterogeneous Information Retrieval Benchmark](https://mlsanity.com/paper/2601.08363/) (2026)
- [ADORE: Iterative Query Expansion with Retrieval-Grounded Relevance Feedback](https://mlsanity.com/paper/2606.13905/) (2026)
- [On the Necessity of World Knowledge for Mitigating Missing Labels in Extreme Classification](https://mlsanity.com/paper/2408.09585/) (2024)
- [REANIMATOR: Reanimate Retrieval Test Collections with Extracted and Synthetic Resources](https://mlsanity.com/paper/2504.07584/) (2025)
- [Towards Fair Rankings: Leveraging LLMs for Gender Bias Detection and Measurement](https://mlsanity.com/paper/2506.22372/) (2025)
- [FinCPRG: A Bidirectional Generation Pipeline for Hierarchical Queries and Rich Relevance in Financial Chinese Passage Retrieval](https://mlsanity.com/paper/2508.02222/) (2025)
- [Re-Rankers as Relevance Judges](https://mlsanity.com/paper/2601.04455/) (2026)
- [From Noise to Order: Learning to Rank via Denoising Diffusion](https://mlsanity.com/paper/2602.11453/) (2026)
- [SPECTRA: Synthetic IR Test Collections with Relevance Oracles and Controlled Distractor Diagnostics](https://mlsanity.com/paper/2605.31575/) (2026)
- [The "Curse of Knowledge" in LLM Query Simulation: Concept Provenance for Tracing Answer-Side Intrusion](https://mlsanity.com/paper/2608.25245/) (2026)
- [FASHION-RQ: Mining Hierarchical Semantic Structure for Automated Evaluation of Fashion Retrieval](https://mlsanity.com/paper/2610.17841/) (2026)

## Abstract (arXiv)

Test collections play a vital role in evaluation of information retrieval (IR) systems. Obtaining a diverse set of user queries for test collection construction can be challenging, and acquiring relevance judgments, which indicate the appropriateness of retrieved documents to a query, is often costly and resource-intensive. Generating synthetic datasets using Large Language Models (LLMs) has recently gained significant attention in various applications. In IR, while previous work exploited the capabilities of LLMs to generate synthetic queries or documents to augment training data and improve the performance of ranking models, using LLMs for constructing synthetic test collections is relatively unexplored. Previous studies demonstrate that LLMs have the potential to generate synthetic relevance judgments for use in the evaluation of IR systems. In this paper, we comprehensively investigate whether it is possible to use LLMs to construct fully synthetic test collections by generating not only synthetic judgments but also synthetic queries. In particular, we analyse whether it is possible to construct reliable synthetic test collections and the potential risks of bias such test collections may exhibit towards LLM-based models. Our experiments indicate that using LLMs it is possible to construct synthetic test collections that can reliably be used for retrieval evaluation.