# SPECTRA: Synthetic IR Test Collections with Relevance Oracles and Controlled Distractor Diagnostics

arXiv: https://arxiv.org/abs/2605.31575 · HTML: https://mlsanity.com/paper/2605.31575/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 12 of its 23 references that are in the map.
- First posted on arXiv 29 May 2026.

## Builds on (most cited first, 12 of 12)

- [MS MARCO: A Human Generated MAchine Reading COmprehension Dataset](https://mlsanity.com/paper/1611.09268/) (2016)
- [BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Models](https://mlsanity.com/paper/2104.08663/) (2021)
- [Promptagator: Few-shot Dense Retrieval From 8 Examples](https://mlsanity.com/paper/2209.11755/) (2022)
- [Document Expansion by Query Prediction](https://mlsanity.com/paper/1904.08375/) (2019)
- [InPars: Data Augmentation for Information Retrieval using Large Language Models](https://mlsanity.com/paper/2202.05144/) (2022)
- [Pyserini: An Easy-to-Use Python Toolkit to Support Replicable IR Research with Sparse and Dense Representations](https://mlsanity.com/paper/2102.10073/) (2021)
- [Synthetic Test Collections for Retrieval Evaluation](https://mlsanity.com/paper/2405.07767/) (2024)
- [Efficient Representations for High-Cardinality Categorical Variables in Machine Learning](https://mlsanity.com/paper/2501.05646/) (2025)
- [SynDL: A Large-Scale Synthetic Test Collection for Passage Retrieval](https://mlsanity.com/paper/2408.16312/) (2024)
- [Automating Date Format Detection for Data Visualization](https://mlsanity.com/paper/2501.05640/) (2025)
- [Enhanced Estimation Techniques for Certified Radii in Randomized Smoothing](https://mlsanity.com/paper/2503.08801/) (2025)
- [GenTREC: The First Test Collection Generated by Large Language Models for Evaluating Information Retrieval Systems](https://mlsanity.com/paper/2501.02408/) (2025)

## Abstract (arXiv)

Scalable information retrieval testing needs corpora that are large enough to stress index construction, ranking latency, query routing, and evaluation tooling, yet human-judged test collections remain expensive and may be unavailable when documents are private or still under design. This paper introduces SPECTRA, a reproducible framework for generating synthetic text corpora and retrieval test collections through a separation of latent topical structure, surface text realization, metadata controls, query intent generation, and deterministic relevance oracles. The framework is intended as a diagnostic complement to Cranfield-style and TREC-style evaluation, not as a replacement for human assessment. A single-process Python prototype generated corpora up to 60,000 documents and 9.61 million tokens while preserving controllable long-tail vocabulary growth and producing graded relevance labels for 96 queries. In the local simulation study, generation remained close to linear at roughly 12K to 14K documents per second, estimated Zipf slopes stayed near 0.86 in absolute value, and increasing cross-topic distractor text reduced BM25 nDCG@10 from 1.00 at 2% distractors to 0.43 at 36% distractors. These results show that lightweight synthetic corpora can expose retrieval-system scaling and failure modes before costly collection construction begins.