# REANIMATOR: Reanimate Retrieval Test Collections with Extracted and Synthetic Resources

arXiv: https://arxiv.org/abs/2504.07584 · HTML: https://mlsanity.com/paper/2504.07584/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Generation and context integration, one of 15 directions in Retrieval-augmented generation.
- Builds on 21 of its 45 references that are in the map.
- First posted on arXiv 10 Apr 2025.

## Placed in

- Retrieval-augmented generation / [Generation and context integration](https://mlsanity.com/rag/generation-and-context-integration/)

## Builds on (most cited first, 21 of 21)

- [A Survey on LLM-as-a-Judge](https://mlsanity.com/paper/2411.15594/) (2024)
- [Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference](https://mlsanity.com/paper/2403.04132/) (2024)
- [OpenAssistant Conversations -- Democratizing Large Language Model Alignment](https://mlsanity.com/paper/2304.07327/) (2023)
- [TabFact: A Large-scale Dataset for Table-based Fact Verification](https://mlsanity.com/paper/1909.02164/) (2019)
- [Benchmarking Retrieval-Augmented Generation for Medicine](https://mlsanity.com/paper/2402.13178/) (2024)
- [CORD-19: The COVID-19 Open Research Dataset](https://mlsanity.com/paper/2004.10706/) (2020)
- [Evaluation of Retrieval-Augmented Generation: A Survey](https://mlsanity.com/paper/2405.07437/) (2024)
- [Large language models can accurately predict searcher preferences](https://mlsanity.com/paper/2309.10621/) (2023)
- [Open Domain Question Answering over Tables via Dense Retrieval](https://mlsanity.com/paper/2103.12011/) (2021)
- [Elo Uncovered: Robustness and Best Practices in Language Model Evaluation](https://mlsanity.com/paper/2311.17295/) (2023)
- [UMBRELA: UMbrela is the (Open-Source Reproduction of the) Bing RELevance Assessor](https://mlsanity.com/paper/2406.06519/) (2024)
- [Simplified Data Wrangling with ir_datasets](https://mlsanity.com/paper/2103.02280/) (2021)
- [Don't Use LLMs to Make Relevance Judgments](https://mlsanity.com/paper/2409.15133/) (2024)
- [TREC-COVID: Constructing a Pandemic Information Retrieval Test Collection](https://mlsanity.com/paper/2005.04474/) (2020)
- [Synthetic Test Collections for Retrieval Evaluation](https://mlsanity.com/paper/2405.07767/) (2024)
- [Scientific Table Search Using Keyword Queries](https://mlsanity.com/paper/1707.03423/) (2017)
- [How to Measure the Reproducibility of System-oriented IR Experiments](https://mlsanity.com/paper/2010.13447/) (2020)
- [TabSim: A Siamese Neural Network for Accurate Estimation of Table Similarity](https://mlsanity.com/paper/2008.10856/) (2020)
- [Evaluation of Temporal Change in IR Test Collections](https://mlsanity.com/paper/2407.01373/) (2024)
- [Simulating Users in Interactive Web Table Retrieval](https://mlsanity.com/paper/2310.11931/) (2023)
- [Investigating Bias in Political Search Query Suggestions by Relative Comparison with LLMs](https://mlsanity.com/paper/2410.23879/) (2024)

## Abstract (arXiv)

Retrieval test collections are essential for evaluating information retrieval systems, yet they often lack generalizability across tasks. To overcome this limitation, we introduce REANIMATOR, a versatile framework designed to enable the repurposing of existing test collections by enriching them with extracted and synthetic resources. REANIMATOR enhances test collections from PDF files by parsing full texts and machine-readable tables, as well as related contextual information. It then employs state-of-the-art large language models to produce synthetic relevance labels. Including an optional human-in-the-loop step can help validate the resources that have been extracted and generated. We demonstrate its potential with a revitalized version of the TREC-COVID test collection, showcasing the development of a retrieval-augmented generation system and evaluating the impact of tables on retrieval-augmented generation. REANIMATOR enables the reuse of test collections for new applications, lowering costs and broadening the utility of legacy resources.