REANIMATOR: Reanimate Retrieval Test Collections with Extracted and Synthetic Resources
On the map: Retrieval-augmented generation → Generation and context integration
Abstract (arXiv)
Retrieval test collections are essential for evaluating information retrieval systems, yet they often lack generalizability across tasks. To overcome this limitation, we introduce REANIMATOR, a versatile framework designed to enable the repurposing of existing test collections by enriching them with extracted and synthetic resources. REANIMATOR enhances test collections from PDF files by parsing full texts and machine-readable tables, as well as related contextual information. It then employs state-of-the-art large language models to produce synthetic relevance labels. Including an optional human-in-the-loop step can help validate the resources that have been extracted and generated. We demonstrate its potential with a revitalized version of the TREC-COVID test collection, showcasing the development of a retrieval-augmented generation system and evaluating the impact of tables on retrieval-augmented generation. REANIMATOR enables the reuse of test collections for new applications, lowering costs and broadening the utility of legacy resources.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Generation and context integration, one of 15 directions in Retrieval-augmented generation. Builds on 21 of its 45 references that are in the map. First posted on arXiv 10 Apr 2025.
Builds on
21 of 45 references.
References · 21
- 2024A Survey on LLM-as-a-Judgecited by 1,384
- 2024Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preferencecited by 1,151
- 2023OpenAssistant Conversations -- Democratizing Large Language Model Alignmentcited by 712
- 2019TabFact: A Large-scale Dataset for Table-based Fact Verificationcited by 425
- 2024Benchmarking Retrieval-Augmented Generation for Medicinecited by 326
- 2020CORD-19: The COVID-19 Open Research Datasetcited by 179
- 2024Evaluation of Retrieval-Augmented Generation: A Surveycited by 146
- 2023Large language models can accurately predict searcher preferencescited by 129
- 2021Open Domain Question Answering over Tables via Dense Retrievalcited by 74
- 2023Elo Uncovered: Robustness and Best Practices in Language Model Evaluationcited by 58
- 2024UMBRELA: UMbrela is the (Open-Source Reproduction of the) Bing RELevance Assessorcited by 41
- 2021Simplified Data Wrangling with ir_datasetscited by 27
9 more
- 2024Don't Use LLMs to Make Relevance Judgmentscited by 26
- 2020TREC-COVID: Constructing a Pandemic Information Retrieval Test Collectioncited by 23
- 2024Synthetic Test Collections for Retrieval Evaluationcited by 18
- 2017Scientific Table Search Using Keyword Queriescited by 6
- 2020How to Measure the Reproducibility of System-oriented IR Experimentscited by 6
- 2020TabSim: A Siamese Neural Network for Accurate Estimation of Table Similaritycited by 5
- 2024Evaluation of Temporal Change in IR Test Collectionscited by 4
- 2023Simulating Users in Interactive Web Table Retrievalcited by 1
- 2024Investigating Bias in Political Search Query Suggestions by Relative Comparison with LLMscited by 1
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2025FreshStack: Building Realistic Benchmarks for Evaluating Retrieval on Technical Documents
- 2025Enhancing Retrieval-Augmented Generation with Two-Stage Retrieval: FlashRank Reranking and Query Expansion
- 2026Beyond the Reranker: Do RAG Retrieval Enhancements Help Once a Strong Reranker Is Present?
- 2026FRESCO: Benchmarking and Optimizing Re-rankers for Evolving Semantic Conflict in Retrieval-Augmented Generation
- 2024JudgeRank: Leveraging Large Language Models for Reasoning-Intensive Reranking
- 2025Towards Understanding Bias in Synthetic Data for Evaluation
- 2025Evaluating Hybrid Retrieval Augmented Generation using Dynamic Test Sets: LiveRAG Challenge
- 2023Enhancing Retrieval-Augmented Large Language Models with Iterative Retrieval-Generation Synergy