Still Fresh? Evaluating Temporal Drift in Retrieval Benchmarks
Not on the map. Why?
Abstract (arXiv)
Information retrieval (IR) benchmarks typically follow the Cranfield paradigm, relying on static and predefined corpora. However, temporal changes in technical corpora, such as API deprecations and code reorganizations, can render existing benchmarks stale. In our work, we investigate how temporal corpus drift affects FreshStack, a retrieval benchmark focused on technical domains. We examine two independent corpus snapshots of FreshStack from October 2024 and October 2025 to answer questions about LangChain. Our analysis shows that all but one query posed in 2024 remain fully supported by the 2025 corpus, as relevant documents "migrate" from LangChain to competitor repositories, such as LlamaIndex. Next, we compare the accuracy of retrieval models on both snapshots and observe only minor shifts in model rankings, with overall strong correlation of up to 0.978 Kendall τ at Recall@50. These results suggest that retrieval benchmarks re-judged with evolving temporal corpora can remain reliable for retrieval evaluation. We publicly release all our artifacts at https://github.com/fresh-stack/driftbench.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 15 of its 40 references that are in the map. First posted on arXiv 4 Mar 2026.
Builds on
15 of 40 references.
References · 15
- 2016MS MARCO: A Human Generated MAchine Reading COmprehension Datasetcited by 1,591
- 2025Qwen3 Embedding: Advancing Text Embedding and Reranking Through Foundation Modelscited by 1,175
- 2024M3-Embedding: Multi-Linguality, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillationcited by 1,050
- 2023Towards General Text Embeddings with Multi-stage Contrastive Learningcited by 705
- 2023Large language models can accurately predict searcher preferencescited by 129
- 2024Jasper and Stella: distillation of SOTA embedding modelscited by 109
- 2025Command A: An Enterprise-Ready Large Language Modelcited by 57
- 2025The Great Nugget Recall: Automating Fact Extraction and RAG Evaluation with Large Language Modelscited by 24
- 2025Overview of the TREC 2022 deep learning trackcited by 22
- 2025Overview of the TREC 2021 deep learning trackcited by 18
- 2025FreshStack: Building Realistic Benchmarks for Evaluating Retrieval on Technical Documentscited by 15
- 2025Variations in Relevance Judgments and the Shelf Life of Test Collectionscited by 5
3 more
- 2024Evaluation of Temporal Change in IR Test Collectionscited by 4
- 2024Intelligent Spark Agents: A Modular LangGraph Framework for Scalable, Visualized, and Enhanced Big Data Machine Learning Workflowscited by 4
- 2024RAG Playground: A Framework for Systematic Evaluation of Retrieval Strategies and Prompt Engineering in RAG Systemscited by 4
Built on by
Citing papers per year: 2026: 1.
1 paper.