# Still Fresh? Evaluating Temporal Drift in Retrieval Benchmarks

arXiv: https://arxiv.org/abs/2603.04532 · HTML: https://mlsanity.com/paper/2603.04532/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 40 references that are in the map.
- First posted on arXiv 4 Mar 2026.

## Builds on (most cited first, 15 of 15)

- [MS MARCO: A Human Generated MAchine Reading COmprehension Dataset](https://mlsanity.com/paper/1611.09268/) (2016)
- [Qwen3 Embedding: Advancing Text Embedding and Reranking Through Foundation Models](https://mlsanity.com/paper/2506.05176/) (2025)
- [M3-Embedding: Multi-Linguality, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillation](https://mlsanity.com/paper/2402.03216/) (2024)
- [Towards General Text Embeddings with Multi-stage Contrastive Learning](https://mlsanity.com/paper/2308.03281/) (2023)
- [Large language models can accurately predict searcher preferences](https://mlsanity.com/paper/2309.10621/) (2023)
- [Jasper and Stella: distillation of SOTA embedding models](https://mlsanity.com/paper/2412.19048/) (2024)
- [Command A: An Enterprise-Ready Large Language Model](https://mlsanity.com/paper/2504.00698/) (2025)
- [The Great Nugget Recall: Automating Fact Extraction and RAG Evaluation with Large Language Models](https://mlsanity.com/paper/2504.15068/) (2025)
- [Overview of the TREC 2022 deep learning track](https://mlsanity.com/paper/2507.10865/) (2025)
- [Overview of the TREC 2021 deep learning track](https://mlsanity.com/paper/2507.08191/) (2025)
- [FreshStack: Building Realistic Benchmarks for Evaluating Retrieval on Technical Documents](https://mlsanity.com/paper/2504.13128/) (2025)
- [Variations in Relevance Judgments and the Shelf Life of Test Collections](https://mlsanity.com/paper/2502.20937/) (2025)
- [Evaluation of Temporal Change in IR Test Collections](https://mlsanity.com/paper/2407.01373/) (2024)
- [Intelligent Spark Agents: A Modular LangGraph Framework for Scalable, Visualized, and Enhanced Big Data Machine Learning Workflows](https://mlsanity.com/paper/2412.01490/) (2024)
- [RAG Playground: A Framework for Systematic Evaluation of Retrieval Strategies and Prompt Engineering in RAG Systems](https://mlsanity.com/paper/2412.12322/) (2024)

## Built on by (most cited first, 1 of 1)

- [MiroEval: Benchmarking Multimodal Deep Research Agents in Process and Outcome](https://mlsanity.com/paper/2603.28407/) (2026)

## Abstract (arXiv)

Information retrieval (IR) benchmarks typically follow the Cranfield paradigm, relying on static and predefined corpora. However, temporal changes in technical corpora, such as API deprecations and code reorganizations, can render existing benchmarks stale. In our work, we investigate how temporal corpus drift affects FreshStack, a retrieval benchmark focused on technical domains. We examine two independent corpus snapshots of FreshStack from October 2024 and October 2025 to answer questions about LangChain. Our analysis shows that all but one query posed in 2024 remain fully supported by the 2025 corpus, as relevant documents "migrate" from LangChain to competitor repositories, such as LlamaIndex. Next, we compare the accuracy of retrieval models on both snapshots and observe only minor shifts in model rankings, with overall strong correlation of up to 0.978 Kendall τ at Recall@50. These results suggest that retrieval benchmarks re-judged with evolving temporal corpora can remain reliable for retrieval evaluation. We publicly release all our artifacts at https://github.com/fresh-stack/driftbench.