mlsanityData to 14 Oct 2026

FreshStack: Building Realistic Benchmarks for Evaluating Retrieval on Technical Documents

arXiv 2504.13128 ↗First posted 17 Apr 2025cs.IR cs.AI cs.CLcited by 15

Not on the map. Why?

Abstract (arXiv)

We introduce FreshStack, a holistic framework for automatically building information retrieval (IR) evaluation benchmarks by incorporating challenging questions and answers. FreshStack conducts the following steps: (1) automatic corpus collection from code and technical documentation, (2) nugget generation from community-asked questions and answers, and (3) nugget-level support, retrieving documents using a fusion of retrieval techniques and hybrid architectures. We use FreshStack to build five datasets on fast-growing, recent, and niche topics to ensure the tasks are sufficiently challenging. On FreshStack, existing retrieval models, when applied out-of-the-box, significantly underperform oracle approaches on all five topics, denoting plenty of headroom to improve IR quality. In addition, we identify cases where rerankers do not improve first-stage retrieval accuracy (two out of five topics) and oracle context helps an LLM generator generate a high-quality RAG answer. We hope FreshStack will facilitate future work toward constructing realistic, scalable, and uncontaminated IR and RAG evaluation benchmarks.

Key facts. Cited by 15 papers in the map (data through 14 Oct 2026). Builds on 59 of its 109 references that are in the map. First posted on arXiv 17 Apr 2025.

Builds on

59 of 109 references.

References · 59

  1. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  2. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,754
  3. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,455
  4. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,373
  5. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,955
  6. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,141
  7. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 2,004
  8. 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,998
  9. 2016MS MARCO: A Human Generated MAchine Reading COmprehension Datasetcited by 1,605
  10. 2021Improving language models by retrieving from trillions of tokenscited by 1,293
  11. 2021BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Modelscited by 1,252
  12. 2020Leveraging Passage Retrieval with Generative Models for Open Domain Question Answeringcited by 1,210
38 more
  1. 2024M3-Embedding: Multi-Linguality, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillationcited by 1,073
  2. 2019Generalization through Memorization: Nearest Neighbor Language Modelscited by 800
  3. 2021CodeXGLUE: A Machine Learning Benchmark Dataset for Code Understanding and Generationcited by 744
  4. 2023Towards General Text Embeddings with Multi-stage Contrastive Learningcited by 723
  5. 2023In-Context Retrieval-Augmented Language Modelscited by 702
  6. 2019CodeSearchNet Challenge: Evaluating the State of Semantic Code Searchcited by 654
  7. 2023Enabling Large Language Models to Generate Text with Citationscited by 595
  8. 2022Precise Zero-Shot Dense Retrieval without Relevance Labelscited by 517
  9. 2021ColBERTv2: Effective and Efficient Retrieval via Lightweight Late Interactioncited by 476
  10. 2022Large Language Models Struggle to Learn Long-Tail Knowledgecited by 458
  11. 2023RepoCoder: Repository-Level Code Completion Through Iterative Retrieval and Generationcited by 367
  12. 2023CodeT5+: Open Code Large Language Models for Code Understanding and Generationcited by 333
  13. 2023FreshLLMs: Refreshing Large Language Models with Search Engine Augmentationcited by 318
  14. 2023RAGTruth: A Hallucination Corpus for Developing Trustworthy Retrieval-Augmented Language Modelscited by 283
  15. 2024LiveBench: A Challenging, Contamination-Limited LLM Benchmarkcited by 262
  16. 2022RealTime QA: What's the Answer Right Now?cited by 244
  17. 2024Fact, Fetch, and Reason: A Unified Evaluation of Retrieval-Augmented Generationcited by 162
  18. 2024BRIGHT: A Realistic and Challenging Benchmark for Reasoning-Intensive Retrievalcited by 158
  19. 2024Assisting in Writing Wikipedia-like Articles From Scratch with Large Language Modelscited by 154
  20. 2024jina-embeddings-v3: Multilingual Embeddings With Task LoRAcited by 141
  21. 2023Large language models can accurately predict searcher preferencescited by 131
  22. 2020XOR QA: Cross-lingual Open-Retrieval Question Answeringcited by 121
  23. 2024Jasper and Stella: distillation of SOTA embedding modelscited by 112
  24. 2023Jina Embeddings 2: 8192-Token General-Purpose Text Embeddings for Long Documentscited by 104
  25. 2024CodeRAG-Bench: Can Retrieval Augment Code Generation?cited by 91
  26. 2023Perspectives on Large Language Models for Relevance Judgmentcited by 85
  27. 2024CoIR: A Comprehensive Benchmark for Code Information Retrieval Modelscited by 60
  28. 2024UMBRELA: UMbrela is the (Open-Source Reproduction of the) Bing RELevance Assessorcited by 42
  29. 2024CoRNStack: High-Quality Contrastive Data for Better Code Retrieval and Rerankingcited by 37
  30. 2024RAR-b: Reasoning as Retrieval Benchmarkcited by 36
  31. 2024Ragnar"ok: A Reusable RAG Framework and Baselines for TREC 2024 Retrieval-Augmented Generation Trackcited by 32
  32. 2024Customized Retrieval Augmented Generation and Benchmarking for EDA Tool Documentation QAcited by 28
  33. 2024Initial Nugget Evaluation Results for the TREC 2024 RAG Track with the AutoNuggetizer Frameworkcited by 28
  34. 2024AttributionBench: How Hard is Automatic Attribution Evaluation?cited by 26
  35. 2024CLAPNQ: Cohesive Long-form Answers from Passages in Natural Questions for RAG systemscited by 25
  36. 2021TREC Deep Learning Track: Reusable Test Collections in the Large Data Regimecited by 23
  37. 2024Researchy Questions: A Dataset of Multi-Perspective, Decompositional Questions for LLM Web Agentscited by 23
  38. 2024On the Evaluation of Machine-Generated Reportscited by 23

50 most cited of 59 All 59 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 4; 2026: 11.

15 papers.

Sort:

By area:

  1. 2025On the Theoretical Limitations of Embedding-Based Retrievalcited by 104
  2. 2025SealQA: Raising the Bar for Reasoning in Search-Augmented Language Modelscited by 75
  3. 2025DeepScholar-Bench: A Live Benchmark and Automated Evaluation for Generative Research Synthesiscited by 34
  4. 2026ViDoRe V3: A Comprehensive Evaluation of Retrieval Augmented Generation in Complex Real-World Scenarioscited by 26
  5. 2025The Great Nugget Recall: Automating Fact Extraction and RAG Evaluation with Large Language Modelscited by 24
  6. 2026OBLIQ-Bench: Exposing Overlooked Bottlenecks in Modern Retrievers with Latent and Implicit Queriescited by 8
  7. 2026CORE-Bench: A Comprehensive Benchmark for Code Retrieval in the Era of Agentic Codingcited by 7
  8. 2026MiroEval: Benchmarking Multimodal Deep Research Agents in Process and Outcomecited by 6
  9. 2026Overview of the TREC 2025 Retrieval Augmented Generation (RAG) Trackcited by 3
  10. 2026Skilled AI Agents for Embedded and IoT Systems Developmentcited by 3
  11. 2026IRPAPERS: A Visual Document Benchmark for Scientific Retrieval and Question Answeringcited by 2
  12. 2026IRB: Automated Generation of Robust Factuality Benchmarkscited by 1
3 more
  1. 2026Still Fresh? Evaluating Temporal Drift in Retrieval Benchmarkscited by 1
  2. 2026MDKeyChunker: Single-Call LLM Enrichment with Rolling Keys and Key-Based Restructuring for High-Accuracy RAGcited by 0
  3. 2026ExperienceIndex: Artifact-Grounded Memorycited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Test-time Corpus Feedback: From Retrieval to RAG
  2. 2025mmRAG: A Modular Benchmark for Retrieval-Augmented Generation over Text, Tables, and Knowledge Graphs
  3. 2026Beyond the Reranker: Do RAG Retrieval Enhancements Help Once a Strong Reranker Is Present?
  4. 2025Frustratingly Simple Retrieval Improves Challenging, Reasoning-Intensive Benchmarks
  5. 2025Rankify: A Comprehensive Python Toolkit for Retrieval, Re-Ranking, and Retrieval-Augmented Generation
  6. 2026SciRet: A Compute-Aware Empirical Study of Retrieval and Reranking for Scientific RAG
  7. 2024Enhancing Q&A Text Retrieval with Ranking Models: Benchmarking, fine-tuning and deploying Rerankers for RAG
  8. 2024UDA: A Benchmark Suite for Retrieval Augmented Generation in Real-world Document Analysis