mlsanityData to 7 Oct 2026

Ragnar"ok: A Reusable RAG Framework and Baselines for TREC 2024 Retrieval-Augmented Generation Track

arXiv 2406.16828 ↗First posted 24 Jun 2024cs.IR cs.AI cs.CLcited by 31

On the map: Retrieval-augmented generation → RAG pipeline design and evaluation

Abstract (arXiv)

Did you try out the new Bing Search? Or maybe you fiddled around with Google AI~Overviews? These might sound familiar because the modern-day search stack has recently evolved to include retrieval-augmented generation (RAG) systems. They allow searching and incorporating real-time data into large language models (LLMs) to provide a well-informed, attributed, concise summary in contrast to the traditional search paradigm that relies on displaying a ranked list of documents. Therefore, given these recent advancements, it is crucial to have an arena to build, test, visualize, and systematically evaluate RAG-based search systems. With this in mind, we propose the TREC 2024 RAG Track to foster innovation in evaluating RAG systems. In our work, we lay out the steps we've made towards making this track a reality -- we describe the details of our reusable framework, Ragnar"ok, explain the curation of the new MS MARCO V2.1 collection choice, release the development topics for the track, and standardize the I/O definitions which assist the end user. Next, using Ragnar"ok, we identify and provide key industrial baselines such as OpenAI's GPT-4o or Cohere's Command R+. Further, we introduce a web-based user interface for an interactive arena allowing benchmarking pairwise RAG systems by crowdsourcing. We open-source our Ragnar"ok framework and baselines to achieve a unified standard for future RAG systems.

Key facts. Cited by 31 papers in the map (data through 7 Oct 2026). Placed in RAG pipeline design and evaluation, one of 15 directions in Retrieval-augmented generation. Builds on 46 of its 74 references that are in the map. First posted on arXiv 24 Jun 2024.

Builds on

46 of 74 references.

Closest ancestors · 5

Same direction, newest first

  1. 2024FlashRAG: A Modular Toolkit for Efficient Retrieval-Augmented Generation Research22 May
  2. 2024Question-Based Retrieval using Atomic Units for Enterprise RAG20 May
  3. 2024Generative Information Retrieval Evaluation11 Apr
  4. 2024Corrective Retrieval Augmented Generation29 Jan
  5. 2023Retrieval-Augmented Generation for Large Language Models: A Survey18 Dec

Other references · 41

  1. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  2. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  3. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  4. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  5. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,383
  6. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
  7. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,126
  8. 2023Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflectioncited by 1,800
  9. 2016MS MARCO: A Human Generated MAchine Reading COmprehension Datasetcited by 1,591
  10. 2021Improving language models by retrieving from trillions of tokenscited by 1,278
  11. 2021BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Modelscited by 1,225
  12. 2020Leveraging Passage Retrieval with Generative Models for Open Domain Question Answeringcited by 1,203
29 more
  1. 2024Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preferencecited by 1,151
  2. 2019Generalization through Memorization: Nearest Neighbor Language Modelscited by 788
  3. 2023Towards General Text Embeddings with Multi-stage Contrastive Learningcited by 705
  4. 2022MTEB: Massive Text Embedding Benchmarkcited by 623
  5. 2023Enabling Large Language Models to Generate Text with Citationscited by 584
  6. 2020KILT: a Benchmark for Knowledge Intensive Language Taskscited by 571
  7. 2019ELI5: Long Form Question Answeringcited by 567
  8. 2020Document Ranking with a Pretrained Sequence-to-Sequence Modelcited by 373
  9. 2023Is ChatGPT Good at Search? Investigating Large Language Models as Re-Ranking Agentscited by 356
  10. 2024Benchmarking Retrieval-Augmented Generation for Medicinecited by 326
  11. 2022ASQA: Factoid Questions Meet Long-Form Answerscited by 231
  12. 2019Gradio: Hassle-Free Sharing and Testing of ML Models in the Wildcited by 176
  13. 2023RankZephyr: Effective and Robust Zero-Shot Listwise Reranking is a Breeze!cited by 151
  14. 2023ExpertQA: Expert-Curated Questions and Attributed Answerscited by 96
  15. 2021The Expando-Mono-Duo Design Pattern for Text Ranking with Pretrained Sequence-to-Sequence Modelscited by 81
  16. 2023Zero-Shot Listwise Document Reranking with a Large Language Modelcited by 81
  17. 2023RankVicuna: Zero-Shot Listwise Document Reranking with Open-Source Large Language Modelscited by 69
  18. 2024SPLADE-v3: New baselines for SPLADEcited by 69
  19. 2024Arctic-Embed: Scalable, Efficient, and Accurate Text Embedding Modelscited by 60
  20. 2023HAGRID: A Human-LLM Collaborative Dataset for Generative Information-Seeking with Attributioncited by 55
  21. 2024Financial Report Chunking for Effective Retrieval Augmented Generationcited by 46
  22. 2020AQuaMuSe: Automatically Generating Datasets for Query-Based Multi-Document Summarizationcited by 42
  23. 2024UMBRELA: UMbrela is the (Open-Source Reproduction of the) Bing RELevance Assessorcited by 41
  24. 2023Scaling Down, LiTting Up: Efficient Zero-Shot Listwise Reranking with Seq2seq Encoder-Decoder Modelscited by 26
  25. 2024CLAPNQ: Cohesive Long-form Answers from Passages in Natural Questions for RAG systemscited by 23
  26. 2024On the Evaluation of Machine-Generated Reportscited by 23
  27. 2024Researchy Questions: A Dataset of Multi-Perspective, Decompositional Questions for LLM Web Agentscited by 22
  28. 2024TREC iKAT 2023: The Interactive Knowledge Assistance Track Overviewcited by 15
  29. 2024A Comparison of Methods for Evaluating Generative IRcited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 6; 2025: 13; 2026: 12.

31 papers.

Sort:

By area:

  1. 2025A Systematic Review of Key Retrieval-Augmented Generation (RAG) Systems: Progress, Gaps, and Future Directionscited by 30
  2. 2024Initial Nugget Evaluation Results for the TREC 2024 RAG Track with the AutoNuggetizer Frameworkcited by 28
  3. 2025The Great Nugget Recall: Automating Fact Extraction and RAG Evaluation with Large Language Modelscited by 24
  4. 2024A Large-Scale Study of Relevance Assessments with Large Language Models: An Initial Lookcited by 18
  5. 2025GaRAGe: A Benchmark with Grounding Annotations for RAG Evaluationcited by 18
  6. 2025FreshStack: Building Realistic Benchmarks for Evaluating Retrieval on Technical Documentscited by 15
  7. 2025InfoDeepSeek: Benchmarking Agentic Information Seeking for Retrieval-Augmented Generationcited by 15
  8. 2024MIRAGE-Bench: Automatic Multilingual Benchmark Arena for Retrieval-Augmented Generation Systemscited by 14
  9. 2025Support Evaluation for the TREC 2024 RAG Track: Comparing Human versus LLM Judgescited by 14
  10. 2024Drowning in Documents: Consequences of Scaling Reranker Inferencecited by 8
  11. 2024Inference Scaling for Bridging Retrieval and Augmented Generationcited by 7
  12. 2025RAVine: Reality-Aligned Evaluation for Agentic Searchcited by 6
19 more
  1. 2025AdaGReS:Adaptive Greedy Context Selection via Redundancy-Aware Scoring for Token-Budgeted RAGcited by 6
  2. 2025Evaluating the Robustness of Retrieval-Augmented Generation to Adversarial Evidence in the Health Domaincited by 5
  3. 2024An Early FIRST Reproduction and Improvements to Single-Token Decoding for Fast Listwise Rerankingcited by 4
  4. 2025An Efficient Rubric-based Generative Verifier for Search-Augmented LLMscited by 4
  5. 2026Deep Research, Shallow Evaluation: A Case Study in Meta-Evaluation for Long-Form QA Benchmarkscited by 4
  6. 2026OwlerLite: Scope- and Freshness-Aware Web Retrieval for LLM Assistantscited by 3
  7. 2026Overview of the TREC 2025 Retrieval Augmented Generation (RAG) Trackcited by 3
  8. 2025The Viability of Crowdsourcing for RAG Evaluationcited by 2
  9. 2025A Vision for Geo-Temporal Deep Research Systems: Towards Comprehensive, Transparent, and Reproducible Geo-Temporal Information Synthesiscited by 2
  10. 2026CoverageBench: Evaluating Information Coverage across Tasks and Domainscited by 2
  11. 2026Beyond Relevance: On the Relationship Between Retrieval and RAG Information Coveragecited by 1
  12. 2026Evaluation of Agents under Simulated AI Marketplace Dynamicscited by 1
  13. 2026Can QPP Choose the Right Query Variant? Evaluating Query Variant Selection for RAG Pipelinescited by 1
  14. 2025The SMeL Test: A simple benchmark for media literacy in language modelscited by 0
  15. 2026RMIT-ADM+S at the MMU-RAG NeurIPS 2025 Competitioncited by 0
  16. 2026RAGe: A Retrieval-Augmented Generation Evaluation Frameworkcited by 0
  17. 2026Persona Conditioning as an Assessor-Sensitivity Probe for LLM-Based IR Evaluationcited by 0
  18. 2026Projecting BrowseComp-Plus onto ClimbMix: Toward More Realistic Corpora for Agentic Searchcited by 0
  19. 2026Q2D-Web: A Large-Scale Benchmark for Retrieval in Agentic RAG Systemscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024RAG Foundry: A Framework for Enhancing LLMs for Retrieval Augmented Generation
  2. 2025Retrieval Augmented Generation Evaluation in the Era of Large Language Models: A Comprehensive Survey
  3. 2025Engineering RAG Systems for Real-World Applications: Design, Development, and Evaluation
  4. 2024A Comprehensive Survey of Retrieval-Augmented Generation (RAG): Evolution, Current Landscape and Future Directions
  5. 2024Towards Optimizing a Retrieval Augmented Generation using Large Language Model on Academic Data
  6. 2024A Survey on Retrieval-Augmented Text Generation for Large Language Models
  7. 2024RAGBench: Explainable Benchmark for Retrieval-Augmented Generation Systems
  8. 2024R^3AG: First Workshop on Refined and Reliable Retrieval Augmented Generation