mlsanityData to 7 Oct 2026

BERGEN: A Benchmarking Library for Retrieval-Augmented Generation

arXiv 2407.01102 ↗First posted 1 Jul 2024cs.CL cs.IRcited by 31

On the map: Retrieval-augmented generation → RAG pipeline design and evaluation

Abstract (arXiv)

Retrieval-Augmented Generation allows to enhance Large Language Models with external knowledge. In response to the recent popularity of generative LLMs, many RAG approaches have been proposed, which involve an intricate number of different configurations such as evaluation datasets, collections, metrics, retrievers, and LLMs. Inconsistent benchmarking poses a major challenge in comparing approaches and understanding the impact of each component in the pipeline. In this work, we study best practices that lay the groundwork for a systematic evaluation of RAG and present BERGEN, an end-to-end library for reproducible research standardizing RAG experiments. In an extensive study focusing on QA, we benchmark different state-of-the-art retrievers, rerankers, and LLMs. Additionally, we analyze existing RAG metrics and datasets. Our open-source library BERGEN is available under urlhttps://github.com/naver/bergen.

Key facts. Cited by 31 papers in the map (data through 7 Oct 2026). Placed in RAG pipeline design and evaluation, one of 15 directions in Retrieval-augmented generation. Builds on 59 of its 80 references that are in the map. First posted on arXiv 1 Jul 2024.

Builds on

59 of 80 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024FlashRAG: A Modular Toolkit for Efficient Retrieval-Augmented Generation Research22 May
  2. 2024The Power of Noise: Redefining Retrieval for RAG Systems26 Jan
  3. 2023ARES: An Automated Evaluation Framework for Retrieval-Augmented Generation Systems16 Nov

Other references · 56

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  5. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  6. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  7. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,072
  8. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,779
  9. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  10. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,686
  11. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  12. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
38 more
  1. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,895
  2. 2023Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflectioncited by 1,800
  3. 2016MS MARCO: A Human Generated MAchine Reading COmprehension Datasetcited by 1,591
  4. 2020MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformerscited by 1,414
  5. 2021BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Modelscited by 1,225
  6. 2022Text Embeddings by Weakly-Supervised Contrastive Pre-trainingcited by 1,216
  7. 2020Leveraging Passage Retrieval with Generative Models for Open Domain Question Answeringcited by 1,203
  8. 2021Unsupervised Dense Information Retrieval with Contrastive Learningcited by 1,203
  9. 2017Mixed Precision Trainingcited by 1,135
  10. 2022When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memoriescited by 1,103
  11. 2024M3-Embedding: Multi-Linguality, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillationcited by 1,050
  12. 2022Atlas: Few-shot Learning with Retrieval Augmented Language Modelscited by 940
  13. 2016Bag of Tricks for Efficient Text Classificationcited by 869
  14. 2023DSPy: Compiling Declarative Language Model Calls into Self-Improving Pipelinescited by 803
  15. 2017Crowdsourcing Multiple Choice Science Questionscited by 742
  16. 2023Towards General Text Embeddings with Multi-stage Contrastive Learningcited by 705
  17. 2023In-Context Retrieval-Augmented Language Modelscited by 694
  18. 2019Passage Re-ranking with BERTcited by 629
  19. 2023Instruction Tuning for Large Language Models: A Surveycited by 617
  20. 2024TinyLlama: An Open-Source Small Language Modelcited by 604
  21. 2020KILT: a Benchmark for Knowledge Intensive Language Taskscited by 571
  22. 2019ELI5: Long Form Question Answeringcited by 567
  23. 2018Wizard of Wikipedia: Knowledge-Powered Conversational agentscited by 530
  24. 2020TyDi QA: A Benchmark for Information-Seeking Question Answering in Typologically Diverse Languagescited by 486
  25. 2016FastText.zip: Compressing text classification modelscited by 331
  26. 2024Nomic Embed: Training a Reproducible Long Context Text Embeddercited by 255
  27. 2021Efficiently Teaching an Effective Dense Retriever with Balanced Topic Aware Samplingcited by 238
  28. 2022ASQA: Factoid Questions Meet Long-Form Answerscited by 231
  29. 2021Unsupervised Corpus Aware Language Model Pre-training for Dense Passage Retrievalcited by 203
  30. 2023RA-DIT: Retrieval-Augmented Dual Instruction Tuningcited by 194
  31. 2023SOLAR 10.7B: Scaling Large Language Models with Simple yet Effective Depth Up-Scalingcited by 172
  32. 2020MKQA: A Linguistically Diverse Benchmark for Multilingual Open Domain Question Answeringcited by 151
  33. 2024Fine-grained Hallucination Detection and Editing for Language Modelscited by 146
  34. 2020XOR QA: Cross-lingual Open-Retrieval Question Answeringcited by 121
  35. 2022RetroMAE: Pre-Training Retrieval-oriented Language Models Via Masked Auto-Encodercited by 118
  36. 2023How to Train Your DRAGON: Diverse Augmentation Towards Generalizable Dense Retrievalcited by 116
  37. 2022From Distillation to Hard Negative Sampling: Making Sparse Neural IR Models More Effectivecited by 98
  38. 2023AnglE-optimized Text Embeddingscited by 98

50 most cited of 56 All 56 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 3; 2025: 18; 2026: 10.

31 papers.

Sort:

By area:

  1. 2024Sufficient Context: A New Lens on Retrieval Augmented Generation Systemscited by 69
  2. 2024Context Embeddings for Efficient Answer Generation in RAGcited by 26
  3. 2025PISCO: Pretty Simple Compression for Retrieval-Augmented Generationcited by 20
  4. 2025PhantomWiki: On-Demand Datasets for Reasoning and Retrieval Evaluationcited by 14
  5. 2025Position: Text Embeddings Should Capture Implicit Semantics, Not Just Surface Meaningcited by 10
  6. 2025Investigating the Robustness of Retrieval-Augmented Generation at the Query Levelcited by 10
  7. 2025Controlled Retrieval-augmented Context Evaluation for Long-form RAGcited by 9
  8. 2026Diffusion-Pretrained Dense and Contextual Embeddingscited by 9
  9. 2025Memory in Large Language Models: Mechanisms, Evaluation and Evolutioncited by 8
  10. 2025Can LLMs Be Trusted for Evaluating RAG Systems? A Survey of Methods and Datasetscited by 7
  11. 2024RAG Foundry: A Framework for Enhancing LLMs for Retrieval Augmented Generationcited by 5
  12. 2025LRAGE: Legal Retrieval Augmented Generation Evaluation Toolcited by 5
19 more
  1. 2025mmRAG: A Modular Benchmark for Retrieval-Augmented Generation over Text, Tables, and Knowledge Graphscited by 5
  2. 2025NAVER LABS Europe Submission to the Instruction-following Trackcited by 5
  3. 2026SAGEO Arena: A Realistic Environment for Evaluating Search-Augmented Generative Engine Optimizationcited by 5
  4. 2025OSCAR: Online Soft Compression And Rerankingcited by 4
  5. 2025Improve Rule Retrieval and Reasoning with Self-Induction and Relevance ReEstimatecited by 3
  6. 2025Quality-Aware Translation Tagging in Multilingual RAG systemcited by 3
  7. 2026SpeechMapper: Speech-to-text Embedding Projector for LLMscited by 3
  8. 2025DiffLoRA: Differential Low-Rank Adapters for Large Language Modelscited by 1
  9. 2025Beyond Benchmarks: LLM Evaluation with an Anthropomorphic and Lifecycle-oriented Roadmapcited by 1
  10. 2025Incentive-Aligned Multi-Source LLM Summariescited by 1
  11. 2025M4-RAG: A Massive-Scale Multilingual Multi-Cultural Multimodal RAGcited by 1
  12. 2026Fixed RAG Compression Collapses Measured Reader Scalingcited by 1
  13. 2026Towards Query-Agnostic RAG Evaluation via Query Coverage and Claim Verifiabilitycited by 1
  14. 2025References Indeed Matter? Reference-Free Preference Optimization for Conversational Query Reformulationcited by 0
  15. 2026BEATS: Bootstrapping E-commerce Attribute Taxonomies for Search through Iterative Human-AI Collaborationcited by 0
  16. 2026Compression Beyond the Uncompressed: A Two-Stage Training Recipe for Soft Context Compression in RAGcited by 0
  17. 2026RAGMark: A Comprehensive Framework for Benchmarking Retrieval-Augmented Generation Systemscited by 0
  18. 2026Lost in Conversation or Lost in Translation? Diagnosing Multi-Turn Degradation in RAGcited by 0
  19. 2026Mapping the RAG Landscape: A Four Axis Taxonomy of Efficiency, Defense, Interactivity, and Reasoningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Evaluation of Retrieval-Augmented Generation: A Survey
  2. 2024XRAG: eXamining the Core -- Benchmarking Foundational Components in Advanced Retrieval-Augmented Generation
  3. 2025When Retrieval Succeeds and Fails: Rethinking Retrieval-Augmented Generation for LLMs
  4. 2025Retrieval Augmented Generation Evaluation in the Era of Large Language Models: A Comprehensive Survey
  5. 2023Retrieval-Augmented Generation for Large Language Models: A Survey
  6. 2025PRGB Benchmark: A Robust Placeholder-Assisted Algorithm for Benchmarking Retrieval-Augmented Generation
  7. 2024CoFE-RAG: A Comprehensive Full-chain Evaluation Framework for Retrieval-Augmented Generation with Enhanced Data Diversity
  8. 2026Revisiting RAG Retrievers: An Information Theoretic Benchmark