mlsanityData to 14 Oct 2026

MIRAGE-Bench: Automatic Multilingual Benchmark Arena for Retrieval-Augmented Generation Systems

arXiv 2410.13716 ↗First posted 17 Oct 2024cs.CL cs.AIcited by 14

On the map: Retrieval-augmented generation → RAG pipeline design and evaluation

Abstract (arXiv)

Traditional retrieval-augmented generation (RAG) benchmarks evaluate systems using heuristic-based metrics, but these require human preferences as the ground truth for reference. In contrast, arena-based benchmarks, where systems compete against each other, require an expensive large language model (LLM) as a judge for a reliable evaluation. We present a simple efficient technique to combine the best of both worlds. The idea is to train a surrogate judge using heuristic metrics as input, to output the LLM as a judge prediction. In our work, we develop MIRAGE-Bench, a synthetic arena-based RAG benchmark for 18 diverse languages on Wikipedia focused on multilingual answer generation evaluation. It extensively couples both heuristic features and LLM as a judge for evaluation. We benchmark 19 multilingual LLMs, and observe a high correlation (Kendall Tau (τ) = 0.909) using our surrogate judge and between GPT-4o as a teacher using the Bradley-Terry framework. Our results show proprietary and large open-source LLMs currently dominate on MIRAGE-Bench. Our code and datasets are made publicly available here: https://github.com/vectara/mirage-bench.

Key facts. Cited by 14 papers in the map (data through 14 Oct 2026). Placed in RAG pipeline design and evaluation, one of 15 directions in Retrieval-augmented generation. Builds on 50 of its 81 references that are in the map. First posted on arXiv 17 Oct 2024.

Builds on

50 of 81 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024Ragnar"ok: A Reusable RAG Framework and Baselines for TREC 2024 Retrieval-Augmented Generation Track24 Jun
  2. 2024CRAG -- Comprehensive RAG Benchmark7 Jun
  3. 2023Retrieval-Augmented Generation for Large Language Models: A Survey18 Dec

Other references · 47

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  3. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  4. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  5. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,935
  6. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,824
  7. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,754
  8. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,598
  9. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,276
  10. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,141
  11. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,917
  12. 2016MS MARCO: A Human Generated MAchine Reading COmprehension Datasetcited by 1,605
35 more
  1. 2021Improving language models by retrieving from trillions of tokenscited by 1,293
  2. 2021DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharingcited by 1,254
  3. 2021BEIR: A Heterogenous Benchmark for Zero-shot Evaluation of Information Retrieval Modelscited by 1,252
  4. 2020Leveraging Passage Retrieval with Generative Models for Open Domain Question Answeringcited by 1,210
  5. 2024Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preferencecited by 1,184
  6. 2020ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERTcited by 1,178
  7. 2023Large Language Models are not Fair Evaluatorscited by 1,085
  8. 2024M3-Embedding: Multi-Linguality, Multi-Functionality, Multi-Granularity Text Embeddings Through Self-Knowledge Distillationcited by 1,073
  9. 2019Generalization through Memorization: Nearest Neighbor Language Modelscited by 800
  10. 2024LLM Evaluators Recognize and Favor Their Own Generationscited by 686
  11. 2023Enabling Large Language Models to Generate Text with Citationscited by 595
  12. 2019ELI5: Long Form Question Answeringcited by 575
  13. 2020TyDi QA: A Benchmark for Information-Seeking Question Answering in Typologically Diverse Languagescited by 488
  14. 2021MAUVE: Measuring the Gap Between Neural Text and Human Text using Divergence Frontierscited by 470
  15. 2021Dynabench: Rethinking Benchmarking in NLPcited by 459
  16. 2024Humans or LLMs as the Judge? A Study on Judgement Biasescited by 323
  17. 2023FreshLLMs: Refreshing Large Language Models with Search Engine Augmentationcited by 318
  18. 2023Benchmarking Large Language Models in Retrieval-Augmented Generationcited by 312
  19. 2023RAGTruth: A Hallucination Corpus for Developing Trustworthy Retrieval-Augmented Language Modelscited by 283
  20. 2024RAFT: Adapting Language Model to Domain Specific RAGcited by 262
  21. 2024Judging the Judges: Evaluating Alignment and Vulnerabilities in LLMs-as-Judgescited by 205
  22. 2024Aya 23: Open Weight Releases to Further Multilingual Progresscited by 123
  23. 2024Pride and Prejudice: LLM Amplifies Self-Bias in Self-Refinementcited by 114
  24. 2019Neural Machine Translation: A Review and Surveycited by 100
  25. 2024ChatQA: Surpassing GPT-4 on Conversational QA and RAGcited by 90
  26. 2024MixEval: Deriving Wisdom of the Crowd from LLM Benchmark Mixturescited by 74
  27. 2023HAGRID: A Human-LLM Collaborative Dataset for Generative Information-Seeking with Attributioncited by 56
  28. 2023SEAHORSE: A Multilingual, Multifaceted Dataset for Summarization Evaluationcited by 42
  29. 2024CLAPNQ: Cohesive Long-form Answers from Passages in Natural Questions for RAG systemscited by 25
  30. 2023Leveraging LLMs for Synthesizing Training Data Across Many Languages in Multilingual Dense Retrievalcited by 24
  31. 2024On the Evaluation of Machine-Generated Reportscited by 23
  32. 2023"Knowing When You Don't Know": A Multilingual Relevance Assessment Dataset for Robust Retrieval-Augmented Generationcited by 13
  33. 2024A Comparison of Methods for Evaluating Generative IRcited by 11
  34. 2024Evaluating RAG-Fusion with RAGElo: an Automated Elo-based Frameworkcited by 11
  35. 2022Moving Beyond Downstream Task Accuracy for Information Retrieval Benchmarkingcited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 10; 2026: 4.

14 papers.

Sort:

By area:

  1. 2025The Great Nugget Recall: Automating Fact Extraction and RAG Evaluation with Large Language Modelscited by 24
  2. 2025Multilingual Retrieval-Augmented Generation for Knowledge-Intensive Taskcited by 12
  3. 2025On the Consistency of Multilingual Context Utilization in Retrieval-Augmented Generationcited by 10
  4. 2025XRAG: Cross-lingual Retrieval-Augmented Generationcited by 9
  5. 2025Memory in Large Language Models: Mechanisms, Evaluation and Evolutioncited by 8
  6. 2025MEMERAG: A Multilingual End-to-End Meta-Evaluation Benchmark for Retrieval Augmented Generationcited by 7
  7. 2025Can LLMs Be Trusted for Evaluating RAG Systems? A Survey of Methods and Datasetscited by 7
  8. 2025LLMs cannot spot math errors, even when allowed to peek into the solutioncited by 7
  9. 2025Phonetic Perturbations Reveal Tokenizer-Rooted Safety Gaps in LLMscited by 2
  10. 2026Typhoon-S: Minimal Open Post-Training for Sovereign Large Language Modelscited by 2
  11. 2025M4-RAG: A Massive-Scale Multilingual Multi-Cultural Multimodal RAGcited by 1
  12. 2026Challenges and Recommendations for LLM-as-a-Judge in Multilingual Settings and for Low-Resource Languagescited by 1
2 more
  1. 2026PolyWorkBench: Benchmarking LLM Agents for Cross-Lingual Long-Horizon Workflowscited by 1
  2. 2026CORAL: Adaptive Retrieval Loop for Culturally-Aligned Multilingual RAGcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025MIRAGE: A Metric-Intensive Benchmark for Retrieval-Augmented Generation Evaluation
  2. 2025PRGB Benchmark: A Robust Placeholder-Assisted Algorithm for Benchmarking Retrieval-Augmented Generation
  3. 2026LLM-as-a-Judge for Low-Resource Languages: Adapting Ragas and Comparative Ranking for Romanian
  4. 2024BERGEN: A Benchmarking Library for Retrieval-Augmented Generation
  5. 2026LIT-RAGBench: Benchmarking Generator Capabilities of Large Language Models in Retrieval-Augmented Generation
  6. 2024GroUSE: A Benchmark to Evaluate Evaluators in Grounded Question Answering
  7. 2025Judge as A Judge: Improving the Evaluation of Retrieval-Augmented Generation through the Judge-Consistency of Large Language Models
  8. 2025MTRAG: A Multi-Turn Conversational Benchmark for Evaluating Retrieval-Augmented Generation Systems