mlsanityData to 14 Oct 2026

Can we Evaluate RAGs with Synthetic Data?

arXiv 2508.11758 ↗First posted 15 Aug 2025cs.CL cs.AIcited by 2

On the map: Synthetic data and self-improvement for LLMs → LLM synthetic data generation

Abstract (arXiv)

We investigate whether synthetic question-answer (QA) data generated by large language models (LLMs) can serve as an effective proxy for human-labeled benchmarks when the latter is unavailable. We assess the reliability of synthetic benchmarks across two experiments: one varying retriever parameters while keeping the generator fixed, and another varying the generator with fixed retriever parameters. Across four datasets, of which two open-domain and two proprietary, we find that synthetic benchmarks reliably rank the RAGs varying in terms of retriever configuration, aligning well with human-labeled benchmark baselines. However, they do not consistently produce reliable RAG rankings when comparing generator architectures. The breakdown possibly arises from a combination of task mismatch between the synthetic and human benchmarks, and stylistic bias favoring certain generators.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in LLM synthetic data generation, one of 13 directions in Synthetic data and self-improvement for LLMs. Builds on 26 of its 30 references that are in the map. First posted on arXiv 15 Aug 2025.

Builds on

26 of 30 references.

References · 26

  1. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  2. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,148
  3. 2023Retrieval-Augmented Generation for Large Language Models: A Surveycited by 2,212
  4. 2022Survey of Hallucination in Natural Language Generationcited by 2,075
  5. 2016MS MARCO: A Human Generated MAchine Reading COmprehension Datasetcited by 1,605
  6. 2023A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questionscited by 1,601
  7. 2020On Faithfulness and Factuality in Abstractive Summarizationcited by 900
  8. 2020KILT: a Benchmark for Knowledge Intensive Language Taskscited by 573
  9. 2023Ragas: Automated Evaluation of Retrieval Augmented Generationcited by 418
  10. 2023A Survey of Hallucination in Large Foundation Modelscited by 368
  11. 2024Benchmarking Retrieval-Augmented Generation for Medicinecited by 331
  12. 2023Benchmarking Large Language Models in Retrieval-Augmented Generationcited by 312
14 more
  1. 2023ARES: An Automated Evaluation Framework for Retrieval-Augmented Generation Systemscited by 265
  2. 2024MultiHop-RAG: Benchmarking Retrieval-Augmented Generation for Multi-Hop Queriescited by 257
  3. 2022ASQA: Factoid Questions Meet Long-Form Answerscited by 234
  4. 2024Evaluation of Retrieval-Augmented Generation: A Surveycited by 146
  5. 2019Synthetic QA Corpora Generation with Roundtrip Consistencycited by 143
  6. 2024Seven Failure Points When Engineering a Retrieval Augmented Generation Systemcited by 139
  7. 2024Enhancing LLM Factual Accuracy with RAG to Counter Hallucinations: A Case Study on Domain-Specific Queries in Private Knowledge-Basescited by 77
  8. 2023RECALL: A Benchmark for LLMs Robustness against External Counterfactual Knowledgecited by 51
  9. 2023ReEval: Automatic Hallucination Evaluation for Retrieval-Augmented Large Language Models via Transferable Adversarial Attackscited by 28
  10. 2024Let LLMs Take on the Latest Challenges! A Chinese Dynamic Question Answering Benchmarkcited by 22
  11. 2024DomainRAG: A Chinese Benchmark for Evaluating Domain-specific Retrieval-Augmented Generationcited by 16
  12. 2024Evaluating the Efficacy of Open-Source LLMs in Enterprise-Specific RAG Systems: A Comparative Study of Performance and Scalabilitycited by 15
  13. 2024Mindful-RAG: A Study of Points of Failure in Retrieval Augmented Generationcited by 12
  14. 2025Can LLMs Be Trusted for Evaluating RAG Systems? A Survey of Methods and Datasetscited by 7

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026The Coverage Illusion: From Pre-retrieval Routing Failure to Post-retrieval Cascades in a Production RAG Systemcited by 0
  2. 2026Mapping Similarity Spaces across Embedding Models with Synthetic Query Probingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Beyond Benchmark Scores: How Synthetic and Authentic Query Distributions Diverge in RAG Evaluation
  2. 2024Unveiling the Flaws: Exploring Imperfections in Synthetic Data and Mitigation Strategies for Large Language Models
  3. 2025MTRAG: A Multi-Turn Conversational Benchmark for Evaluating Retrieval-Augmented Generation Systems
  4. 2024Efficacy of Synthetic Data as a Benchmark
  5. 2026How Fine-Grained Should a RAG Benchmark Be? A Hierarchical Framework for Synthetic Question Generation
  6. 2024Balancing Cost and Effectiveness of Synthetic Data Generation Strategies for LLMs
  7. 2025Accelerating Unbiased LLM Evaluation via Synthetic Feedback
  8. 2025Generating Diverse Q&A Benchmarks for RAG Evaluation with DataMorgana