mlsanityData to 7 Oct 2026

How Significant Are the Real Performance Gains? An Unbiased Evaluation Framework for GraphRAG

arXiv 2506.06331 ↗First posted 31 May 2025cs.CL cs.AI cs.IRcited by 2

On the map: Retrieval-augmented generation → Knowledge-graph and structured-knowledge RAG

Abstract (arXiv)

By retrieving contexts from knowledge graphs, graph-based retrieval-augmented generation (GraphRAG) enhances large language models (LLMs) to generate quality answers for user questions. Many GraphRAG methods have been proposed and reported inspiring performance in answer quality. However, we observe that the current answer evaluation framework for GraphRAG has two critical flaws, i.e., unrelated questions and evaluation biases, which may lead to biased or even wrong conclusions on performance. To tackle the two flaws, we propose an unbiased evaluation framework that uses graph-text-grounded question generation to produce questions that are more related to the underlying dataset and an unbiased evaluation procedure to eliminate the biases in LLM-based answer assessment. We apply our unbiased framework to evaluate 3 representative GraphRAG methods and find that their performance gains are much more moderate than reported previously. Although our evaluation framework may still have flaws, it calls for scientific evaluations to lay solid foundations for GraphRAG research.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in Knowledge-graph and structured-knowledge RAG, one of 15 directions in Retrieval-augmented generation. Builds on 41 of its 67 references that are in the map. First posted on arXiv 31 May 2025.

Builds on

41 of 67 references.

Closest ancestors · 8

Same direction, newest first

  1. 2025HyperGraphRAG: Retrieval-Augmented Generation via Hypergraph-Structured Knowledge Representation27 Mar
  2. 2025KET-RAG: A Cost-Efficient Multi-Granular Indexing Framework for Graph-RAG13 Feb
  3. 2025A Survey of Graph Retrieval-Augmented Generation for Customized Large Language Models21 Jan
  4. 2024Retrieval-Augmented Generation with Graphs (GraphRAG)31 Dec
  5. 2024Graph of Records: Boosting Retrieval Augmented Generation for Long-context Summarization with Graphs14 Oct
  6. 2024LightRAG: Simple and Fast Retrieval-Augmented Generation8 Oct
  7. 2024From Local to Global: A Graph RAG Approach to Query-Focused Summarization24 Apr
  8. 2024G-Retriever: Retrieval-Augmented Generation for Textual Graph Understanding and Question Answering12 Feb

Other references · 33

  1. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  2. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,160
  3. 2023Retrieval-Augmented Generation for Large Language Models: A Surveycited by 2,185
  4. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  5. 2023A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questionscited by 1,582
  6. 2023A Survey on Evaluation of Large Language Modelscited by 1,227
  7. 2023Can Large Language Models Be an Alternative to Human Evaluations?cited by 729
  8. 2020Knowledge Graphscited by 619
  9. 2024A Survey on RAG Meeting LLMs: Towards Retrieval-Augmented Large Language Modelscited by 450
  10. 2020How Can We Know When Language Models Know? On the Calibration of Language Models for Question Answeringcited by 439
  11. 2023Large Language Models Are State-of-the-Art Evaluators of Translation Qualitycited by 416
  12. 2023Reasoning on Graphs: Faithful and Interpretable Large Language Model Reasoningcited by 326
21 more
  1. 2023Benchmarking Large Language Models in Retrieval-Augmented Generationcited by 307
  2. 2024A Comprehensive Survey of Hallucination Mitigation Techniques in Large Language Modelscited by 266
  3. 2022Data Contamination: From Memorization to Exploitationcited by 234
  4. 2025From RAG to Memory: Non-Parametric Continual Learning for Large Language Modelscited by 223
  5. 2023Survey on Factuality in Large Language Models: Knowledge, Retrieval and Domain-Specificitycited by 204
  6. 2023GPT-NER: Named Entity Recognition via Large Language Modelscited by 158
  7. 2024Evaluation of Retrieval-Augmented Generation: A Surveycited by 146
  8. 2022Improving the Domain Adaptation of Retrieval Augmented Generation (RAG) Models for Open Domain Question Answeringcited by 126
  9. 2024Graph Chain-of-Thought: Augmenting Large Language Models by Reasoning on Graphscited by 116
  10. 2023LLM-Eval: Unified Multi-Dimensional Automatic Evaluation for Open-Domain Conversations with Large Language Modelscited by 100
  11. 2024Evaluating Retrieval Quality in Retrieval-Augmented Generationcited by 90
  12. 2025HalluLens: LLM Hallucination Benchmarkcited by 88
  13. 2024CRUD-RAG: A Comprehensive Chinese Benchmark for Retrieval-Augmented Generation of Large Language Modelscited by 50
  14. 2024RAG-QA Arena: Evaluating Domain Robustness for Long-form Retrieval Augmented Question Answeringcited by 43
  15. 2023Evaluating Open-QA Evaluationcited by 35
  16. 2025The Law of Knowledge Overshadowing: Towards Understanding, Predicting, and Preventing LLM Hallucinationcited by 22
  17. 2024QGEval: Benchmarking Multi-dimensional Evaluation for Question Generationcited by 17
  18. 2020Exploring Question-Specific Rewards for Generating Deep Questionscited by 9
  19. 2025SAGE: A Framework of Precise Retrieval for RAGcited by 8
  20. 2024Evaluating Quality of Answers for Retrieval-Augmented Generation: A Strong LLM Is All You Needcited by 7
  21. 2024Are Expert-Level Language Models Expert-Level Annotators?cited by 3

Built on by

Cited by year

Citing papers per year: 2025: 2.

2 papers.

  1. 2025Scalable and Reliable Evaluation of AI Knowledge Retrieval Systems: RIKER and the Coherent Simulated Universecited by 3
  2. 2025Aligning LLMs for the Classroom with Knowledge-Based Retrieval -- A Comparative RAG Studycited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2025GraphRAG-Bench: Challenging Domain-Specific Reasoning for Evaluating Graph Retrieval-Augmented Generation
  2. 2025RAG vs. GraphRAG: A Systematic Evaluation and Key Insights
  3. 2025Beyond Static Retrieval: Opportunities and Pitfalls of Iterative Retrieval in GraphRAG
  4. 2025When to use Graphs in RAG: A Comprehensive Analysis for Graph Retrieval-Augmented Generation
  5. 2025NodeRAG: Structuring Graph-based RAG with Heterogeneous Nodes
  6. 2025Comparing RAG and GraphRAG for Page-Level Retrieval Question Answering on a Math Textbook
  7. 2026The Answer Path and the Grounding Instruction in LLM Question Answering over Knowledge Graphs
  8. 2025GraphRAFT: Retrieval Augmented Fine-Tuning for Knowledge Graphs on Graph Databases