mlsanityData to 14 Oct 2026

A Pilot Empirical Study on When and How to Use Knowledge Graphs as Retrieval Augmented Generation

arXiv 2502.20854 ↗First posted 28 Feb 2025cs.AI cs.CLcited by 0

On the map: Retrieval-augmented generation → Knowledge-graph and structured-knowledge RAG

Abstract (arXiv)

The integration of Knowledge Graphs (KGs) into the Retrieval Augmented Generation (RAG) framework has attracted significant interest, with early studies showing promise in mitigating hallucinations and improving model accuracy. However, a systematic understanding and comparative analysis of the rapidly emerging KG-RAG methods are still lacking. This paper seeks to lay the foundation for systematically answering the question of when and how to use KG-RAG by analyzing their performance in various application scenarios associated with different technical configurations. After outlining the mind map using KG-RAG framework and summarizing its popular pipeline, we conduct a pilot empirical study of KG-RAG works to reimplement and evaluate 6 KG-RAG methods across 9 datasets in diverse domains and scenarios, analyzing the impact of 9 KG-RAG configurations in combination with 17 LLMs, and combining Metacognition with KG-RAG as a pilot attempt. Our results underscore the critical role of appropriate application conditions and optimal configurations of KG-RAG components.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Knowledge-graph and structured-knowledge RAG, one of 15 directions in Retrieval-augmented generation. Builds on 51 of its 67 references that are in the map. First posted on arXiv 28 Feb 2025.

Builds on

51 of 67 references.

Closest ancestors · 4

Same direction, newest first

  1. 2025A Survey of Graph Retrieval-Augmented Generation for Customized Large Language Models21 Jan
  2. 2024Simple Is Effective: The Roles of Graphs and Large Language Models in Knowledge-Graph-Based Retrieval-Augmented Generation28 Oct
  3. 2024Graph Retrieval-Augmented Generation: A Survey15 Aug
  4. 2024From Local to Global: A Graph RAG Approach to Query-Focused Summarization24 Apr

Other references · 47

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2024The Llama 3 Herd of Modelscited by 14,715
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  5. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  6. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  7. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,512
  8. 2024Qwen2.5 Technical Reportcited by 4,942
  9. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,455
  10. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,211
  11. 2022Emergent Abilities of Large Language Modelscited by 2,512
  12. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,276
35 more
  1. 2020REALM: Retrieval-Augmented Language Model Pre-Trainingcited by 2,141
  2. 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,998
  3. 2024Mixtral of Expertscited by 1,823
  4. 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,822
  5. 2024DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Modelcited by 1,135
  6. 2022When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memoriescited by 1,128
  7. 2024ChatGLM: A Family of Large Language Models from GLM-130B to GLM-4 All Toolscited by 1,121
  8. 2024Yi: Open Foundation Models by 01.AIcited by 719
  9. 2023Unifying Large Language Models and Knowledge Graphs: A Roadmapcited by 668
  10. 2024A Survey on RAG Meeting LLMs: Towards Retrieval-Augmented Large Language Modelscited by 452
  11. 2024A Systematic Survey of Prompt Engineering in Large Language Models: Techniques and Applicationscited by 431
  12. 2018The Web as a Knowledge-base for Answering Complex Questionscited by 410
  13. 2023ChatDoctor: A Medical Chat Model Fine-Tuned on a Large Language Model Meta-AI (LLaMA) Using Medical Domain Knowledgecited by 325
  14. 2024A Comprehensive Survey of Hallucination Mitigation Techniques in Large Language Modelscited by 269
  15. 2023Think-on-Graph: Deep and Responsible Reasoning of Large Language Model on Knowledge Graphcited by 238
  16. 2019JEC-QA: A Legal-Domain Question Answering Datasetcited by 116
  17. 2023CMB: A Comprehensive Medical Benchmark in Chinesecited by 115
  18. 2022Re2G: Retrieve, Rerank, Generatecited by 102
  19. 2024A Comprehensive Survey of Hallucination in Large Language, Image, Video and Audio Foundation Modelscited by 100
  20. 2024CRAG -- Comprehensive RAG Benchmarkcited by 97
  21. 2023MindMap: Knowledge Graph Prompting Sparks Graph of Thoughts in Large Language Modelscited by 73
  22. 2024StructRAG: Boosting Knowledge Intensive Reasoning of LLMs via Inference-time Hybrid Information Structurizationcited by 71
  23. 2023Mitigating Large Language Model Hallucinations via Autonomous Knowledge Graph-based Retrofittingcited by 70
  24. 2024Retrieval Augmented Generation (RAG) and Beyond: A Comprehensive Survey on How to Make your LLMs use External Data More Wiselycited by 66
  25. 2023Metacognitive Prompting Improves Understanding in Large Language Modelscited by 58
  26. 2023LLatrieval: LLM-Verified Retrieval for Verifiable Generationcited by 52
  27. 2023KG-GPT: A General Framework for Reasoning on Knowledge Graphs Using Large Language Modelscited by 32
  28. 2023Biomedical knowledge graph-optimized prompt generation for large language modelscited by 31
  29. 2024Metacognitive Retrieval-Augmented Large Language Modelscited by 26
  30. 2023HyKGE: A Hypothesis Knowledge Graph Enhanced Framework for Accurate and Reliable Medical LLMs Responsescited by 25
  31. 2024Can Knowledge Graphs Make Large Language Models More Trustworthy? An Empirical Study Over Open-ended Question Answeringcited by 24
  32. 2025Enhancing Retrieval-Augmented Generation: A Study of Best Practicescited by 23
  33. 2024Similarity is Not All You Need: Endowing Retrieval Augmented Generation with Multi Layered Thoughtscited by 16
  34. 2023ExplainCPE: A Free-text Explanation Benchmark of Chinese Pharmacist Examinationcited by 10
  35. 2024Reasoning on Efficient Knowledge Paths:Knowledge Graph Guides Large Language Model for Domain Question Answeringcited by 9

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Knowledge Graph-Guided Retrieval Augmented Generation
  2. 2025Towards Self-cognitive Exploration: Metacognitive Knowledge Graph Retrieval Augmented Generation
  3. 2025Evaluating Knowledge Graph Based Retrieval Augmented Generation Methods under Knowledge Incompleteness
  4. 2025Graph-based Approaches and Functionalities in Retrieval-Augmented Generation: A Comprehensive Survey
  5. 2025Retrieval-Augmented Generation with Hierarchical Knowledge
  6. 2024Empowering Large Language Models to Set up a Knowledge Retrieval Indexer via Self-Learning
  7. 2025A Survey on Knowledge-Oriented Retrieval-Augmented Generation
  8. 2026EvoRAG: Making Knowledge Graph-based RAG Automatically Evolve through Feedback-driven Backpropagation