mlsanityData to 14 Oct 2026

A Knowledge Graph and a Tripartite Evaluation Framework Make Retrieval-Augmented Generation Scalable and Transparent

arXiv 2509.19209 ↗First posted 23 Sep 2025cs.IRcited by 1

On the map: Retrieval-augmented generation → RAG pipeline design and evaluation

Abstract (arXiv)

Large Language Models (LLMs) have significantly enhanced conversational Artificial Intelligence(AI) chatbots; however, domain-specific accuracy and the avoidance of factual inconsistencies remain pressing challenges, particularly for large datasets. Designing an effective chatbot with appropriate methods and evaluating its effectiveness is among the challenges in this domain. This study presents a Retrieval Augmented Generation (RAG) chatbot that harnesses a knowledge graph and vector search retrieval to deliver precise, context-rich responses in an exemplary use case from over high-volume engineering project-related emails, thereby minimising the need for document chunking. A central innovation of this work is the introduction of RAG Evaluation (RAG-Eval), a novel chain-of-thought LLM-based tripartite evaluation framework specifically developed to assess RAG applications. This framework operates in parallel with the chatbot, jointly assessing the user's query, the retrieved document, and the generated response, enabling a holistic evaluation across multiple quality metrics like query relevance, factual accuracy, coverage, coherence and fluency. The resulting scoring system is provided directly to users as a confidence score (1 to 100%), enabling quick identification of possible misaligned or incomplete answers. This proposed approach promotes transparency and rapid verification by incorporating metadata email IDs, timestamps into responses. Experimental comparisons against BERTScore and G-EVAL for summarisation evaluation tasks confirm its effectiveness, and empirical analysis also shows RAG-Eval reliably detects factual gaps and query mismatches, thereby fostering trust in high demand, data centric environments. These findings highlight a scalable path for developing accurate, user-verifiable chatbots that bridge the gap between high-level conversational fluency and factual accuracy.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in RAG pipeline design and evaluation, one of 15 directions in Retrieval-augmented generation. Builds on 24 of its 48 references that are in the map. First posted on arXiv 23 Sep 2025.

Builds on

24 of 48 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024FACTS About Building Retrieval Augmented Generation-based Chatbots10 Jul
  2. 2024Improving Retrieval for RAG based Question Answering Models on Financial Documents23 Mar

Other references · 22

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  3. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  4. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,695
  5. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,512
  6. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,276
  7. 2024Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameterscited by 1,986
  8. 2023A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questionscited by 1,601
  9. 2024Retrieval-Augmented Generation with Knowledge Graphs for Customer Service Question Answeringcited by 96
  10. 2022The Text Anonymization Benchmark (TAB): A Dedicated Corpus and Evaluation Framework for Text Anonymizationcited by 95
  11. 2024HybridRAG: Integrating Knowledge Graphs and Vector Retrieval Augmented Generation for Efficient Information Extractioncited by 93
  12. 2023ReST meets ReAct: Self-Improvement for Multi-Step Reasoning LLM Agentcited by 81
10 more
  1. 2024Fine-tuning Large Language Models for Domain-specific Machine Translationcited by 43
  2. 2024Memory Is All You Need: An Overview of Compute-in-Memory Architectures for Accelerating Large Language Model Inferencecited by 15
  3. 2024Tabular Embedding Model (TEM): Finetuning Embedding Models For Tabular RAG Applicationscited by 8
  4. 2024From RAGs to riches: Utilizing large language models to write documents for clinical trialscited by 6
  5. 2024Dial-insight: Fine-tuning Large Language Models with High-Quality Domain-Specific Data Preventing Capability Collapsecited by 4
  6. 2024Retrieval Augmented Generation Integrated Large Language Models in Smart Contract Vulnerability Detectioncited by 4
  7. 2025PythonPal: Enhancing Online Programming Education through Chatbot-Driven Personalized Feedbackcited by 2
  8. 2020Practical Hash-based Anonymity for MAC Addressescited by 1
  9. 2025Sample-Efficient Language Model for Hinglish Conversational AIcited by 1
  10. 2025Privacy-Preserving Anonymization of System and Network Event Logs Using Salt-Based Hashing and Temporal Noisecited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026TRIAGE: Trustworthy Retrieval Instrumentation And Graph Evaluationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025HybridRAG: A Practical LLM-based ChatBot Framework based on Pre-Generated Q&A over Raw Unstructured Documents
  2. 2026Enhancing Scientific Literature Chatbots with Retrieval-Augmented Generation: A Performance Evaluation of Vector and Graph-Based Systems
  3. 2024Evaluating the Retrieval Component in LLM-Based Question Answering Systems
  4. 2025From Documents to Dialogue: Building KG-RAG Enhanced AI Assistants
  5. 2024A Fine-tuning Enhanced RAG System with Quantized Influence Measure as AI Judge
  6. 2024RAG-Fusion: a New Take on Retrieval-Augmented Generation
  7. 2025Personalizing Large Language Models using Retrieval Augmented Generation and Knowledge Graph
  8. 2024RAGBench: Explainable Benchmark for Retrieval-Augmented Generation Systems