# Mitigating Hallucinations in Large Language Models via Self-Refinement-Enhanced Knowledge Retrieval

arXiv: https://arxiv.org/abs/2405.06545 · HTML: https://mlsanity.com/paper/2405.06545/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 40 references that are in the map.
- First posted on arXiv 10 May 2024.

## Builds on (most cited first, 26 of 26)

- [LLaMA: Open and Efficient Foundation Language Models](https://mlsanity.com/paper/2302.13971/) (2023)
- [Large Language Models Encode Clinical Knowledge](https://mlsanity.com/paper/2212.13138/) (2022)
- [Inference-Time Intervention: Eliciting Truthful Answers from a Language Model](https://mlsanity.com/paper/2306.03341/) (2023)
- [On Faithfulness and Factuality in Abstractive Summarization](https://mlsanity.com/paper/2005.00661/) (2020)
- [Unifying Large Language Models and Knowledge Graphs: A Roadmap](https://mlsanity.com/paper/2306.08302/) (2023)
- [The Internal State of an LLM Knows When It's Lying](https://mlsanity.com/paper/2304.13734/) (2023)
- [DoLa: Decoding by Contrasting Layers Improves Factuality in Large Language Models](https://mlsanity.com/paper/2309.03883/) (2023)
- [RARR: Researching and Revising What Language Models Say, Using Language Models](https://mlsanity.com/paper/2210.08726/) (2022)
- [FacTool: Factuality Detection in Generative AI -- A Tool Augmented Framework for Multi-Task and Multi-Domain Scenarios](https://mlsanity.com/paper/2307.13528/) (2023)
- [On the Robustness of ChatGPT: An Adversarial and Out-of-distribution Perspective](https://mlsanity.com/paper/2302.12095/) (2023)
- [Factuality Enhanced Language Models for Open-Ended Text Generation](https://mlsanity.com/paper/2206.04624/) (2022)
- [Fine-Tuning or Retrieval? Comparing Knowledge Injection in LLMs](https://mlsanity.com/paper/2312.05934/) (2023)
- [On Hallucination and Predictive Uncertainty in Conditional Language Generation](https://mlsanity.com/paper/2103.15025/) (2021)
- [GreaseLM: Graph REASoning Enhanced Language Models for Question Answering](https://mlsanity.com/paper/2201.08860/) (2022)
- [Sources of Hallucination by Large Language Models on Inference Tasks](https://mlsanity.com/paper/2305.14552/) (2023)
- [Verify-and-Edit: A Knowledge-Enhanced Chain-of-Thought Framework](https://mlsanity.com/paper/2305.03268/) (2023)
- [A Question-Entailment Approach to Question Answering](https://mlsanity.com/paper/1901.08079/) (2019)
- [Neural Path Hunter: Reducing Hallucination in Dialogue Systems via Path Grounding](https://mlsanity.com/paper/2104.08455/) (2021)
- [Enhancing Uncertainty-Based Hallucination Detection with Stronger Focus](https://mlsanity.com/paper/2311.13230/) (2023)
- [Mitigating Large Language Model Hallucinations via Autonomous Knowledge Graph-based Retrofitting](https://mlsanity.com/paper/2311.13314/) (2023)
- [Why Does ChatGPT Fall Short in Providing Truthful Answers?](https://mlsanity.com/paper/2304.10513/) (2023)
- [RHO (ρ): Reducing Hallucination in Open-domain Dialogues with Knowledge Grounding](https://mlsanity.com/paper/2212.01588/) (2022)
- [In-Context Sharpness as Alerts: An Inner Representation Perspective for Hallucination Mitigation](https://mlsanity.com/paper/2403.01548/) (2024)
- [Knowledge Solver: Teaching LLMs to Search for Domain Knowledge from Knowledge Graphs](https://mlsanity.com/paper/2309.03118/) (2023)
- [FLEEK: Factual Error Detection and Correction with Evidence Retrieved from External Knowledge](https://mlsanity.com/paper/2310.17119/) (2023)
- [Large Language Models Meet Knowledge Graphs to Answer Factoid Questions](https://mlsanity.com/paper/2310.02166/) (2023)

## Built on by (most cited first, 16 of 16)

- [Retrieval-Augmented Generation with Graphs (GraphRAG)](https://mlsanity.com/paper/2501.00309/) (2024)
- [Agent KB: Leveraging Cross-Domain Experience for Agentic Problem Solving](https://mlsanity.com/paper/2507.06229/) (2025)
- [Step-GUI Technical Report](https://mlsanity.com/paper/2512.15431/) (2025)
- [Position: Multimodal Large Language Models Can Significantly Advance Scientific Reasoning](https://mlsanity.com/paper/2502.02871/) (2025)
- [VeriSafe Agent: Safeguarding Mobile GUI Agent via Logic-based Action Verification](https://mlsanity.com/paper/2503.18492/) (2025)
- [Towards Reliable Medical Question Answering: Techniques and Challenges in Mitigating Hallucinations in Language Models](https://mlsanity.com/paper/2408.13808/) (2024)
- [Halu-J: Critique-Based Hallucination Judge](https://mlsanity.com/paper/2407.12943/) (2024)
- [SCOPE: Selective Conformal Optimized Pairwise LLM Judging](https://mlsanity.com/paper/2602.13110/) (2026)
- [RAG-KG-IL: A Multi-Agent Hybrid Framework for Reducing Hallucinations and Enhancing LLM Reasoning through RAG and Incremental Knowledge Graph Learning Integration](https://mlsanity.com/paper/2503.13514/) (2025)
- [LAD-RAG: Layout-aware Dynamic RAG for Visually-Rich Document Understanding](https://mlsanity.com/paper/2510.07233/) (2025)
- [A Survey on Proactive Defense Strategies Against Misinformation in Large Language Models](https://mlsanity.com/paper/2507.05288/) (2025)
- [Psychomatics -- A Multidisciplinary Framework for Understanding Artificial Minds](https://mlsanity.com/paper/2407.16444/) (2024)
- [TPU-Gen: LLM-Driven Custom Tensor Processing Unit Generator](https://mlsanity.com/paper/2503.05951/) (2025)
- [Noise Augmented Fine Tuning for Mitigating Hallucinations in Large Language Models](https://mlsanity.com/paper/2504.03302/) (2025)
- [From Found to Designed: Concepts as a Design Axis for Large Language Models](https://mlsanity.com/paper/2607.26825/) (2026)
- [Explainable Recommendations at Scale: LLM Rationales for YouTube Music Artist Discovery](https://mlsanity.com/paper/2609.23877/) (2026)

## Abstract (arXiv)

Large language models (LLMs) have demonstrated remarkable capabilities across various domains, although their susceptibility to hallucination poses significant challenges for their deployment in critical areas such as healthcare. To address this issue, retrieving relevant facts from knowledge graphs (KGs) is considered a promising method. Existing KG-augmented approaches tend to be resource-intensive, requiring multiple rounds of retrieval and verification for each factoid, which impedes their application in real-world scenarios. In this study, we propose Self-Refinement-Enhanced Knowledge Graph Retrieval (Re-KGR) to augment the factuality of LLMs' responses with less retrieval efforts in the medical field. Our approach leverages the attribution of next-token predictive probability distributions across different tokens, and various model layers to primarily identify tokens with a high potential for hallucination, reducing verification rounds by refining knowledge triples associated with these tokens. Moreover, we rectify inaccurate content using retrieved knowledge in the post-processing stage, which improves the truthfulness of generated responses. Experimental results on a medical dataset demonstrate that our approach can enhance the factual capability of LLMs across various foundational models as evidenced by the highest scores on truthfulness.