# Minimizing Factual Inconsistency and Hallucination in Large Language Models

arXiv: https://arxiv.org/abs/2311.13878 · HTML: https://mlsanity.com/paper/2311.13878/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 15 of its 27 references that are in the map.
- First posted on arXiv 23 Nov 2023.

## Builds on (most cited first, 15 of 15)

- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment](https://mlsanity.com/paper/2303.16634/) (2023)
- [Improving language models by retrieving from trillions of tokens](https://mlsanity.com/paper/2112.04426/) (2021)
- [PubMedQA: A Dataset for Biomedical Research Question Answering](https://mlsanity.com/paper/1909.06146/) (2019)
- [When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memories](https://mlsanity.com/paper/2212.10511/) (2022)
- [MPNet: Masked and Permuted Pre-training for Language Understanding](https://mlsanity.com/paper/2004.09297/) (2020)
- [How Much Knowledge Can You Pack Into the Parameters of a Language Model?](https://mlsanity.com/paper/2002.08910/) (2020)
- [Augmented Language Models: a Survey](https://mlsanity.com/paper/2302.07842/) (2023)
- [FinBERT: A Pretrained Language Model for Financial Communications](https://mlsanity.com/paper/2006.08097/) (2020)
- [Factual Error Correction for Abstractive Summarization Models](https://mlsanity.com/paper/2010.08712/) (2020)
- [Subgraph Retrieval Enhanced Model for Multi-hop Knowledge Base Question Answering](https://mlsanity.com/paper/2202.13296/) (2022)
- [Generating Diverse and Consistent QA pairs from Contexts with Information-Maximizing Hierarchical Conditional VAEs](https://mlsanity.com/paper/2005.13837/) (2020)
- [ExtraPhrase: Efficient Data Augmentation for Abstractive Summarization](https://mlsanity.com/paper/2201.05313/) (2022)
- [Large-Scale Knowledge Synthesis and Complex Information Retrieval from Biomedical Documents](https://mlsanity.com/paper/2302.06854/) (2023)

## Built on by (most cited first, 5 of 5)

- [MyVLM: Personalizing VLMs for User-Specific Queries](https://mlsanity.com/paper/2403.14599/) (2024)
- [Graph Machine Learning in the Era of Large Language Models (LLMs)](https://mlsanity.com/paper/2404.14928/) (2024)
- [A Scoping Review of Natural Language Processing in Addressing Medically Inaccurate Information: Errors, Misinformation, and Hallucination](https://mlsanity.com/paper/2505.00008/) (2025)
- [Self-reflecting Large Language Models: A Hegelian Dialectical Approach](https://mlsanity.com/paper/2501.14917/) (2025)
- [Automating the Detection of Requirement Dependencies Using Large Language Models](https://mlsanity.com/paper/2602.22456/) (2026)

## Abstract (arXiv)

Large Language Models (LLMs) are widely used in critical fields such as healthcare, education, and finance due to their remarkable proficiency in various language-related tasks. However, LLMs are prone to generating factually incorrect responses or "hallucinations," which can lead to a loss of credibility and trust among users. To address this issue, we propose a multi-stage framework that generates the rationale first, verifies and refines incorrect ones, and uses them as supporting references to generate the answer. The generated rationale enhances the transparency of the answer and our framework provides insights into how the model arrived at this answer, by using this rationale and the references to the context. In this paper, we demonstrate its effectiveness in improving the quality of responses to drug-related inquiries in the life sciences industry. Our framework improves traditional Retrieval Augmented Generation (RAG) by enabling OpenAI GPT-3.5-turbo to be 14-25% more faithful and 16-22% more accurate on two datasets. Furthermore, fine-tuning samples based on our framework improves the accuracy of smaller open-access LLMs by 33-42% and competes with RAG on commercial models.