# Detecting Hallucination and Coverage Errors in Retrieval Augmented Generation for Controversial Topics

arXiv: https://arxiv.org/abs/2403.08904 · HTML: https://mlsanity.com/paper/2403.08904/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- Placed in Hallucination detection in RAG, one of 15 directions in Retrieval-augmented generation.
- Builds on 28 of its 43 references that are in the map.
- First posted on arXiv 13 Mar 2024.

## Placed in

- Retrieval-augmented generation / [Hallucination detection in RAG](https://mlsanity.com/rag/hallucination-detection-in-rag/)

## Builds on (most cited first, 28 of 28)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks](https://mlsanity.com/paper/2005.11401/) (2020)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [PaLM: Scaling Language Modeling with Pathways](https://mlsanity.com/paper/2204.02311/) (2022)
- [The Power of Scale for Parameter-Efficient Prompt Tuning](https://mlsanity.com/paper/2104.08691/) (2021)
- [Scaling Instruction-Finetuned Language Models](https://mlsanity.com/paper/2210.11416/) (2022)
- [LaMDA: Language Models for Dialog Applications](https://mlsanity.com/paper/2201.08239/) (2022)
- [Retrieval Augmentation Reduces Hallucination in Conversation](https://mlsanity.com/paper/2104.07567/) (2021)
- [The Woman Worked as a Babysitter: On Biases in Language Generation](https://mlsanity.com/paper/1909.01326/) (2019)
- [ToTTo: A Controlled Table-To-Text Generation Dataset](https://mlsanity.com/paper/2004.14373/) (2020)
- [Ex Machina: Personal Attacks Seen at Scale](https://mlsanity.com/paper/1610.08914/) (2016)
- [Visualizing and Understanding Neural Models in NLP](https://mlsanity.com/paper/1506.01066/) (2015)
- [Hurdles to Progress in Long-form Question Answering](https://mlsanity.com/paper/2103.06332/) (2021)
- [A Survey on Retrieval-Augmented Text Generation](https://mlsanity.com/paper/2202.01110/) (2022)
- [DART: Open-Domain Structured Data Record to Text Generation](https://mlsanity.com/paper/2007.02871/) (2020)
- [On the Origin of Hallucinations in Conversational Models: Is it the Datasets or the Models?](https://mlsanity.com/paper/2204.07931/) (2022)
- [The Language Interpretability Tool: Extensible, Interactive Visualizations and Analysis for NLP Models](https://mlsanity.com/paper/2008.05122/) (2020)
- [The elephant in the interpretability room: Why use attention as explanation when we have saliency methods?](https://mlsanity.com/paper/2010.05607/) (2020)
- [Handling Divergent Reference Texts when Evaluating Table-to-Text Generation](https://mlsanity.com/paper/1906.01081/) (2019)
- [Language Model Behavior: A Comprehensive Survey](https://mlsanity.com/paper/2303.11504/) (2023)
- [Detecting and Mitigating Hallucinations in Machine Translation: Model Internal Workings Alone Do Well, Sentence Similarity Even Better](https://mlsanity.com/paper/2212.08597/) (2022)
- [Analyzing the Source and Target Contributions to Predictions in Neural Machine Translation](https://mlsanity.com/paper/2010.10907/) (2020)
- [Extraction of Salient Sentences from Labelled Documents](https://mlsanity.com/paper/1412.6815/) (2014)
- [Towards Opening the Black Box of Neural Machine Translation: Source and Target Interpretations of the Transformer](https://mlsanity.com/paper/2205.11631/) (2022)
- [Evaluating Saliency Methods for Neural Language Models](https://mlsanity.com/paper/2104.05824/) (2021)
- [Towards Faithfulness in Open Domain Table-to-text Generation from an Entity-centric View](https://mlsanity.com/paper/2102.08585/) (2021)
- [Towards Agile Text Classifiers for Everyone](https://mlsanity.com/paper/2302.06541/) (2023)
- [Generation Challenges: Results of the Accuracy Evaluation Shared Task](https://mlsanity.com/paper/2108.05644/) (2021)

## Built on by (most cited first, 9 of 9)

- [LettuceDetect: A Hallucination Detection Framework for RAG Applications](https://mlsanity.com/paper/2502.17125/) (2025)
- [Enhancing Retrieval-Augmented Generation: A Study of Best Practices](https://mlsanity.com/paper/2501.07391/) (2025)
- [DioR: Adaptive Cognitive Detection and Contextual Retrieval Optimization for Dynamic Retrieval-Augmented Generation](https://mlsanity.com/paper/2504.10198/) (2025)
- [Can Large Language Models Replace Human Subjects? A Large-Scale Replication of Scenario-Based Experiments in Psychology and Management](https://mlsanity.com/paper/2409.00128/) (2024)
- [ConQRet: Benchmarking Fine-Grained Evaluation of Retrieval Augmented Argumentation with LLM Judges](https://mlsanity.com/paper/2412.05206/) (2024)
- [Hybrid Retrieval for Hallucination Mitigation in Large Language Models: A Comparative Analysis](https://mlsanity.com/paper/2504.05324/) (2025)
- [On A Scale From 1 to 5: Quantifying Hallucination in Faithfulness Evaluation](https://mlsanity.com/paper/2410.12222/) (2024)
- [First Hallucination Tokens Are Different from Conditional Ones](https://mlsanity.com/paper/2507.20836/) (2025)
- [Improving Neutral Point-of-View Generation with Data- and Parameter-Efficient RL](https://mlsanity.com/paper/2503.03654/) (2025)

## Abstract (arXiv)

We explore a strategy to handle controversial topics in LLM-based chatbots based on Wikipedia's Neutral Point of View (NPOV) principle: acknowledge the absence of a single true answer and surface multiple perspectives. We frame this as retrieval augmented generation, where perspectives are retrieved from a knowledge base and the LLM is tasked with generating a fluent and faithful response from the given perspectives. As a starting point, we use a deterministic retrieval system and then focus on common LLM failure modes that arise during this approach to text generation, namely hallucination and coverage errors. We propose and evaluate three methods to detect such errors based on (1) word-overlap, (2) salience, and (3) LLM-based classifiers. Our results demonstrate that LLM-based classifiers, even when trained only on synthetic errors, achieve high error detection performance, with ROC AUC scores of 95.3% for hallucination and 90.5% for coverage error detection on unambiguous error cases. We show that when no training data is available, our other methods still yield good results on hallucination (84.0%) and coverage error (85.2%) detection.