# Provenance: A Light-weight Fact-checker for Retrieval Augmented LLM Generation Output

arXiv: https://arxiv.org/abs/2411.01022 · HTML: https://mlsanity.com/paper/2411.01022/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Placed in RAG robustness and adaptive retrieval, one of 15 directions in Retrieval-augmented generation.
- Builds on 12 of its 18 references that are in the map.
- First posted on arXiv 1 Nov 2024.

## Placed in

- Retrieval-augmented generation / [RAG robustness and adaptive retrieval](https://mlsanity.com/rag/rag-robustness-and-adaptive-retrieval/)

## Builds on (most cited first, 12 of 12)

- [HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering](https://mlsanity.com/paper/1809.09600/) (2018)
- [TruthfulQA: Measuring How Models Mimic Human Falsehoods](https://mlsanity.com/paper/2109.07958/) (2021)
- [The Curious Case of Neural Text Degeneration](https://mlsanity.com/paper/1904.09751/) (2019)
- [Language Models (Mostly) Know What They Know](https://mlsanity.com/paper/2207.05221/) (2022)
- [MS MARCO: A Human Generated MAchine Reading COmprehension Dataset](https://mlsanity.com/paper/1611.09268/) (2016)
- [SummaC: Re-Visiting NLI-based Models for Inconsistency Detection in Summarization](https://mlsanity.com/paper/2111.09525/) (2021)
- [JudgeLM: Fine-tuned Large Language Models are Scalable Judges](https://mlsanity.com/paper/2310.17631/) (2023)
- [A Stitch in Time Saves Nine: Detecting and Mitigating Hallucinations of LLMs by Validating Low-Confidence Generation](https://mlsanity.com/paper/2307.03987/) (2023)
- [Do Language Models Know When They're Hallucinating References?](https://mlsanity.com/paper/2305.18248/) (2023)
- [Lynx: An Open Source Hallucination Evaluation Model](https://mlsanity.com/paper/2407.08488/) (2024)
- [Evaluating and Modeling Attribution for Cross-Lingual Question Answering](https://mlsanity.com/paper/2305.14332/) (2023)
- [Neural Models for Sequence Chunking](https://mlsanity.com/paper/1701.04027/) (2017)

## Built on by (most cited first, 4 of 4)

- [Chain-of-Thought Prompting Obscures Hallucination Cues in Large Language Models: An Empirical Evaluation](https://mlsanity.com/paper/2506.17088/) (2025)
- [Hallucination to Truth: A Review of Fact-Checking and Factuality Evaluation in Large Language Models](https://mlsanity.com/paper/2508.03860/) (2025)
- [FActBench: A Benchmark for Fine-grained Automatic Evaluation of LLM-Generated Text in the Medical Domain](https://mlsanity.com/paper/2509.02198/) (2025)
- [Beyond Explicit Refusals: Soft-Failure Attacks on Retrieval-Augmented Generation](https://mlsanity.com/paper/2604.18663/) (2026)

## Abstract (arXiv)

We present a light-weight approach for detecting nonfactual outputs from retrieval-augmented generation (RAG). Given a context and putative output, we compute a factuality score that can be thresholded to yield a binary decision to check the results of LLM-based question-answering, summarization, or other systems. Unlike factuality checkers that themselves rely on LLMs, we use compact, open-source natural language inference (NLI) models that yield a freely accessible solution with low latency and low cost at run-time, and no need for LLM fine-tuning. The approach also enables downstream mitigation and correction of hallucinations, by tracing them back to specific context chunks. Our experiments show high area under the ROC curve (AUC) across a wide range of relevant open source datasets, indicating the effectiveness of our method for fact-checking RAG output.