# Metric Ensembles For Hallucination Detection

arXiv: https://arxiv.org/abs/2310.10495 · HTML: https://mlsanity.com/paper/2310.10495/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 29 references that are in the map.
- First posted on arXiv 16 Oct 2023.

## Builds on (most cited first, 17 of 17)

- [Self-Consistency Improves Chain of Thought Reasoning in Language Models](https://mlsanity.com/paper/2203.11171/) (2022)
- [Sparks of Artificial General Intelligence: Early experiments with GPT-4](https://mlsanity.com/paper/2303.12712/) (2023)
- [Survey of Hallucination in Natural Language Generation](https://mlsanity.com/paper/2202.03629/) (2022)
- [Improving Factuality and Reasoning in Language Models through Multiagent Debate](https://mlsanity.com/paper/2305.14325/) (2023)
- [LaMDA: Language Models for Dialog Applications](https://mlsanity.com/paper/2201.08239/) (2022)
- [SelfCheckGPT: Zero-Resource Black-Box Hallucination Detection for Generative Large Language Models](https://mlsanity.com/paper/2303.08896/) (2023)
- [Evaluating the Factual Consistency of Abstractive Text Summarization](https://mlsanity.com/paper/1910.12840/) (2019)
- [SummaC: Re-Visiting NLI-based Models for Inconsistency Detection in Summarization](https://mlsanity.com/paper/2111.09525/) (2021)
- [Asking and Answering Questions to Evaluate the Factual Consistency of Summaries](https://mlsanity.com/paper/2004.04228/) (2020)
- [News Summarization and Evaluation in the Era of GPT-3](https://mlsanity.com/paper/2209.12356/) (2022)
- [FEQA: A Question Answering Evaluation Framework for Faithfulness Assessment in Abstractive Summarization](https://mlsanity.com/paper/2005.03754/) (2020)
- [Assessing The Factual Accuracy of Generated Text](https://mlsanity.com/paper/1905.13322/) (2019)
- [Towards Question-Answering as an Automatic Metric for Evaluating the Content Quality of a Summary](https://mlsanity.com/paper/2010.00490/) (2020)
- [The Factual Inconsistency Problem in Abstractive Text Summarization: A Survey](https://mlsanity.com/paper/2104.14839/) (2021)
- [SummIt: Iterative Text Summarization via ChatGPT](https://mlsanity.com/paper/2305.14835/) (2023)
- [Estimating Accuracy from Unlabeled Data: A Probabilistic Logic Approach](https://mlsanity.com/paper/1705.07086/) (2017)
- [SMART: Sentences as Basic Units for Text Evaluation](https://mlsanity.com/paper/2208.01030/) (2022)

## Built on by (most cited first, 2 of 2)

- [Securing Large Language Models: Addressing Bias, Misinformation, and Prompt Attacks](https://mlsanity.com/paper/2409.08087/) (2024)
- [Large Language Model for Mental Health: A Systematic Review](https://mlsanity.com/paper/2403.15401/) (2024)

## Abstract (arXiv)

Abstractive text summarization has garnered increased interest as of late, in part due to the proliferation of large language models (LLMs). One of the most pressing problems related to generation of abstractive summaries is the need to reduce "hallucinations," information that was not included in the document being summarized, and which may be wholly incorrect. Due to this need, a wide array of metrics estimating consistency with the text being summarized have been proposed. We examine in particular a suite of unsupervised metrics for summary consistency, and measure their correlations with each other and with human evaluation scores in the wiki_bio_gpt3_hallucination dataset. We then compare these evaluations to models made from a simple linear ensemble of these metrics. We find that LLM-based methods outperform other unsupervised metrics for hallucination detection. We also find that ensemble methods can improve these scores even further, provided that the metrics in the ensemble have sufficiently similar and uncorrelated error rates. Finally, we present an ensemble method for LLM-based evaluations that we show improves over this previous SOTA.