# Verify with Caution: The Pitfalls of Relying on Imperfect Factuality Metrics

arXiv: https://arxiv.org/abs/2501.14883 · HTML: https://mlsanity.com/paper/2501.14883/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 31 references that are in the map.
- First posted on arXiv 24 Jan 2025.

## Builds on (most cited first, 19 of 19)

- [Beyond Accuracy: Behavioral Testing of NLP models with CheckList](https://mlsanity.com/paper/2005.04118/) (2020)
- [Prometheus 2: An Open Source Language Model Specialized in Evaluating Other Language Models](https://mlsanity.com/paper/2405.01535/) (2024)
- [RAGTruth: A Hallucination Corpus for Developing Trustworthy Retrieval-Augmented Language Models](https://mlsanity.com/paper/2401.00396/) (2023)
- [MiniCheck: Efficient Fact-Checking of LLMs on Grounding Documents](https://mlsanity.com/paper/2404.10774/) (2024)
- [Measuring Attribution in Natural Language Generation Models](https://mlsanity.com/paper/2112.12870/) (2021)
- [To Ship or Not to Ship: An Extensive Evaluation of Automatic Metrics for Machine Translation](https://mlsanity.com/paper/2107.10821/) (2021)
- [Tangled up in BLEU: Reevaluating the Evaluation of Automatic Machine Translation Evaluation Metrics](https://mlsanity.com/paper/2006.06264/) (2020)
- [Complex Claim Verification with Evidence Retrieved in the Wild](https://mlsanity.com/paper/2305.11859/) (2023)
- [LongEval: Guidelines for Human Evaluation of Faithfulness in Long-form Summarization](https://mlsanity.com/paper/2301.13298/) (2023)
- [Foundational Autoraters: Taming Large Language Models for Better Automatic Evaluation](https://mlsanity.com/paper/2407.10817/) (2024)
- [TofuEval: Evaluating Hallucinations of LLMs on Topic-Focused Dialogue Summarization](https://mlsanity.com/paper/2402.13249/) (2024)
- [PURR: Efficiently Editing Language Model Hallucinations by Denoising Language Model Corruptions](https://mlsanity.com/paper/2305.14908/) (2023)
- [Perturbation CheckLists for Evaluating NLG Evaluation Metrics](https://mlsanity.com/paper/2109.05771/) (2021)
- [Understanding Retrieval Augmentation for Long-Form Question Answering](https://mlsanity.com/paper/2310.12150/) (2023)
- [AttributionBench: How Hard is Automatic Attribution Evaluation?](https://mlsanity.com/paper/2402.15089/) (2024)
- [On Positional Bias of Faithfulness for Long-form Summarization](https://mlsanity.com/paper/2410.23609/) (2024)
- [NLG Evaluation Metrics Beyond Correlation Analysis: An Empirical Metric Preference Checklist](https://mlsanity.com/paper/2305.08566/) (2023)
- [Do Automatic Factuality Metrics Measure Factuality? A Critical Evaluation](https://mlsanity.com/paper/2411.16638/) (2024)
- [Schroedinger's Threshold: When the AUC doesn't predict Accuracy](https://mlsanity.com/paper/2404.03344/) (2024)

## Built on by (most cited first, 8 of 8)

- [How to Correctly Report LLM-as-a-Judge Evaluations](https://mlsanity.com/paper/2511.21140/) (2025)
- [Verifying the Verifiers: Unveiling Pitfalls and Potentials in Fact Verifiers](https://mlsanity.com/paper/2506.13342/) (2025)
- [Evaluating Evaluation Metrics -- The Mirage of Hallucination Detection](https://mlsanity.com/paper/2504.18114/) (2025)
- [All Claims Are Equal, but Some Claims Are More Equal Than Others: Importance-Sensitive Factuality Evaluation of LLM Generations](https://mlsanity.com/paper/2510.07083/) (2025)
- [How College Students Use AI to Navigate Course Readings: Evidence from an Eight-Week Study](https://mlsanity.com/paper/2602.09907/) (2026)
- [Does Finetuning with Scientific Data Increase Hallucinations? A Multi-domain Factuality Evaluation of LLMs](https://mlsanity.com/paper/2606.21359/) (2026)
- [Can We Trust the Judges? Validation of Factuality Evaluation Methods via Answer Perturbation](https://mlsanity.com/paper/2609.15561/) (2026)
- [RAIM: Robust Aggregation of Inexpensive Models for Hallucination Detection](https://mlsanity.com/paper/2609.39229/) (2026)

## Abstract (arXiv)

Improvements in large language models have led to increasing optimism that they can serve as reliable evaluators of natural language generation outputs. In this paper, we challenge this optimism by thoroughly re-evaluating five state-of-the-art factuality metrics on a collection of 11 datasets for summarization, retrieval-augmented generation, and question answering. We find that these evaluators are inconsistent with each other and often misestimate system-level performance, both of which can lead to a variety of pitfalls. We further show that these metrics exhibit biases against highly paraphrased outputs and outputs that draw upon faraway parts of the source documents. We urge users of these factuality metrics to proceed with caution and manually validate the reliability of these metrics in their domain of interest before proceeding.