# Comprehensiveness Metrics for Automatic Evaluation of Factual Recall in Text Generation

arXiv: https://arxiv.org/abs/2510.07926 · HTML: https://mlsanity.com/paper/2510.07926/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 46 references that are in the map.
- First posted on arXiv 9 Oct 2025.

## Builds on (most cited first, 18 of 18)

- [TruthfulQA: Measuring How Models Mimic Human Falsehoods](https://mlsanity.com/paper/2109.07958/) (2021)
- [FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation](https://mlsanity.com/paper/2305.14251/) (2023)
- [Asking and Answering Questions to Evaluate the Factual Consistency of Summaries](https://mlsanity.com/paper/2004.04228/) (2020)
- [FEQA: A Question Answering Evaluation Framework for Faithfulness Assessment in Abstractive Summarization](https://mlsanity.com/paper/2005.03754/) (2020)
- [FacTool: Factuality Detection in Generative AI -- A Tool Augmented Framework for Multi-Task and Multi-Domain Scenarios](https://mlsanity.com/paper/2307.13528/) (2023)
- [QuestEval: Summarization Asks for Fact-based Evaluation](https://mlsanity.com/paper/2103.12693/) (2021)
- [VERISCORE: Evaluating the factuality of verifiable claims in long-form text generation](https://mlsanity.com/paper/2406.19276/) (2024)
- [ConflictBank: A Benchmark for Evaluating the Influence of Knowledge Conflicts in LLM](https://mlsanity.com/paper/2408.12076/) (2024)
- [Factuality of Large Language Models: A Survey](https://mlsanity.com/paper/2402.02420/) (2024)
- [Question Answering as an Automatic Evaluation Metric for News Article Summarization](https://mlsanity.com/paper/1906.00318/) (2019)
- [SimpleQA Verified: A Reliable Factuality Benchmark to Measure Parametric Knowledge](https://mlsanity.com/paper/2509.07968/) (2025)
- [WikiContradict: A Benchmark for Evaluating LLMs on Real-World Knowledge Conflicts from Wikipedia](https://mlsanity.com/paper/2406.13805/) (2024)
- [Long2RAG: Evaluating Long-Context & Long-Form Retrieval-Augmented Generation with Key Point Recall](https://mlsanity.com/paper/2410.23000/) (2024)
- [MQAG: Multiple-choice Question Answering and Generation for Assessing Information Consistency in Summarization](https://mlsanity.com/paper/2301.12307/) (2023)
- [FactBench: A Dynamic Benchmark for In-the-Wild Language Model Factuality Evaluation](https://mlsanity.com/paper/2410.22257/) (2024)
- [The Great Nugget Recall: Automating Fact Extraction and RAG Evaluation with Large Language Models](https://mlsanity.com/paper/2504.15068/) (2025)
- [Beyond Factual Accuracy: Evaluating Coverage of Diverse Factual Information in Long-form Text Generation](https://mlsanity.com/paper/2501.03545/) (2025)
- [VeriFact: Enhancing Long-Form Factuality Evaluation with Refined Fact Extraction and Reference Facts](https://mlsanity.com/paper/2505.09701/) (2025)

## Built on by (most cited first, 2 of 2)

- [Beyond Precision: Importance-Aware Recall for Factuality Evaluation in Long-Form LLM Generation](https://mlsanity.com/paper/2604.03141/) (2026)
- [FactCorrector: A Graph-Inspired Approach to Long-Form Factuality Correction of Large Language Models](https://mlsanity.com/paper/2601.11232/) (2026)

## Abstract (arXiv)

Despite demonstrating remarkable performance across a wide range of tasks, large language models (LLMs) have also been found to frequently produce outputs that are incomplete or selectively omit key information. In sensitive domains, such omissions can result in significant harm comparable to that posed by factual inaccuracies, including hallucinations. In this study, we address the challenge of evaluating the comprehensiveness of LLM-generated texts, focusing on the detection of missing information or underrepresented viewpoints. We investigate three automated evaluation metrics: (1) an NLI-based method that decomposes texts into atomic statements and uses natural language inference (NLI) to identify missing facts, (2) a Q&A-based metric that extracts question-answer pairs and compares responses across sources, and (3) an end-to-end approach that directly identifies missing content using LLMs. Our experiments demonstrate the surprising effectiveness of the simple end-to-end metric compared to more complex metrics, though at the cost of reduced robustness, interpretability and result granularity. We further assess the comprehensiveness of responses from several popular open-weight LLMs when answering user queries based on multiple sources.