# All Claims Are Equal, but Some Claims Are More Equal Than Others: Importance-Sensitive Factuality Evaluation of LLM Generations

arXiv: https://arxiv.org/abs/2510.07083 · HTML: https://mlsanity.com/paper/2510.07083/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 25 references that are in the map.
- First posted on arXiv 8 Oct 2025.

## Builds on (most cited first, 18 of 18)

- [HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answering](https://mlsanity.com/paper/1809.09600/) (2018)
- [TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehension](https://mlsanity.com/paper/1705.03551/) (2017)
- [FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generation](https://mlsanity.com/paper/2305.14251/) (2023)
- [BRIGHT: A Realistic and Challenging Benchmark for Reasoning-Intensive Retrieval](https://mlsanity.com/paper/2407.12883/) (2024)
- [VERISCORE: Evaluating the factuality of verifiable claims in long-form text generation](https://mlsanity.com/paper/2406.19276/) (2024)
- [On Measures of Biases and Harms in NLP](https://mlsanity.com/paper/2108.03362/) (2021)
- [Molecular Facts: Desiderata for Decontextualization in LLM Fact Verification](https://mlsanity.com/paper/2406.20079/) (2024)
- [A Closer Look at Claim Decomposition](https://mlsanity.com/paper/2403.11903/) (2024)
- [Initial Nugget Evaluation Results for the TREC 2024 RAG Track with the AutoNuggetizer Framework](https://mlsanity.com/paper/2411.09607/) (2024)
- [WildHallucinations: Evaluating Long-form Factuality in LLMs with Real-World Entity Queries](https://mlsanity.com/paper/2407.17468/) (2024)
- [DnDScore: Decontextualization and Decomposition for Factuality Verification in Long-Form Text Generation](https://mlsanity.com/paper/2412.13175/) (2024)
- [Core: Robust Factual Precision with Informative Sub-Claim Identification](https://mlsanity.com/paper/2407.03572/) (2024)
- [Beyond Factual Accuracy: Evaluating Coverage of Diverse Factual Information in Long-form Text Generation](https://mlsanity.com/paper/2501.03545/) (2025)
- [Localizing Factual Inconsistencies in Attributable Text Generation](https://mlsanity.com/paper/2410.07473/) (2024)
- [Verify with Caution: The Pitfalls of Relying on Imperfect Factuality Metrics](https://mlsanity.com/paper/2501.14883/) (2025)
- [Conversational Gold: Evaluating Personalized Conversational Search System using Gold Nuggets](https://mlsanity.com/paper/2503.09902/) (2025)
- [CLATTER: Comprehensive Entailment Reasoning for Hallucination Detection](https://mlsanity.com/paper/2506.05243/) (2025)
- [How Does Response Length Affect Long-Form Factuality](https://mlsanity.com/paper/2505.23295/) (2025)

## Built on by (most cited first, 6 of 6)

- [Beyond Precision: Importance-Aware Recall for Factuality Evaluation in Long-Form LLM Generation](https://mlsanity.com/paper/2604.03141/) (2026)
- [Two-Level Meta-Rubrics for Evaluating Open-Ended Generation: GAMUT, a Benchmark for Factual Completeness](https://mlsanity.com/paper/2607.19322/) (2026)
- [Your Mouse and Eyes Secretly Leak Your Preference: LLM Alignment using Implicit Feedback from Users](https://mlsanity.com/paper/2606.20482/) (2026)
- [VetScore: Risk-Weighted Fact Verification for Veterinary Long-Form QA with Citations](https://mlsanity.com/paper/2608.03675/) (2026)
- [A Proposed Rubric for Evaluating Expressed Clinical Reasoning in Large Language Model Responses](https://mlsanity.com/paper/2609.37788/) (2026)
- [A rubric landscape for evaluating clinical reasoning in large language models: what exists, what is missing, and what needs to be combined](https://mlsanity.com/paper/2610.01938/) (2026)

## Abstract (arXiv)

Existing methods for evaluating the factuality of large language model (LLM) responses treat all claims as equally important. This results in misleading evaluations when vital information is missing or incorrect as it receives the same weight as peripheral details, raising the question: how can we reliably detect such differences when there are errors in key information? Current approaches that measure factuality tend to be insensitive to omitted or false key information. To investigate this lack of sensitivity, we construct VITALERRORS, a benchmark of 6,733 queries with minimally altered LLM responses designed to omit or falsify key information. Using this dataset, we demonstrate the insensitivities of existing evaluation metrics to key information errors. To address this gap, we introduce VITAL, a set of metrics that provide greater sensitivity in measuring the factuality of responses by incorporating the relevance and importance of claims with respect to the query. Our analysis demonstrates that VITAL metrics more reliably detect errors in key information than previous methods. Our dataset, metrics, and analysis provide a foundation for more accurate and robust assessment of LLM factuality.