mlsanityData to 7 Oct 2026

Beyond Factual Accuracy: Evaluating Coverage of Diverse Factual Information in Long-form Text Generation

arXiv 2501.03545 ↗First posted 7 Jan 2025cs.CLcited by 15

Not on the map. Why?

Abstract (arXiv)

This paper presents ICAT, an evaluation framework for measuring coverage of diverse factual information in long-form text generation. ICAT breaks down a long output text into a list of atomic claims and not only verifies each claim through retrieval from a (reliable) knowledge source, but also computes the alignment between the atomic factual claims and various aspects expected to be presented in the output. We study three implementations of the ICAT framework, each with a different assumption on the availability of aspects and alignment method. By adopting data from the diversification task in the TREC Web Track and the ClueWeb corpus, we evaluate the ICAT framework. We demonstrate strong correlation with human judgments and provide comprehensive evaluation across multiple state-of-the-art LLMs. Our framework further offers interpretable and fine-grained analysis of diversity and coverage. Its modular design allows for easy adaptation to different domains and datasets, making it a valuable tool for evaluating the qualitative aspects of long-form responses produced by LLMs.

Key facts. Cited by 15 papers in the map (data through 7 Oct 2026). Builds on 24 of its 39 references that are in the map. First posted on arXiv 7 Jan 2025.

Builds on

24 of 39 references.

References · 24

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2024The Llama 3 Herd of Modelscited by 14,192
  3. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  4. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  5. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  6. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
  7. 2017Billion-scale similarity search with GPUscited by 2,477
  8. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,205
  9. 2018FEVER: a large-scale dataset for Fact Extraction and VERificationcited by 1,362
  10. 2023FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generationcited by 1,259
  11. 2021DeBERTaV3: Improving DeBERTa using ELECTRA-Style Pre-Training with Gradient-Disentangled Embedding Sharingcited by 1,219
  12. 2020BLEURT: Learning Robust Metrics for Text Generationcited by 1,196
12 more
  1. 2015A Diversity-Promoting Objective Function for Neural Conversation Modelscited by 1,068
  2. 2019Adversarial NLI: A New Benchmark for Natural Language Understandingcited by 798
  3. 2024LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation Methodscited by 451
  4. 2022Towards a Unified Multi-Dimensional Evaluator for Text Generationcited by 264
  5. 2023OpenChat: Advancing Open-source Language Models with Mixed-Quality Datacited by 250
  6. 2020USR: An Unsupervised and Reference Free Evaluation Metric for Dialog Generationcited by 186
  7. 2020Automatic Machine Translation Evaluation in Many Languages via Zero-Shot Paraphrasingcited by 114
  8. 2024VERISCORE: Evaluating the factuality of verifiable claims in long-form text generationcited by 114
  9. 2024Arctic-Embed: Scalable, Efficient, and Accurate Text Embedding Modelscited by 60
  10. 2024Initial Nugget Evaluation Results for the TREC 2024 RAG Track with the AutoNuggetizer Frameworkcited by 28
  11. 2021Assessing Reference-Free Peer Evaluation for Machine Translationcited by 8
  12. 2024An Exam-based Evaluation Approach Beyond Traditional Relevance Judgmentscited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 7; 2026: 8.

15 papers.

Sort:

By area:

  1. 2025LaMP-QA: A Benchmark for Personalized Long-form Question Answeringcited by 30
  2. 2025ExPerT: Effective and Explainable Evaluation of Personalized Long-Form Text Generationcited by 26
  3. 2025Mitigating Manipulation and Enhancing Persuasion: A Reflective Multi-Agent Approach for Legal Argument Generationcited by 8
  4. 2025All Claims Are Equal, but Some Claims Are More Equal Than Others: Importance-Sensitive Factuality Evaluation of LLM Generationscited by 6
  5. 2026Beyond Precision: Importance-Aware Recall for Factuality Evaluation in Long-Form LLM Generationcited by 4
  6. 2025Comprehensiveness Metrics for Automatic Evaluation of Factual Recall in Text Generationcited by 2
  7. 2025LongWeave: A Long-Form Generation Benchmark Bridging Real-World Relevance and Verifiabilitycited by 2
  8. 2026Mind2Report: Expert-Level Commercial Report Synthesis via Cognitive Deep Research Agentcited by 2
  9. 2026Beyond Relevance: On the Relationship Between Retrieval and RAG Information Coveragecited by 1
  10. 2026LongSumEval: Question-Answering Based Evaluation and Feedback-Driven Refinement for Long Document Summarizationcited by 1
  11. 2026CREDENCE: Claim Reduction for Decomposition & Enhanced Credibility -- Semantic Metrics and Convergence Analysiscited by 1
  12. 2025Plan-and-Refine: Diverse and Comprehensive Retrieval-Augmented Generationcited by 0
3 more
  1. 2026Parametric Knowledge in RAG-SFT for Domain-Specific Document Generationcited by 0
  2. 2026Local-to-Global Sentence-Level Graph Reranking for Scientific Synthesiscited by 0
  3. 2026LOOMSUM:Weaving Quantitative and Narrative Evidence for Faithful Long Text-Table Summarizationcited by 0