mlsanityData to 14 Oct 2026

Long-Form Information Alignment Evaluation Beyond Atomic Facts

arXiv 2505.15792 ↗First posted 21 May 2025cs.CL cs.AI cs.LGcited by 4

Not on the map. Why?

Abstract (arXiv)

Information alignment evaluators are vital for various NLG evaluation tasks and trustworthy LLM deployment, reducing hallucinations and enhancing user trust. Current fine-grained methods, like FactScore, verify facts individually but neglect inter-fact dependencies, enabling subtle vulnerabilities. In this work, we introduce MontageLie, a challenging benchmark that constructs deceptive narratives by "montaging" truthful statements without introducing explicit hallucinations. We demonstrate that both coarse-grained LLM-based evaluators and current fine-grained frameworks are susceptible to this attack, with AUC-ROC scores falling below 65%. To enable more robust fine-grained evaluation, we propose DoveScore, a novel framework that jointly verifies factual accuracy and event-order consistency. By modeling inter-fact relationships, DoveScore outperforms existing fine-grained methods by over 8%, providing a more robust solution for long-form text alignment evaluation. Our code and datasets are available at https://github.com/dannalily/DoveScore.

Key facts. Cited by 4 papers in the map (data through 14 Oct 2026). Builds on 33 of its 54 references that are in the map. First posted on arXiv 21 May 2025.

Builds on

33 of 54 references.

References · 33

  1. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,512
  2. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,276
  3. 2023Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflectioncited by 1,839
  4. 2018FEVER: a large-scale dataset for Fact Extraction and VERificationcited by 1,375
  5. 2023FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generationcited by 1,282
  6. 2020BLEURT: Learning Robust Metrics for Text Generationcited by 1,203
  7. 2021BARTScore: Evaluating Generated Text as Text Generationcited by 722
  8. 2019Evaluating the Factual Consistency of Abstractive Text Summarizationcited by 584
  9. 2021SummaC: Re-Visiting NLI-based Models for Inconsistency Detection in Summarizationcited by 407
  10. 2023Evaluating Verifiability in Generative Search Enginescited by 303
  11. 2023AlignScore: Evaluating Factual Consistency with a Unified Alignment Functioncited by 288
  12. 2024MiniCheck: Efficient Fact-Checking of LLMs on Grounding Documentscited by 257
21 more
  1. 2021QuestEval: Summarization Asks for Fact-based Evaluationcited by 244
  2. 2021QAFactEval: Improved QA-Based Factual Consistency Evaluation for Summarizationcited by 236
  3. 2021BookSum: A Collection of Datasets for Long-form Narrative Summarizationcited by 184
  4. 2023AVeriTeC: A Dataset for Real-world Claim Verification with Evidence from the Webcited by 144
  5. 2024VERISCORE: Evaluating the factuality of verifiable claims in long-form text generationcited by 117
  6. 2022Understanding Factual Errors in Summarization: Errors, Summarizers, Datasets, Error Detectorscited by 104
  7. 2024FineSurE: Fine-grained Summarization Evaluation using LLMscited by 83
  8. 2023ChatGPT as a Factual Inconsistency Evaluator for Text Summarizationcited by 75
  9. 2024TofuEval: Evaluating Hallucinations of LLMs on Topic-Focused Dialogue Summarizationcited by 66
  10. 2021Compression, Transduction, and Creation: A Unified Framework for Evaluating Natural Language Generationcited by 60
  11. 2023EX-FEVER: A Dataset for Multi-hop Explainable Fact Verificationcited by 36
  12. 2024Atomic Fact Decomposition Helps Attributed Question Answeringcited by 21
  13. 2024TrustScore: Reference-Free Evaluation of LLM Response Trustworthinesscited by 19
  14. 2025LLM-Generated Fake News Induces Truth Decay in News Ecosystem: A Case Study on Neural News Recommendationcited by 19
  15. 2022Knowledge Graph Reasoning with Logics and Embeddings: Survey and Perspectivecited by 18
  16. 2024Prompting Large Language Models with Knowledge Graphs for Question Answering Involving Long-tail Factscited by 14
  17. 2024Fine-grained and Explainable Factuality Evaluation for Multimodal Summarizationcited by 10
  18. 2024CHECKWHY: Causal Fact Verification via Argument Structurecited by 9
  19. 2024Learning When to Retrieve, What to Rewrite, and How to Respond in Conversational QAcited by 9
  20. 2025Self-Reasoning Language Models: Unfold Hidden Reasoning Chains with Few Reasoning Catalystcited by 6
  21. 2025Masking in Multi-hop QA: An Analysis of How Language Models Perform with Context Permutationcited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 3.

4 papers.

  1. 2026Two-Level Meta-Rubrics for Evaluating Open-Ended Generation: GAMUT, a Benchmark for Factual Completenesscited by 2
  2. 2026The Deliberative Illusion: Diagnosing Factual Attrition and Stance Homogenization in Multi-Agent LLM Deliberationcited by 1
  3. 2025Uncovering and Mitigating Transient Blindness in Multimodal Model Editingcited by 0
  4. 2026Judge Like Human Examiners: A Weighted Importance Multi-Point Evaluation Framework for Generative Tasks with Long-form Answerscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Trusted Source Alignment in Large Language Models
  2. 2024FactAlign: Long-form Factuality Alignment of Large Language Models
  3. 2025AlignCheck: a Semantic Open-Domain Metric for Factual Consistency Assessment
  4. 2025Incentive-Aligned Multi-Source LLM Summaries
  5. 2026Beyond Shapley: An Influence-Based Data Auditing Pipeline for LLM Alignment and Evaluation
  6. 2026FAITH: Factuality Alignment through Integrating Trustworthiness and Honestness
  7. 2023LLMs as Factual Reasoners: Insights from Existing Benchmarks and Beyond
  8. 2025Can LLMs extract human-like fine-grained evidence for evidence-based fact-checking?