mlsanityData to 7 Oct 2026

Zero-shot Factual Consistency Evaluation Across Domains

arXiv 2408.04114 ↗First posted 7 Aug 2024cs.CL cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

This work addresses the challenge of factual consistency in text generation systems. We unify the tasks of Natural Language Inference, Summarization Evaluation, Factuality Verification and Factual Consistency Evaluation to train models capable of evaluating the factual consistency of source-target pairs across diverse domains. We rigorously evaluate these against eight baselines on a comprehensive benchmark suite comprising 22 datasets that span various tasks, domains, and document lengths. Results demonstrate that our method achieves state-of-the-art performance on this heterogeneous benchmark while addressing efficiency concerns and attaining cross-domain generalization.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 53 of its 65 references that are in the map. First posted on arXiv 7 Aug 2024.

Builds on

53 of 65 references.

References · 53

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  5. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  6. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  7. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,462
  8. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  9. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  10. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
  11. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,140
  12. 2015A large annotated corpus for learning natural language inferencecited by 1,837
38 more
  1. 2018Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarizationcited by 1,308
  2. 2023FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generationcited by 1,259
  3. 2015Teaching Machines to Read and Comprehendcited by 1,039
  4. 2020On Faithfulness and Factuality in Abstractive Summarizationcited by 895
  5. 2023A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivitycited by 888
  6. 2019Adversarial NLI: A New Benchmark for Natural Language Understandingcited by 798
  7. 2021BARTScore: Evaluating Generated Text as Text Generationcited by 719
  8. 2020SummEval: Re-evaluating Summarization Evaluationcited by 666
  9. 2019SAMSum Corpus: A Human-annotated Dialogue Dataset for Abstractive Summarizationcited by 530
  10. 2023Benchmarking Large Language Models for News Summarizationcited by 444
  11. 2020Dataset Cartography: Mapping and Diagnosing Datasets with Training Dynamicscited by 423
  12. 2021SummaC: Re-Visiting NLI-based Models for Inconsistency Detection in Summarizationcited by 402
  13. 2021QMSum: A New Benchmark for Query-based Multi-domain Meeting Summarizationcited by 363
  14. 2021XL-Sum: Large-Scale Multilingual Abstractive Summarization for 44 Languagescited by 346
  15. 2023HaluEval: A Large-Scale Hallucination Evaluation Benchmark for Large Language Modelscited by 329
  16. 2023Evaluating Verifiability in Generative Search Enginescited by 297
  17. 2023AlignScore: Evaluating Factual Consistency with a Unified Alignment Functioncited by 284
  18. 2024MiniCheck: Efficient Fact-Checking of LLMs on Grounding Documentscited by 251
  19. 2021QuestEval: Summarization Asks for Fact-based Evaluationcited by 241
  20. 2021QAFactEval: Improved QA-Based Factual Consistency Evaluation for Summarizationcited by 233
  21. 2024Leak, Cheat, Repeat: Data Contamination and Evaluation Malpractices in Closed-Source LLMscited by 224
  22. 2022WANLI: Worker and AI Collaboration for Natural Language Inference Dataset Creationcited by 200
  23. 2020WikiLingua: A New Benchmark Dataset for Cross-Lingual Abstractive Summarizationcited by 154
  24. 2020MLSUM: The Multilingual Summarization Corpuscited by 121
  25. 2023WiCE: Real-World Entailment for Claims in Wikipediacited by 118
  26. 2022Understanding Factual Errors in Summarization: Errors, Summarizers, Datasets, Error Detectorscited by 103
  27. 2023ExpertQA: Expert-Curated Questions and Attributed Answerscited by 96
  28. 2022Evaluating the Factual Consistency of Large Language Models Through News Summarizationcited by 90
  29. 2023Factcheck-Bench: Fine-Grained Evaluation Benchmark for Automatic Fact-checkerscited by 86
  30. 2023MeetingBank: A Benchmark Dataset for Meeting Summarizationcited by 81
  31. 2023Benchmarking Generation and Evaluation Capabilities of Large Language Models for Instruction Controllable Summarizationcited by 69
  32. 2024TofuEval: Evaluating Hallucinations of LLMs on Topic-Focused Dialogue Summarizationcited by 66
  33. 2024A Chain-of-Thought Is as Strong as Its Weakest Link: A Benchmark for Verifiers of Reasoning Chainscited by 63
  34. 2021Fool Me Twice: Entailment from Wikipedia Gamificationcited by 50
  35. 2023Understanding Retrieval Augmentation for Long-Form Question Answeringcited by 45
  36. 2023SEAHORSE: A Multilingual, Multifaceted Dataset for Summarization Evaluationcited by 42
  37. 2022On Improving Summarization Factual Consistency from Natural Language Feedbackcited by 38
  38. 2021Does Putting a Linguist in the Loop Improve NLU Data Collection?cited by 29

50 most cited of 53 All 53 (JSON)

Built on by

Not cited yet.