mlsanityData to 14 Oct 2026

HaRiM^+: Evaluating Summary Quality with Hallucination Risk

arXiv 2211.12118 ↗First posted 22 Nov 2022cs.CLcited by 6

Not on the map. Why?

Abstract (arXiv)

One of the challenges of developing a summarization model arises from the difficulty in measuring the factual inconsistency of the generated text. In this study, we reinterpret the decoder overconfidence-regularizing objective suggested in (Miao et al., 2021) as a hallucination risk measurement to better estimate the quality of generated summaries. We propose a reference-free metric, HaRiM+, which only requires an off-the-shelf summarization model to compute the hallucination risk based on token likelihoods. Deploying it requires no additional training of models or ad-hoc modules, which usually need alignment to human judgments. For summary-quality estimation, HaRiM+ records state-of-the-art correlation to human judgment on three summary-quality annotation sets: FRANK, QAGS, and SummEval. We hope that our work, which merits the use of summarization models, facilitates the progress of both automated evaluation and generation of summary.

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Builds on 36 of its 50 references that are in the map. First posted on arXiv 22 Nov 2022.

Builds on

36 of 50 references.

References · 36

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  4. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,893
  5. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,512
  6. 2014Sequence to Sequence Learning with Neural Networkscited by 2,986
  7. 2018Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarizationcited by 1,317
  8. 2019PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarizationcited by 1,102
  9. 2020Multilingual Denoising Pre-training for Neural Machine Translationcited by 1,035
  10. 2020On Faithfulness and Factuality in Abstractive Summarizationcited by 900
  11. 2016Abstractive Text Summarization Using Sequence-to-Sequence RNNs and Beyondcited by 812
  12. 2021BARTScore: Evaluating Generated Text as Text Generationcited by 722
24 more
  1. 2020SummEval: Re-evaluating Summarization Evaluationcited by 674
  2. 2015Generating Sentences from a Continuous Spacecited by 612
  3. 2019Evaluating the Factual Consistency of Abstractive Text Summarizationcited by 584
  4. 2019Text Summarization with Pretrained Encoderscited by 556
  5. 2020Asking and Answering Questions to Evaluate the Factual Consistency of Summariescited by 378
  6. 2020FEQA: A Question Answering Evaluation Framework for Faithfulness Assessment in Abstractive Summarizationcited by 293
  7. 2021Understanding Factuality in Abstractive Summarization with FRANK: A Benchmark for Factuality Metricscited by 287
  8. 2021QuestEval: Summarization Asks for Fact-based Evaluationcited by 244
  9. 2017Why We Need New Evaluation Metrics for NLGcited by 233
  10. 2018Bottom-Up Abstractive Summarizationcited by 177
  11. 2022BRIO: Bringing Order to Abstractive Summarizationcited by 161
  12. 2020Detecting Hallucinated Content in Conditional Neural Sequence Generationcited by 152
  13. 2020Evaluating Factuality in Generation with Dependency-level Entailmentcited by 122
  14. 2018Learning to Write with Cooperative Discriminatorscited by 119
  15. 2020Automatic Machine Translation Evaluation in Many Languages via Zero-Shot Paraphrasingcited by 115
  16. 2019Answers Unite! Unsupervised Metrics for Reinforced Summarization Modelscited by 92
  17. 2020Towards Question-Answering as an Automatic Metric for Evaluating the Content Quality of a Summarycited by 88
  18. 2021A Statistical Analysis of Summarization Evaluation Metrics using Resampling Methodscited by 72
  19. 2021Factual Consistency Evaluation for Text Summarization via Counterfactual Estimationcited by 37
  20. 2021Prevent the Language Model from being Overconfident in Neural Machine Translationcited by 34
  21. 2021Bringing Structure into Summaries: a Faceted Summarization Dataset for Long Scientific Documentscited by 33
  22. 2021Conditional Adversarial Networks for Multi-Domain Text Classificationcited by 16
  23. 2022Conditional Bilingual Mutual Information Based Adaptive Training for Neural Machine Translationcited by 6
  24. 2022MaskEval: Weighted MLM-Based Evaluation for Text Summarization and Simplificationcited by 3

Built on by

Cited by year

Citing papers per year: 2023: 1; 2024: 5.

6 papers.

  1. 2024Hallucination Detection and Hallucination Mitigation: An Investigationcited by 21
  2. 2024An Audit on the Perspectives and Challenges of Hallucinations in NLPcited by 17
  3. 2023Investigating Hallucinations in Pruned Large Language Models for Abstractive Summarizationcited by 16
  4. 2024Fake Artificial Intelligence Generated Contents (FAIGC): A Survey of Theories, Detection Methods, and Opportunitiescited by 8
  5. 2024A Survey of Automatic Hallucination Evaluation on Natural Language Generationcited by 7
  6. 2024Evaluation of Large Language Models for Summarization Tasks in the Medical Domain: A Narrative Reviewcited by 3