mlsanityData to 13 Oct 2026

Zero-shot Faithfulness Evaluation for Text Summarization with Foundation Language Model

arXiv 2310.11648 ↗First posted 18 Oct 2023cs.CLcited by 21

Not on the map. Why?

Abstract (arXiv)

Despite tremendous improvements in natural language generation, summarization models still suffer from the unfaithfulness issue. Previous work evaluates faithfulness either using models trained on the other tasks or in-domain synthetic data, or prompting a large model such as ChatGPT. This paper proposes to do zero-shot faithfulness evaluation simply with a moderately-sized foundation language model. We introduce a new metric FFLM, which is a combination of probability changes based on the intuition that prefixing a piece of text that is consistent with the output will increase the probability of predicting the output. Experiments show that FFLM performs competitively with or even outperforms ChatGPT on both inconsistency detection and faithfulness rating with 24x fewer parameters. FFLM also achieves improvements over other strong baselines.

Key facts. Cited by 21 papers in the map (data through 13 Oct 2026). Builds on 36 of its 54 references that are in the map. First posted on arXiv 18 Oct 2023.

Builds on

36 of 54 references.

References · 36

  1. 2020Language Models are Few-Shot Learnerscited by 32,320
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,072
  3. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,884
  4. 2019BERTScore: Evaluating Text Generation with BERTcited by 5,480
  5. 2018Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarizationcited by 1,312
  6. 2019PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarizationcited by 1,100
  7. 2023LIMA: Less Is More for Alignmentcited by 1,091
  8. 2020StereoSet: Measuring stereotypical bias in pretrained language modelscited by 918
  9. 2016Abstractive Text Summarization Using Sequence-to-Sequence RNNs and Beyondcited by 810
  10. 2021BARTScore: Evaluating Generated Text as Text Generationcited by 720
  11. 2020SummEval: Re-evaluating Summarization Evaluationcited by 668
  12. 2019Evaluating the Factual Consistency of Abstractive Text Summarizationcited by 580
24 more
  1. 2023Benchmarking Large Language Models for News Summarizationcited by 444
  2. 2023Is ChatGPT a Good NLG Evaluator? A Preliminary Studycited by 421
  3. 2021SummaC: Re-Visiting NLI-based Models for Inconsistency Detection in Summarizationcited by 404
  4. 2019PAWS: Paraphrase Adversaries from Word Scramblingcited by 400
  5. 2020Asking and Answering Questions to Evaluate the Factual Consistency of Summariescited by 377
  6. 2020FEQA: A Question Answering Evaluation Framework for Faithfulness Assessment in Abstractive Summarizationcited by 292
  7. 2021Understanding Factuality in Abstractive Summarization with FRANK: A Benchmark for Factuality Metricscited by 285
  8. 2021QuestEval: Summarization Asks for Fact-based Evaluationcited by 241
  9. 2023Human-like Summarization Evaluation with ChatGPTcited by 126
  10. 2020Evaluating Factuality in Generation with Dependency-level Entailmentcited by 122
  11. 2022FaithDial: A Faithful Benchmark for Information-Seeking Dialoguecited by 108
  12. 2019Assessing The Factual Accuracy of Generated Textcited by 103
  13. 2020Improved Natural Language Generation via Loss Truncationcited by 75
  14. 2021DocNLI: A Large-scale Dataset for Document-level Natural Language Inferencecited by 75
  15. 2020What Have We Achieved on Text Summarization?cited by 65
  16. 2021Keep it Simple: Unsupervised Simplification of Multi-Paragraph Textcited by 45
  17. 2021Factual Consistency Evaluation for Text Summarization via Counterfactual Estimationcited by 37
  18. 2021Prevent the Language Model from being Overconfident in Neural Machine Translationcited by 34
  19. 2022Falsesum: Generating Document-level NLI Examples for Recognizing Factual Inconsistency in Summarizationcited by 34
  20. 2023A Trip Towards Fairness: Bias and De-Biasing in Large Language Modelscited by 32
  21. 2021Training Dynamics for Text Summarization Modelscited by 25
  22. 2020Improving Truthfulness of Headline Generationcited by 20
  23. 2021Play the Shannon Game With Language Models: A Human-Free Approach to Summary Evaluationcited by 12
  24. 2022CoP: Factual Inconsistency Detection by Controlling the Preferencecited by 4

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2024: 9; 2025: 9; 2026: 3.

21 papers.

Sort:

By area:

  1. 2024A Comprehensive Survey on Process-Oriented Automatic Text Summarization with Exploration of LLM-Based Methodscited by 101
  2. 2024FaithEval: Can Your Language Model Stay Faithful to Context, Even If "The Moon is Made of Marshmallows"cited by 86
  3. 2024LLM-based NLG Evaluation: Current Status and Challengescited by 72
  4. 2025Can LLMs Replace Human Evaluators? An Empirical Study of LLM-as-a-Judge in Software Engineeringcited by 67
  5. 2024A Systematic Survey of Text Summarization: From Statistical Methods to Large Language Modelscited by 52
  6. 2024Leveraging Large Language Models for NLG Evaluation: Advances and Challengescited by 37
  7. 2025R3: Robust Rubric-Agnostic Reward Modelscited by 22
  8. 2025Benchmarking LLM Faithfulness in RAG with Evolving Leaderboardscited by 20
  9. 2025Support Evaluation for the TREC 2024 RAG Track: Comparing Human versus LLM Judgescited by 14
  10. 2025Sequential-NIAH: A Needle-In-A-Haystack Benchmark for Extracting Sequential Needles from Long Contextscited by 9
  11. 2024A Survey of Automatic Hallucination Evaluation on Natural Language Generationcited by 7
  12. 2024Large Language Models Are Active Critics in NLG Evaluationcited by 6
9 more
  1. 2024Language Models can Evaluate Themselves via Probability Discrepancycited by 5
  2. 2026Who Gets Which Message? Auditing Demographic Bias in LLM-Generated Targeted Textcited by 3
  3. 2024Less is More for Improving Automatic Evaluation of Factual Consistencycited by 2
  4. 2025Mind the Style Gap: Meta-Evaluation of Style and Attribute Transfer Metricscited by 2
  5. 2025LecEval: An Automated Metric for Multimodal Knowledge Acquisition in Multimedia Learningcited by 1
  6. 2025BHRAM-IL: A Benchmark for Hallucination Recognition and Assessment in Multiple Indian Languagescited by 1
  7. 2026Detecting Hallucinations in Retrieval-Augmented Generation through Grounding-Aware Sensitivity by Perturbation (GASP)cited by 1
  8. 2025From Ground Trust to Truth: Disparities in Offensive Language Judgments on Contemporary Korean Political Discoursecited by 0
  9. 2026On Improving Faithfulness of Podcasts from Documentscited by 0