# Faithfulness-Aware Decoding Strategies for Abstractive Summarization

arXiv: https://arxiv.org/abs/2303.03278 · HTML: https://mlsanity.com/paper/2303.03278/

## Key facts

- Cited by 31 papers in the map (data through 14 Oct 2026).
- Builds on 38 of its 47 references that are in the map.
- First posted on arXiv 6 Mar 2023.

## Builds on (most cited first, 30 of 38)

- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [BERTScore: Evaluating Text Generation with BERT](https://mlsanity.com/paper/1904.09675/) (2019)
- [The Curious Case of Neural Text Degeneration](https://mlsanity.com/paper/1904.09751/) (2019)
- [Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarization](https://mlsanity.com/paper/1808.08745/) (2018)
- [PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization](https://mlsanity.com/paper/1912.08777/) (2019)
- [Teaching Machines to Read and Comprehend](https://mlsanity.com/paper/1506.03340/) (2015)
- [On Faithfulness and Factuality in Abstractive Summarization](https://mlsanity.com/paper/2005.00661/) (2020)
- [Sequence-Level Knowledge Distillation](https://mlsanity.com/paper/1606.07947/) (2016)
- [Evaluating the Factual Consistency of Abstractive Text Summarization](https://mlsanity.com/paper/1910.12840/) (2019)
- [Datasets: A Community Library for Natural Language Processing](https://mlsanity.com/paper/2109.02846/) (2021)
- [Understanding Factuality in Abstractive Summarization with FRANK: A Benchmark for Factuality Metrics](https://mlsanity.com/paper/2104.13346/) (2021)
- [QuestEval: Summarization Asks for Fact-based Evaluation](https://mlsanity.com/paper/2103.12693/) (2021)
- [WikiHow: A Large Scale Text Summarization Dataset](https://mlsanity.com/paper/1810.09305/) (2018)
- [NeuroLogic A*esque Decoding: Constrained Text Generation with Lookahead Heuristics](https://mlsanity.com/paper/2112.08726/) (2021)
- [SimCLS: A Simple Framework for Contrastive Learning of Abstractive Summarization](https://mlsanity.com/paper/2106.01890/) (2021)
- [Annotating and Modeling Fine-grained Factuality in Summarization](https://mlsanity.com/paper/2104.04302/) (2021)
- [CLIFF: Contrastive Learning for Improving Faithfulness and Factuality in Abstractive Summarization](https://mlsanity.com/paper/2109.09209/) (2021)
- [Entity-level Factual Consistency of Abstractive Text Summarization](https://mlsanity.com/paper/2102.09130/) (2021)
- [If beam search is the answer, what was the question?](https://mlsanity.com/paper/2010.02650/) (2020)
- [On NMT Search Errors and Model Errors: Cat Got Your Tongue?](https://mlsanity.com/paper/1908.10090/) (2019)
- [Best-Worst Scaling More Reliable than Rating Scales: A Case Study on Sentiment Intensity Annotation](https://mlsanity.com/paper/1712.01765/) (2017)
- [Improving Faithfulness in Abstractive Summarization with Contrast Candidate Generation and Selection](https://mlsanity.com/paper/2104.09061/) (2021)
- [Pre-trained Summarization Distillation](https://mlsanity.com/paper/2010.13002/) (2020)
- [Mutual Information Alleviates Hallucinations in Abstractive Summarization](https://mlsanity.com/paper/2210.13210/) (2022)
- [Multi-Fact Correction in Abstractive Text Summarization](https://mlsanity.com/paper/2010.02443/) (2020)
- [SummaReranker: A Multi-Task Mixture-of-Experts Re-ranking Framework for Abstractive Summarization](https://mlsanity.com/paper/2203.06569/) (2022)
- [Faithful or Extractive? On Mitigating the Faithfulness-Abstractiveness Trade-off in Abstractive Summarization](https://mlsanity.com/paper/2108.13684/) (2021)
- [FactPEGASUS: Factuality-Aware Pre-training and Fine-tuning for Abstractive Summarization](https://mlsanity.com/paper/2205.07830/) (2022)
- [FactGraph: Evaluating Factuality in Summarization with Semantic Graph Representations](https://mlsanity.com/paper/2204.06508/) (2022)

## Built on by (most cited first, 30 of 31)

- [Break the Sequential Dependency of LLM Inference Using Lookahead Decoding](https://mlsanity.com/paper/2402.02057/) (2024)
- [From Matching to Generation: A Survey on Generative Information Retrieval](https://mlsanity.com/paper/2404.14851/) (2024)
- [The Curious Case of Hallucinatory (Un)answerability: Finding Truths in the Hidden States of Over-Confident Large Language Models](https://mlsanity.com/paper/2310.11877/) (2023)
- [DeAL: Decoding-time Alignment for Large Language Models](https://mlsanity.com/paper/2402.06147/) (2024)
- [AI Hallucinations: A Misnomer Worth Clarifying](https://mlsanity.com/paper/2401.06796/) (2024)
- [Generating Summaries with Controllable Readability Levels](https://mlsanity.com/paper/2310.10623/) (2023)
- [Deconstructing The Ethics of Large Language Models from Long-standing Issues to New-emerging Dilemmas: A Survey](https://mlsanity.com/paper/2406.05392/) (2024)
- [Localizing Factual Inconsistencies in Attributable Text Generation](https://mlsanity.com/paper/2410.07473/) (2024)
- [REAL Sampling: Boosting Factuality and Diversity of Open-Ended Generation via Asymptotic Entropy](https://mlsanity.com/paper/2406.07735/) (2024)
- [Out of Style: RAG's Fragility to Linguistic Variation](https://mlsanity.com/paper/2504.08231/) (2025)
- [Read Your Own Mind: Reasoning Helps Surface Self-Confidence Signals in LLMs](https://mlsanity.com/paper/2505.23845/) (2025)
- [Key-Element-Informed sLLM Tuning for Document Summarization](https://mlsanity.com/paper/2406.04625/) (2024)
- [Neural Natural Language Processing for Long Texts: A Survey on Classification and Summarization](https://mlsanity.com/paper/2305.16259/) (2023)
- [Model-based Preference Optimization in Abstractive Summarization without Human Feedback](https://mlsanity.com/paper/2409.18618/) (2024)
- [HistAlign: Improving Context Dependency in Language Generation by Aligning with History](https://mlsanity.com/paper/2305.04782/) (2023)
- [Multi-Dimensional Optimization for Text Summarization via Reinforcement Learning](https://mlsanity.com/paper/2406.00303/) (2024)
- [Constrained Decoding with Speculative Lookaheads](https://mlsanity.com/paper/2412.10418/) (2024)
- [A Survey on Proactive Defense Strategies Against Misinformation in Large Language Models](https://mlsanity.com/paper/2507.05288/) (2025)
- [Analysis of Plan-based Retrieval for Grounded Text Generation](https://mlsanity.com/paper/2408.10490/) (2024)
- [Correction with Backtracking Reduces Hallucination in Summarization](https://mlsanity.com/paper/2310.16176/) (2023)
- [STRONG -- Structure Controllable Legal Opinion Summary Generation](https://mlsanity.com/paper/2309.17280/) (2023)
- [On the Depth between Beam Search and Exhaustive Search for Text Generation](https://mlsanity.com/paper/2308.13696/) (2023)
- [Open-domain Implicit Format Control for Large Language Model Generation](https://mlsanity.com/paper/2408.04392/) (2024)
- [Dual-Stage Value-Guided Inference with Margin-Based Reward Adjustment for Fast and Faithful VLM Captioning](https://mlsanity.com/paper/2506.15649/) (2025)
- [Dont Add, dont Miss: Effective Content Preserving Generation from Pre-Selected Text Spans](https://mlsanity.com/paper/2310.09017/) (2023)
- [HalluCana: Fixing LLM Hallucination with A Canary Lookahead](https://mlsanity.com/paper/2412.07965/) (2024)
- [PrefixNLI: Detecting Factual Inconsistencies as Soon as They Arise](https://mlsanity.com/paper/2511.01359/) (2025)
- [Selective Regenerative Decoding: Trajectory-Level Intervention for Inference-Time Reasoning](https://mlsanity.com/paper/2608.24338/) (2026)
- [Probabilistic Guarantees for Reducing Contextual Hallucinations in LLMs](https://mlsanity.com/paper/2601.00641/) (2026)
- [BLooP: Zero-Shot Abstractive Summarization using Large Language Models with Bigram Lookahead Promotion](https://mlsanity.com/paper/2603.11415/) (2026)

## Abstract (arXiv)

Despite significant progress in understanding and improving faithfulness in abstractive summarization, the question of how decoding strategies affect faithfulness is less studied. We present a systematic study of the effect of generation techniques such as beam search and nucleus sampling on faithfulness in abstractive summarization. We find a consistent trend where beam search with large beam sizes produces the most faithful summaries while nucleus sampling generates the least faithful ones. We propose two faithfulness-aware generation methods to further improve faithfulness over current generation techniques: (1) ranking candidates generated by beam search using automatic faithfulness metrics and (2) incorporating lookahead heuristics that produce a faithfulness score on the future summary. We show that both generation methods significantly improve faithfulness across two datasets as evaluated by four automatic faithfulness metrics and human evaluation. To reduce computational cost, we demonstrate a simple distillation approach that allows the model to generate faithful summaries with just greedy decoding. Our code is publicly available at https://github.com/amazon-science/faithful-summarization-generation