mlsanityData to 7 Oct 2026

Don't Say What You Don't Know: Improving the Consistency of Abstractive Summarization by Constraining Beam Search

arXiv 2203.08436 ↗First posted 16 Mar 2022cs.CLcited by 23

Not on the map. Why?

Abstract (arXiv)

Abstractive summarization systems today produce fluent and relevant output, but often "hallucinate" statements not supported by the source text. We analyze the connection between hallucinations and training data, and find evidence that models hallucinate because they train on target summaries that are unsupported by the source. Based on our findings, we present PINOCCHIO, a new decoding method that improves the consistency of a transformer-based abstractive summarizer by constraining beam search to avoid hallucinations. Given the model states and outputs at a given step, PINOCCHIO detects likely model hallucinations based on various measures of attribution to the source text. PINOCCHIO backtracks to find more consistent output, and can opt to produce no summary at all when no consistent generation can be found. In experiments, we find that PINOCCHIO improves the consistency of generation (in terms of F1) by an average of~67% on two abstractive summarization datasets.

Key facts. Cited by 23 papers in the map (data through 7 Oct 2026). Builds on 37 of its 47 references that are in the map. First posted on arXiv 16 Mar 2022.

Builds on

37 of 47 references.

References · 37

  1. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  2. 2014Sequence to Sequence Learning with Neural Networkscited by 2,974
  3. 2017Decoupled Weight Decay Regularizationcited by 1,722
  4. 2016Google's Neural Machine Translation System: Bridging the Gap between Human and Machine Translationcited by 1,471
  5. 2019SciBERT: A Pretrained Language Model for Scientific Textcited by 1,404
  6. 2018Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarizationcited by 1,308
  7. 2017Get To The Point: Summarization with Pointer-Generator Networkscited by 1,281
  8. 2019PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarizationcited by 1,100
  9. 2015Teaching Machines to Read and Comprehendcited by 1,039
  10. 2020On Faithfulness and Factuality in Abstractive Summarizationcited by 895
  11. 2016Abstractive Text Summarization Using Sequence-to-Sequence RNNs and Beyondcited by 809
  12. 2019Unified Language Model Pre-training for Natural Language Understanding and Generationcited by 668
25 more
  1. 2020SummEval: Re-evaluating Summarization Evaluationcited by 666
  2. 2019Evaluating the Factual Consistency of Abstractive Text Summarizationcited by 578
  3. 2018A Discourse-Aware Attention Model for Abstractive Summarization of Long Documentscited by 477
  4. 2020Asking and Answering Questions to Evaluate the Factual Consistency of Summariescited by 376
  5. 2018Generating Wikipedia by Summarizing Long Sequencescited by 333
  6. 2020FEQA: A Question Answering Evaluation Framework for Faithfulness Assessment in Abstractive Summarizationcited by 292
  7. 2021Understanding Factuality in Abstractive Summarization with FRANK: A Benchmark for Factuality Metricscited by 285
  8. 2017Lexically Constrained Decoding for Sequence Generation Using Grid Beam Searchcited by 234
  9. 2019BIGPATENT: A Large-Scale Dataset for Abstractive and Coherent Summarizationcited by 155
  10. 2019Hierarchical Transformers for Multi-Document Summarizationcited by 132
  11. 2021Entity-level Factual Consistency of Abstractive Text Summarizationcited by 118
  12. 2019Optimizing the Factual Correctness of a Summary: A Study of Summarizing Radiology Reportscited by 96
  13. 2020Multi-Fact Correction in Abstractive Text Summarizationcited by 66
  14. 2020What Have We Achieved on Text Summarization?cited by 65
  15. 2019Sticking to the Facts: Confident Decoding for Faithful Data-to-Text Generationcited by 62
  16. 2020Towards Faithful Neural Table-to-Text Generation with Content-Matching Constraintscited by 48
  17. 2020Understanding Neural Abstractive Summarization Models via Uncertaintycited by 34
  18. 2020Constrained Abstractive Summarization: Preserving Factual Consistency with Constrained Generationcited by 26
  19. 2020Improving Truthfulness of Headline Generationcited by 20
  20. 2019Controlling the Amount of Verbatim Copying in Abstractive Summarizationcited by 17
  21. 2019Can Unconditional Language Models Recover Arbitrary Sentences?cited by 15
  22. 2020Adversarial NLI for Factual Correctness in Text Summarisation Modelscited by 13
  23. 2020Discovering Useful Sentence Representations from Large Pretrained Language Modelscited by 6
  24. 2020Looking Beyond Sentence-Level Natural Language Inference for Downstream Taskscited by 6
  25. 2020High-Precision Extraction of Emerging Concepts from Scientific Literaturecited by 5

Built on by

Cited by year

Citing papers per year: 2022: 6; 2023: 10; 2024: 6; 2025: 1.

23 papers.

Sort:

By area:

  1. 2023Trusting Your Evidence: Hallucinate Less with Context-aware Decodingcited by 319
  2. 2022Language Generation Models Can Cause Harm: So What Can We Do About It? An Actionable Surveycited by 80
  3. 2022ECTSum: A New Benchmark Dataset For Bullet Point Summarization of Long Earnings Call Transcriptscited by 59
  4. 2022Multi-LexSum: Real-World Summaries of Civil Rights Lawsuits at Multiple Granularitiescited by 47
  5. 2022Correcting Diverse Factual Errors in Abstractive Summarization via Post-Editing and Language Model Infillingcited by 38
  6. 2023Beyond Summarization: Designing AI Support for Real-World Expository Writing Taskscited by 20
  7. 2023Investigating Hallucinations in Pruned Large Language Models for Abstractive Summarizationcited by 16
  8. 2024Cheetah: Natural Language Generation for 517 African Languagescited by 15
  9. 2022Entity-based SpanCopy for Abstractive Summarization to Improve the Factual Consistencycited by 11
  10. 2022Improved Beam Search for Hallucination Mitigation in Abstractive Summarizationcited by 10
  11. 2023Expository Text Generation: Imitate, Retrieve, Paraphrasecited by 8
  12. 2023Revisiting the Architectures like Pointer Networks to Efficiently Improve the Next Word Distribution, Summarization Factuality, and Beyondcited by 7
11 more
  1. 2024Model-based Preference Optimization in Abstractive Summarization without Human Feedbackcited by 7
  2. 2023A Method to Automate the Discharge Summary Hospital Course for Neurology Patientscited by 6
  3. 2024Mitigating Hallucination in Abstractive Summarization with Domain-Conditional Mutual Informationcited by 6
  4. 2023Parameter Efficient Audio Captioning With Faithful Guidance Using Audio-text Shared Latent Representationcited by 4
  5. 2023Correction with Backtracking Reduces Hallucination in Summarizationcited by 4
  6. 2023Medical Text Simplification: Optimizing for Readability with Unlikelihood Training and Reranked Beam Search Decodingcited by 3
  7. 2023On the Depth between Beam Search and Exhaustive Search for Text Generationcited by 2
  8. 2024On the Benefits of Fine-Grained Loss Truncation: A Case Study on Factuality in Summarizationcited by 2
  9. 2024Entity-level Factual Adaptiveness of Fine-tuning based Abstractive Summarization Modelscited by 1
  10. 2025Learning from Self Critique and Refinement for Faithful LLM Summarizationcited by 1
  11. 2024TOPICAL: TOPIC Pages AutomagicaLlycited by 0