mlsanityData to 14 Oct 2026

PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization

arXiv 1912.08777 ↗First posted 18 Dec 2019cs.CLcited by 1,102

Not on the map. Why?

Cited by 23 in LLM safety and security, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Recent work pre-training Transformers with self-supervised objectives on large text corpora has shown great success when fine-tuned on downstream NLP tasks including text summarization. However, pre-training objectives tailored for abstractive text summarization have not been explored. Furthermore there is a lack of systematic evaluation across diverse domains. In this work, we propose pre-training large Transformer-based encoder-decoder models on massive text corpora with a new self-supervised objective. In PEGASUS, important sentences are removed/masked from an input document and are generated together as one output sequence from the remaining sentences, similar to an extractive summary. We evaluated our best PEGASUS model on 12 downstream summarization tasks spanning news, science, stories, instructions, emails, patents, and legislative bills. Experiments demonstrate it achieves state-of-the-art performance on all 12 downstream datasets measured by ROUGE scores. Our model also shows surprising performance on low-resource summarization, surpassing previous state-of-the-art results on 6 datasets with only 1000 examples. Finally we validated our results using human evaluation and show that our model summaries achieve human performance on multiple datasets.

Key facts. Cited by 1,102 papers in the map (data through 14 Oct 2026). First posted on arXiv 18 Dec 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2020: 2; 2021: 192; 2022: 260; 2023: 269; 2024: 194; 2025: 121; 2026: 64.

1,102 papers.

Sort:

By area:

  1. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  2. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,975
  3. 2020Learning to summarize from human feedbackcited by 2,588
  4. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,234
  5. 2023A Survey on Hallucination in Large Language Models: Principles, Taxonomy, Challenges, and Open Questionscited by 1,601
  6. 2021Conceptual 12M: Pushing Web-Scale Image-Text Pre-Training To Recognize Long-Tail Visual Conceptscited by 1,216
  7. 2021GLM: General Language Model Pretraining with Autoregressive Blank Infillingcited by 1,031
  8. 2022OFA: Unifying Architectures, Tasks, and Modalities Through a Simple Sequence-to-Sequence Learning Frameworkcited by 745
  9. 2021BARTScore: Evaluating Generated Text as Text Generationcited by 722
  10. 2023LLM-Blender: Ensembling Large Language Models with Pairwise Ranking and Generative Fusioncited by 564
  11. 2021ByT5: Towards a token-free future with pre-trained byte-to-byte modelscited by 509
  12. 2023Benchmarking Large Language Models for News Summarizationcited by 445
38 more
  1. 2023Harnessing the Power of LLMs in Practice: A Survey on ChatGPT and Beyondcited by 411
  2. 2021SummaC: Re-Visiting NLI-based Models for Inconsistency Detection in Summarizationcited by 407
  3. 2022ST-MoE: Designing Stable and Transferable Sparse Expert Modelscited by 399
  4. 2023Can AI-Generated Text be Reliably Detected?cited by 386
  5. 2023A Real-World WebAgent with Planning, Long Context Understanding, and Program Synthesiscited by 353
  6. 2021XL-Sum: Large-Scale Multilingual Abstractive Summarization for 44 Languagescited by 347
  7. 2021Efficient Attentions for Long Document Summarizationcited by 345
  8. 2022News Summarization and Evaluation in the Era of GPT-3cited by 327
  9. 2021Pre-Trained Models: Past, Present and Futurecited by 319
  10. 2023OBELICS: An Open Web-Scale Filtered Dataset of Interleaved Image-Text Documentscited by 316
  11. 2021All That's 'Human' Is Not Gold: Evaluating Human Evaluation of Generated Textcited by 287
  12. 2023From Pretraining Data to Language Models to Downstream Tasks: Tracking the Trails of Political Biases Leading to Unfair NLP Modelscited by 260
  13. 2023A Survey of Confidence Estimation and Calibration in Large Language Modelscited by 256
  14. 2021QuestEval: Summarization Asks for Fact-based Evaluationcited by 244
  15. 2021The GEM Benchmark: Natural Language Generation, its Evaluation and Metricscited by 239
  16. 2021PTR: Prompt Tuning with Rules for Text Classificationcited by 238
  17. 2021QAFactEval: Improved QA-Based Factual Consistency Evaluation for Summarizationcited by 236
  18. 2021LongT5: Efficient Text-To-Text Transformer for Long Sequencescited by 232
  19. 2024Many-Shot In-Context Learningcited by 206
  20. 2021R-Drop: Regularized Dropout for Neural Networkscited by 202
  21. 2022Out-of-Distribution Detection and Selective Generation for Conditional Language Modelscited by 197
  22. 2021Want To Reduce Labeling Cost? GPT-3 Can Helpcited by 196
  23. 2022Co-Writing Screenplays and Theatre Scripts with Language Models: An Evaluation by Industry Professionalscited by 191
  24. 2021BookSum: A Collection of Datasets for Long-form Narrative Summarizationcited by 184
  25. 2022PromptCast: A New Prompt-based Learning Paradigm for Time Series Forecastingcited by 172
  26. 2023AnnoLLM: Making Large Language Models to Be Better Crowdsourced Annotatorscited by 172
  27. 2022BRIO: Bringing Order to Abstractive Summarizationcited by 161
  28. 2023mPLUG-2: A Modularized Multi-modal Foundation Model Across Text, Image and Videocited by 154
  29. 2022Revisiting the Gold Standard: Grounding Summarization Evaluation with Robust Human Evaluationcited by 142
  30. 2021SimCLS: A Simple Framework for Contrastive Learning of Abstractive Summarizationcited by 132
  31. 2021Annotating and Modeling Fine-grained Factuality in Summarizationcited by 131
  32. 2021AMMUS : A Survey of Transformer-based Pretrained Models in Natural Language Processingcited by 131
  33. 2021Hallucinated but Factual! Inspecting the Factuality of Hallucinations in Abstractive Summarizationcited by 129
  34. 2022GrIPS: Gradient-free, Edit-based Instruction Search for Prompting Large Language Modelscited by 127
  35. 2022Pretrained Language Models for Text Generation: A Surveycited by 122
  36. 2022Calibrating Sequence likelihood Improves Conditional Language Generationcited by 121
  37. 2021CLIFF: Contrastive Learning for Improving Faithfulness and Factuality in Abstractive Summarizationcited by 120
  38. 2021Entity-level Factual Consistency of Abstractive Text Summarizationcited by 118

50 most cited of 1,102 All 1,102 (JSON)