mlsanityData to 14 Oct 2026

Text Summarization with Pretrained Encoders

arXiv 1908.08345 ↗First posted 22 Aug 2019cs.CL cs.LGcited by 556

Not on the map. Why?

Abstract (arXiv)

Bidirectional Encoder Representations from Transformers (BERT) represents the latest incarnation of pretrained language models which have recently advanced a wide range of natural language processing tasks. In this paper, we showcase how BERT can be usefully applied in text summarization and propose a general framework for both extractive and abstractive models. We introduce a novel document-level encoder based on BERT which is able to express the semantics of a document and obtain representations for its sentences. Our extractive model is built on top of this encoder by stacking several inter-sentence Transformer layers. For abstractive summarization, we propose a new fine-tuning schedule which adopts different optimizers for the encoder and the decoder as a means of alleviating the mismatch between the two (the former is pretrained while the latter is not). We also demonstrate that a two-staged fine-tuning approach can further boost the quality of the generated summaries. Experiments on three datasets show that our model achieves state-of-the-art results across the board in both extractive and abstractive settings. Our code is available at https://github.com/nlpyang/PreSumm

Key facts. Cited by 556 papers in the map (data through 14 Oct 2026). First posted on arXiv 22 Aug 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2020: 1; 2021: 174; 2022: 129; 2023: 115; 2024: 81; 2025: 37; 2026: 19.

556 papers.

Sort:

By area:

  1. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,915
  2. 2021GLM: General Language Model Pretraining with Autoregressive Blank Infillingcited by 1,031
  3. 2023BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Datasetcited by 860
  4. 2021BARTScore: Evaluating Generated Text as Text Generationcited by 722
  5. 2021Recent Advances in Natural Language Processing via Large Pre-Trained Language Models: A Surveycited by 463
  6. 2023Benchmarking Large Language Models for News Summarizationcited by 445
  7. 2021QMSum: A New Benchmark for Query-based Multi-domain Meeting Summarizationcited by 365
  8. 2021XL-Sum: Large-Scale Multilingual Abstractive Summarization for 44 Languagescited by 347
  9. 2021Efficient Attentions for Long Document Summarizationcited by 345
  10. 2021Recursively Summarizing Books with Human Feedbackcited by 290
  11. 2021Understanding Factuality in Abstractive Summarization with FRANK: A Benchmark for Factuality Metricscited by 287
  12. 2021An Empirical Study of Training End-to-End Vision-and-Language Transformerscited by 271
38 more
  1. 2021PTR: Prompt Tuning with Rules for Text Classificationcited by 238
  2. 2021QAFactEval: Improved QA-Based Factual Consistency Evaluation for Summarizationcited by 236
  3. 2023Large Language Models on Graphs: A Comprehensive Surveycited by 192
  4. 2021BookSum: A Collection of Datasets for Long-form Narrative Summarizationcited by 184
  5. 2021DialogSum: A Real-Life Scenario Dialogue Summarization Datasetcited by 183
  6. 2022Truncation Sampling as Language Model Desmoothingcited by 123
  7. 2022Pretrained Language Models for Text Generation: A Surveycited by 122
  8. 2021CLIFF: Contrastive Learning for Improving Faithfulness and Factuality in Abstractive Summarizationcited by 120
  9. 2023Do LLMs Understand User Preferences? Evaluating LLMs On User Rating Predictioncited by 116
  10. 2023Unbiased Watermark for Large Language Modelscited by 111
  11. 2022Understanding Factual Errors in Summarization: Errors, Summarizers, Datasets, Error Detectorscited by 104
  12. 2023Lift Yourself Up: Retrieval-augmented Text Generation with Self Memorycited by 103
  13. 2024A Comprehensive Survey on Process-Oriented Automatic Text Summarization with Exploration of LLM-Based Methodscited by 102
  14. 2021Pretrained Language Models for Text Generation: A Surveycited by 95
  15. 2022Learning to Break the Loop: Analyzing and Mitigating Repetitions for Neural Text Generationcited by 94
  16. 2022BioBART: Pretraining and Evaluation of A Biomedical Generative Language Modelcited by 83
  17. 2021Improving Faithfulness in Abstractive Summarization with Contrast Candidate Generation and Selectioncited by 76
  18. 2023ChatGPT as a Factual Inconsistency Evaluator for Text Summarizationcited by 75
  19. 2020Autoregressive Knowledge Distillation through Imitation Learningcited by 74
  20. 2022An Empirical Survey on Long Document Summarization: Datasets, Models and Metricscited by 74
  21. 2021DialogLM: Pre-trained Model for Long Dialogue Understanding and Summarizationcited by 72
  22. 2021DOC2PPT: Automatic Presentation Slides Generation from Scientific Documentscited by 70
  23. 2023Align and Attend: Multimodal Summarization with Dual Contrastive Lossescited by 65
  24. 2021Prompt-Learning for Fine-Grained Entity Typingcited by 62
  25. 2021PRIMERA: Pyramid-based Masked Sentence Pre-training for Multi-document Summarizationcited by 62
  26. 2022ECTSum: A New Benchmark Dataset For Bullet Point Summarization of Long Earnings Call Transcriptscited by 59
  27. 2023Extractive Summarization via ChatGPT for Faithful Summary Generationcited by 59
  28. 2023An Iterative Optimizing Framework for Radiology Report Summarization with ChatGPTcited by 57
  29. 2021SpanNER: Named Entity Re-/Recognition as Span Predictioncited by 55
  30. 2023FactKB: Generalizable Factuality Evaluation using Language Models Enhanced with Factual Knowledgecited by 55
  31. 2023SummIt: Iterative Text Summarization via ChatGPTcited by 54
  32. 2023Unveiling Security, Privacy, and Ethical Concerns of ChatGPTcited by 53
  33. 2024A Systematic Survey of Text Summarization: From Statistical Methods to Large Language Modelscited by 52
  34. 2021AdaptSum: Towards Low-Resource Domain Adaptation for Abstractive Summarizationcited by 51
  35. 2021A Survey on Dialogue Summarization: Recent Advances and New Frontierscited by 51
  36. 2021ExplainaBoard: An Explainable Leaderboard for NLPcited by 50
  37. 2021D2S: Document-to-Slide Generation Via Query-Based Text Summarizationcited by 50
  38. 2021AutoFreeze: Automatically Freezing Model Blocks to Accelerate Fine-tuningcited by 48

50 most cited of 556 All 556 (JSON)