# Fine-tune BERT for Extractive Summarization

arXiv: https://arxiv.org/abs/1903.10318 · HTML: https://mlsanity.com/paper/1903.10318/

## Key facts

- Cited by 122 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 25 Mar 2019.

## Built on by (most cited first, 30 of 122)

- [Think before you speak: Training Language Models With Pause Tokens](https://mlsanity.com/paper/2310.02226/) (2023)
- [Language model compression with weighted low-rank factorization](https://mlsanity.com/paper/2207.00112/) (2022)
- [DialogSum: A Real-Life Scenario Dialogue Summarization Dataset](https://mlsanity.com/paper/2105.06762/) (2021)
- [A Comprehensive Survey on Process-Oriented Automatic Text Summarization with Exploration of LLM-Based Methods](https://mlsanity.com/paper/2403.02901/) (2024)
- [Element-aware Summarization with Large Language Models: Expert-aligned Evaluation and Chain-of-Thought Method](https://mlsanity.com/paper/2305.13412/) (2023)
- [DOC2PPT: Automatic Presentation Slides Generation from Scientific Documents](https://mlsanity.com/paper/2101.11796/) (2021)
- [An Iterative Optimizing Framework for Radiology Report Summarization with ChatGPT](https://mlsanity.com/paper/2304.08448/) (2023)
- [Sequential Integrated Gradients: a simple but effective method for explaining language models](https://mlsanity.com/paper/2305.15853/) (2023)
- [How Ready are Pre-trained Abstractive Models and LLMs for Legal Case Judgement Summarization?](https://mlsanity.com/paper/2306.01248/) (2023)
- [Natural Language Processing for the Legal Domain: A Survey of Tasks, Datasets, Models, and Challenges](https://mlsanity.com/paper/2410.21306/) (2024)
- [HtmlRAG: HTML is Better Than Plain Text for Modeling Retrieved Knowledge in RAG Systems](https://mlsanity.com/paper/2411.02959/) (2024)
- [LLMRec: Benchmarking Large Language Models on Recommendation Task](https://mlsanity.com/paper/2308.12241/) (2023)
- [Noise Stability Regularization for Improving BERT Fine-tuning](https://mlsanity.com/paper/2107.04835/) (2021)
- [BIDER: Bridging Knowledge Inconsistency for Efficient Retrieval-Augmented LLMs via Key Supporting Evidence](https://mlsanity.com/paper/2402.12174/) (2024)
- [Heterformer: Transformer-based Deep Node Representation Learning on Heterogeneous Text-Rich Networks](https://mlsanity.com/paper/2205.10282/) (2022)
- [OASum: Large-Scale Open Domain Aspect-based Summarization](https://mlsanity.com/paper/2212.09233/) (2022)
- [Two Stones Hit One Bird: Bilevel Positional Encoding for Better Length Extrapolation](https://mlsanity.com/paper/2401.16421/) (2024)
- [Applicability of Large Language Models and Generative Models for Legal Case Judgement Summarization](https://mlsanity.com/paper/2407.12848/) (2024)
- [Enriching and Controlling Global Semantics for Text Summarization](https://mlsanity.com/paper/2109.10616/) (2021)
- [GanLM: Encoder-Decoder Pre-training with an Auxiliary Discriminator](https://mlsanity.com/paper/2212.10218/) (2022)
- [On the Inductive Bias of Masked Language Modeling: From Statistical to Syntactic Dependencies](https://mlsanity.com/paper/2104.05694/) (2021)
- [An Extractive-and-Abstractive Framework for Source Code Summarization](https://mlsanity.com/paper/2206.07245/) (2022)
- [SpikeBERT: A Language Spikformer Learned from BERT with Knowledge Distillation](https://mlsanity.com/paper/2308.15122/) (2023)
- [Attention-based Clinical Note Summarization](https://mlsanity.com/paper/2104.08942/) (2021)
- [How well do you know your summarization datasets?](https://mlsanity.com/paper/2106.11388/) (2021)
- [CodeApex: A Bilingual Programming Evaluation Benchmark for Large Language Models](https://mlsanity.com/paper/2309.01940/) (2023)
- [Salience Allocation as Guidance for Abstractive Summarization](https://mlsanity.com/paper/2210.12330/) (2022)
- [Numerical Optimizations for Weighted Low-rank Estimation on Language Model](https://mlsanity.com/paper/2211.09718/) (2022)
- [Examining Bias in Opinion Summarisation Through the Perspective of Opinion Diversity](https://mlsanity.com/paper/2306.04424/) (2023)
- [Non-autoregressive Transformer-based End-to-end ASR using BERT](https://mlsanity.com/paper/2104.04805/) (2021)

## Abstract (arXiv)

BERT, a pre-trained Transformer model, has achieved ground-breaking performance on multiple NLP tasks. In this paper, we describe BERTSUM, a simple variant of BERT, for extractive summarization. Our system is the state of the art on the CNN/Dailymail dataset, outperforming the previous best-performed system by 1.65 on ROUGE-L. The codes to reproduce our results are available at https://github.com/nlpyang/BertSum