# Abstractive Text Summarization based on Language Model Conditioning and Locality Modeling

arXiv: https://arxiv.org/abs/2003.13027 · HTML: https://mlsanity.com/paper/2003.13027/

## Key facts

- Cited by 9 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 29 Mar 2020.

## Built on by (most cited first, 9 of 9)

- [LIFT: Language-Interfaced Fine-Tuning for Non-Language Machine Learning Tasks](https://mlsanity.com/paper/2206.06565/) (2022)
- [HiStruct+: Improving Extractive Text Summarization with Hierarchical Structure Information](https://mlsanity.com/paper/2203.09629/) (2022)
- [Generation of Highlights from Research Papers Using Pointer-Generator Networks and SciBERT Embeddings](https://mlsanity.com/paper/2302.07729/) (2023)
- [An Analysis of Abstractive Text Summarization Using Pre-trained Models](https://mlsanity.com/paper/2303.12796/) (2023)
- [Can Language Representation Models Think in Bets?](https://mlsanity.com/paper/2210.07519/) (2022)
- [Abstractive Text Summarization using Attentive GRU based Encoder-Decoder](https://mlsanity.com/paper/2302.13117/) (2023)
- [Analysis of Multidomain Abstractive Summarization Using Salience Allocation](https://mlsanity.com/paper/2402.11955/) (2024)
- [Enhancing Abstractive Summarization of Scientific Papers Using Structure Information](https://mlsanity.com/paper/2505.14179/) (2025)
- [On the State of German (Abstractive) Text Summarization](https://mlsanity.com/paper/2301.07095/) (2023)

## Abstract (arXiv)

We explore to what extent knowledge about the pre-trained language model that is used is beneficial for the task of abstractive summarization. To this end, we experiment with conditioning the encoder and decoder of a Transformer-based neural model on the BERT language model. In addition, we propose a new method of BERT-windowing, which allows chunk-wise processing of texts longer than the BERT window size. We also explore how locality modelling, i.e., the explicit restriction of calculations to the local context, can affect the summarization ability of the Transformer. This is done by introducing 2-dimensional convolutional self-attention into the first layers of the encoder. The results of our models are compared to a baseline and the state-of-the-art models on the CNN/Daily Mail dataset. We additionally train our model on the SwissText dataset to demonstrate usability on German. Both models outperform the baseline in ROUGE scores on two datasets and show its superiority in a manual qualitative analysis.