# Abstractive summarization of hospitalisation histories with transformer networks

arXiv: https://arxiv.org/abs/2204.02208 · HTML: https://mlsanity.com/paper/2204.02208/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 20 references that are in the map.
- First posted on arXiv 5 Apr 2022.

## Builds on (most cited first, 17 of 17)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehension](https://mlsanity.com/paper/1910.13461/) (2019)
- [Longformer: The Long-Document Transformer](https://mlsanity.com/paper/2004.05150/) (2020)
- [Neural Machine Translation of Rare Words with Subword Units](https://mlsanity.com/paper/1508.07909/) (2015)
- [Big Bird: Transformers for Longer Sequences](https://mlsanity.com/paper/2007.14062/) (2020)
- [Reformer: The Efficient Transformer](https://mlsanity.com/paper/2001.04451/) (2020)
- [Rethinking Attention with Performers](https://mlsanity.com/paper/2009.14794/) (2020)
- [Get To The Point: Summarization with Pointer-Generator Networks](https://mlsanity.com/paper/1704.04368/) (2017)
- [PEGASUS: Pre-training with Extracted Gap-sentences for Abstractive Summarization](https://mlsanity.com/paper/1912.08777/) (2019)
- [Training Deep Nets with Sublinear Memory Cost](https://mlsanity.com/paper/1604.06174/) (2016)
- [ProphetNet: Predicting Future N-gram for Sequence-to-Sequence Pre-training](https://mlsanity.com/paper/2001.04063/) (2020)
- [Leveraging Pre-trained Checkpoints for Sequence Generation Tasks](https://mlsanity.com/paper/1907.12461/) (2019)
- [Learning to Summarize Radiology Findings](https://mlsanity.com/paper/1809.04698/) (2018)
- [Dr. Summarize: Global Summarization of Medical Dialogue by Exploiting Local Structures](https://mlsanity.com/paper/2009.08666/) (2020)
- [Topic-aware Pointer-Generator Networks for Summarizing Spoken Conversations](https://mlsanity.com/paper/1910.01335/) (2019)

## Built on by (most cited first, 4 of 4)

- [A Method to Automate the Discharge Summary Hospital Course for Neurology Patients](https://mlsanity.com/paper/2305.06416/) (2023)
- [A Survey for Biomedical Text Summarization: From Pre-trained to Large Language Models](https://mlsanity.com/paper/2304.08763/) (2023)
- [Summarizing Radiology Reports Findings into Impressions](https://mlsanity.com/paper/2405.06802/) (2024)
- [Diagnosing Transformers: Illuminating Feature Spaces for Clinical Decision-Making](https://mlsanity.com/paper/2305.17588/) (2023)

## Abstract (arXiv)

In this paper we present a novel approach to abstractive summarization of patient hospitalisation histories. We applied an encoder-decoder framework with Longformer neural network as an encoder and BERT as a decoder. Our experiments show improved quality on some summarization tasks compared with pointer-generator networks. We also conducted a study with experienced physicians evaluating the results of our model in comparison with PGN baseline and human-generated abstracts, which showed the effectiveness of our model.