# Attention-based Clinical Note Summarization

arXiv: https://arxiv.org/abs/2104.08942 · HTML: https://mlsanity.com/paper/2104.08942/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 60 references that are in the map.
- First posted on arXiv 18 Apr 2021.

## Builds on (most cited first, 20 of 20)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Semi-Supervised Classification with Graph Convolutional Networks](https://mlsanity.com/paper/1609.02907/) (2016)
- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [Deep contextualized word representations](https://mlsanity.com/paper/1802.05365/) (2018)
- [What Does BERT Look At? An Analysis of BERT's Attention](https://mlsanity.com/paper/1906.04341/) (2019)
- [Publicly Available Clinical BERT Embeddings](https://mlsanity.com/paper/1904.03323/) (2019)
- [Revealing the Dark Secrets of BERT](https://mlsanity.com/paper/1908.08593/) (2019)
- [A Multiscale Visualization of Attention in the Transformer Model](https://mlsanity.com/paper/1906.05714/) (2019)
- [Explainable Prediction of Medical Codes from Clinical Text](https://mlsanity.com/paper/1802.05695/) (2018)
- [Semi-supervised Sequence Learning](https://mlsanity.com/paper/1511.01432/) (2015)
- [Extractive Summarization as Text Matching](https://mlsanity.com/paper/2004.08795/) (2020)
- [Fine-tune BERT for Extractive Summarization](https://mlsanity.com/paper/1903.10318/) (2019)
- [Leveraging BERT for Extractive Text Summarization on Lectures](https://mlsanity.com/paper/1906.04165/) (2019)
- [How Does BERT Answer Questions? A Layer-Wise Analysis of Transformer Representations](https://mlsanity.com/paper/1909.04925/) (2019)
- [Text Summarization Techniques: A Brief Survey](https://mlsanity.com/paper/1707.02268/) (2017)
- [Graph-based Neural Multi-Document Summarization](https://mlsanity.com/paper/1706.06681/) (2017)
- [Universal Language Model Fine-tuning for Text Classification](https://mlsanity.com/paper/1801.06146/) (2018)
- [Extractive Summarization of EHR Discharge Notes](https://mlsanity.com/paper/1810.12085/) (2018)
- [A Qualitative Evaluation of Language Models on Automatic Question-Answering for COVID-19](https://mlsanity.com/paper/2006.10964/) (2020)

## Built on by (most cited first, 16 of 16)

- [Pre-trained Language Models in Biomedical Domain: A Systematic Survey](https://mlsanity.com/paper/2110.05006/) (2021)
- [Leveraging Medical Knowledge Graphs Into Large Language Models for Diagnosis Prediction: Design and Application Study](https://mlsanity.com/paper/2308.14321/) (2023)
- [Vision Transformers for Small Histological Datasets Learned through Knowledge Distillation](https://mlsanity.com/paper/2305.17370/) (2023)
- [MEDVOC: Vocabulary Adaptation for Fine-tuning Pre-trained Language Models on Medical Text Summarization](https://mlsanity.com/paper/2405.04163/) (2024)
- [Bridging Simplicity and Sophistication using GLinear: A Novel Architecture for Enhanced Time Series Prediction](https://mlsanity.com/paper/2501.01087/) (2025)
- [A Survey for Biomedical Text Summarization: From Pre-trained to Large Language Models](https://mlsanity.com/paper/2304.08763/) (2023)
- [Neural Summarization of Electronic Health Records](https://mlsanity.com/paper/2305.15222/) (2023)
- [Onco-Retriever: Generative Classifier for Retrieval of EHR Records in Oncology](https://mlsanity.com/paper/2404.06680/) (2024)
- [Ethical Challenges and Evolving Strategies in the Integration of Artificial Intelligence into Clinical Practice](https://mlsanity.com/paper/2412.03576/) (2024)
- [Health Text Simplification: An Annotated Corpus for Digestive Cancer Education and Novel Strategies for Reinforcement Learning](https://mlsanity.com/paper/2401.15043/) (2024)
- [FHRFormer: A Self-supervised Transformer Approach for Fetal Heart Rate Inpainting and Forecasting](https://mlsanity.com/paper/2509.20852/) (2025)
- [Query-Guided Self-Supervised Summarization of Nursing Notes](https://mlsanity.com/paper/2407.04125/) (2024)
- [REMEMBER: Retrieval-based Explainable Multimodal Evidence-guided Modeling for Brain Evaluation and Reasoning in Zero- and Few-shot Neurodegenerative Diagnosis](https://mlsanity.com/paper/2504.09354/) (2025)
- [DENSE: Longitudinal Progress Note Generation with Temporal Modeling of Heterogeneous Clinical Notes Across Hospital Visits](https://mlsanity.com/paper/2507.14079/) (2025)
- [FHRFormer: A Self-Supervised Masked Transformer Framework for Fetal Heart Rate Time-Series Inpainting and Forecasting](https://mlsanity.com/paper/2605.29695/) (2026)
- [Evaluating AI Generated Summaries for Cancer Patients](https://mlsanity.com/paper/2608.26154/) (2026)

## Abstract (arXiv)

In recent years, the trend of deploying digital systems in numerous industries has hiked. The health sector has observed an extensive adoption of digital systems and services that generate significant medical records. Electronic health records contain valuable information for prospective and retrospective analysis that is often not entirely exploited because of the complicated dense information storage. The crude purpose of condensing health records is to select the information that holds most characteristics of the original documents based on a reported disease. These summaries may boost diagnosis and save a doctor's time during a saturated workload situation like the COVID-19 pandemic. In this paper, we are applying a multi-head attention-based mechanism to perform extractive summarization of meaningful phrases on clinical notes. Our method finds major sentences for a summary by correlating tokens, segments, and positional embeddings of sentences in a clinical note. The model outputs attention scores that are statistically transformed to extract critical phrases for visualization on the heat-mapping tool and for human use.