# Progress Notes Classification and Keyword Extraction using Attention-based Deep Learning Models with BERT

arXiv: https://arxiv.org/abs/1910.05786 · HTML: https://mlsanity.com/paper/1910.05786/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 13 Oct 2019.

## Built on by (most cited first, 11 of 11)

- [A Survey of Large Language Models in Medicine: Progress, Application, and Challenge](https://mlsanity.com/paper/2311.05112/) (2023)
- [AMMU : A Survey of Transformer-based Biomedical Pretrained Language Models](https://mlsanity.com/paper/2105.00827/) (2021)
- [Phraseformer: Multimodal Key-phrase Extraction using Transformer and Graph Embedding](https://mlsanity.com/paper/2106.04939/) (2021)
- [Born for Auto-Tagging: Faster and better with new objective functions](https://mlsanity.com/paper/2206.07264/) (2022)
- [UTNLP at SemEval-2021 Task 5: A Comparative Analysis of Toxic Span Detection using Attention-based, Named Entity Recognition, and Ensemble Models](https://mlsanity.com/paper/2104.04770/) (2021)
- [WHOSe Heritage: Classification of UNESCO World Heritage "Outstanding Universal Value" Documents with Soft Labels](https://mlsanity.com/paper/2104.05547/) (2021)
- [COVID-19 Fake News Detection Using Bidirectional Encoder Representations from Transformers Based Models](https://mlsanity.com/paper/2109.14816/) (2021)
- [JobHam-place with smart recommend job options and candidate filtering options](https://mlsanity.com/paper/2303.17930/) (2023)
- [Query-Guided Self-Supervised Summarization of Nursing Notes](https://mlsanity.com/paper/2407.04125/) (2024)
- [R-GAT: Cancer Document Classification Leveraging Graph-Based Residual Network for Scenarios with Limited Data](https://mlsanity.com/paper/2410.15198/) (2024)
- [DENSE: Longitudinal Progress Note Generation with Temporal Modeling of Heterogeneous Clinical Notes Across Hospital Visits](https://mlsanity.com/paper/2507.14079/) (2025)

## Abstract (arXiv)

Various deep learning algorithms have been developed to analyze different types of clinical data including clinical text classification and extracting information from 'free text' and so on. However, automate the keyword extraction from the clinical notes is still challenging. The challenges include dealing with noisy clinical notes which contain various abbreviations, possible typos, and unstructured sentences. The objective of this research is to investigate the attention-based deep learning models to classify the de-identified clinical progress notes extracted from a real-world EHR system. The attention-based deep learning models can be used to interpret the models and understand the critical words that drive the correct or incorrect classification of the clinical progress notes. The attention-based models in this research are capable of presenting the human interpretable text classification models. The results show that the fine-tuned BERT with the attention layer can achieve a high classification accuracy of 97.6%, which is higher than the baseline fine-tuned BERT classification model. In this research, we also demonstrate that the attention-based models can identify relevant keywords that are strongly related to the clinical progress note categories.