# Hierarchical Label-wise Attention Transformer Model for Explainable ICD Coding

arXiv: https://arxiv.org/abs/2204.10716 · HTML: https://mlsanity.com/paper/2204.10716/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Builds on 35 of its 55 references that are in the map.
- First posted on arXiv 22 Apr 2022.

## Builds on (most cited first, 30 of 35)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Efficient Estimation of Word Representations in Vector Space](https://mlsanity.com/paper/1301.3781/) (2013)
- [Longformer: The Long-Document Transformer](https://mlsanity.com/paper/2004.05150/) (2020)
- [XLNet: Generalized Autoregressive Pretraining for Language Understanding](https://mlsanity.com/paper/1906.08237/) (2019)
- [BioBERT: a pre-trained biomedical language representation model for biomedical text mining](https://mlsanity.com/paper/1901.08746/) (2019)
- [SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systems](https://mlsanity.com/paper/1905.00537/) (2019)
- [Big Bird: Transformers for Longer Sequences](https://mlsanity.com/paper/2007.14062/) (2020)
- [Domain-Specific Language Model Pretraining for Biomedical Natural Language Processing](https://mlsanity.com/paper/2007.15779/) (2020)
- [Publicly Available Clinical BERT Embeddings](https://mlsanity.com/paper/1904.03323/) (2019)
- [Attention is not Explanation](https://mlsanity.com/paper/1902.10186/) (2019)
- [Attention is not not Explanation](https://mlsanity.com/paper/1908.04626/) (2019)
- [ClinicalBERT: Modeling Clinical Notes and Predicting Hospital Readmission](https://mlsanity.com/paper/1904.05342/) (2019)
- [European Union regulations on algorithmic decision-making and a "right to explanation"](https://mlsanity.com/paper/1606.08813/) (2016)
- [Transfer Learning in Biomedical Natural Language Processing: An Evaluation of BERT and ELMo on Ten Benchmarking Datasets](https://mlsanity.com/paper/1906.05474/) (2019)
- [Explainable Prediction of Medical Codes from Clinical Text](https://mlsanity.com/paper/1802.05695/) (2018)
- [Compressing BERT: Studying the Effects of Weight Pruning on Transfer Learning](https://mlsanity.com/paper/2002.08307/) (2020)
- [ICD Coding from Clinical Text Using Multi-Filter Residual Convolutional Neural Network](https://mlsanity.com/paper/1912.00862/) (2019)
- [On quantitative aspects of model interpretability](https://mlsanity.com/paper/2007.07584/) (2020)
- [Joint Embedding of Words and Labels for Text Classification](https://mlsanity.com/paper/1805.04174/) (2018)
- [Fast Multi-Resolution Transformer Fine-tuning for Extreme Multi-label Text Classification](https://mlsanity.com/paper/2110.00685/) (2021)
- [A Label Attention Model for ICD Coding from Clinical Text](https://mlsanity.com/paper/2007.06351/) (2020)
- [Explainable Automated Coding of Clinical Notes using Hierarchical Label-wise Attention Networks and Label Embedding Initialisation](https://mlsanity.com/paper/2010.15728/) (2020)
- [Multi-Label Classification of Patient Notes a Case Study on ICD Code Assignment](https://mlsanity.com/paper/1709.09587/) (2017)
- [BERT-XML: Large Scale Automated ICD Coding Using BERT Pretraining](https://mlsanity.com/paper/2006.03685/) (2020)
- [Towards Automated ICD Coding Using Deep Learning](https://mlsanity.com/paper/1711.04075/) (2017)
- [Clinical XLNet: Modeling Sequential Clinical Notes and Predicting Prolonged Mechanical Ventilation](https://mlsanity.com/paper/1912.11975/) (2019)
- [Towards BERT-based Automatic ICD Coding: Limitations and Opportunities](https://mlsanity.com/paper/2104.06709/) (2021)
- [Dilated Convolutional Attention Network for Medical Code Assignment from Clinical Text](https://mlsanity.com/paper/2009.14578/) (2020)
- [TransICD: Transformer Based Code-wise Attention Model for Explainable ICD Coding](https://mlsanity.com/paper/2104.10652/) (2021)

## Built on by (most cited first, 13 of 13)

- [MDACE: MIMIC Documents Annotated with Code Evidence](https://mlsanity.com/paper/2307.03859/) (2023)
- [CoRelation: Boosting Automatic ICD Coding Through Contextualized Code Relation Learning](https://mlsanity.com/paper/2402.15700/) (2024)
- [A Two-Stage Decoder for Efficient ICD Coding](https://mlsanity.com/paper/2306.00005/) (2023)
- [Automated ICD Coding using Extreme Multi-label Long Text Transformer-based Models](https://mlsanity.com/paper/2212.05857/) (2022)
- [An Automatic ICD Coding Network Using Partition-Based Label Attention](https://mlsanity.com/paper/2211.08429/) (2022)
- [Large Language Model in Medical Informatics: Direct Classification and Enhanced Text Representations for Automatic ICD Coding](https://mlsanity.com/paper/2411.06823/) (2024)
- [Improving Text-based Early Prediction by Distillation from Privileged Time-Series Text](https://mlsanity.com/paper/2301.10887/) (2023)
- [Simple Data Augmentation Techniques for Chinese Disease Normalization](https://mlsanity.com/paper/2306.01931/) (2023)
- [MHLAT: Multi-hop Label-wise Attention Model for Automatic ICD Coding](https://mlsanity.com/paper/2309.08868/) (2023)
- [Exploring Accurate and Transparent Domain Adaptation in Predictive Healthcare via Concept-Grounded Orthogonal Inference](https://mlsanity.com/paper/2602.12542/) (2026)
- [Data Augmentation Techniques for Chinese Disease Name Normalization](https://mlsanity.com/paper/2501.01195/) (2025)
- [Combining Language and Topic Models for Hierarchical Text Classification](https://mlsanity.com/paper/2507.16490/) (2025)
- [TraceCoder: Towards Traceable ICD Coding via Multi-Source Knowledge Integration](https://mlsanity.com/paper/2510.15267/) (2025)

## Abstract (arXiv)

International Classification of Diseases (ICD) coding plays an important role in systematically classifying morbidity and mortality data. In this study, we propose a hierarchical label-wise attention Transformer model (HiLAT) for the explainable prediction of ICD codes from clinical documents. HiLAT firstly fine-tunes a pretrained Transformer model to represent the tokens of clinical documents. We subsequently employ a two-level hierarchical label-wise attention mechanism that creates label-specific document representations. These representations are in turn used by a feed-forward neural network to predict whether a specific ICD code is assigned to the input clinical document of interest. We evaluate HiLAT using hospital discharge summaries and their corresponding ICD-9 codes from the MIMIC-III database. To investigate the performance of different types of Transformer models, we develop ClinicalplusXLNet, which conducts continual pretraining from XLNet-Base using all the MIMIC-III clinical notes. The experiment results show that the F1 scores of the HiLAT+ClinicalplusXLNet outperform the previous state-of-the-art models for the top-50 most frequent ICD-9 codes from MIMIC-III. Visualisations of attention weights present a potential explainability tool for checking the face validity of ICD code predictions.