# TransICD: Transformer Based Code-wise Attention Model for Explainable ICD Coding

arXiv: https://arxiv.org/abs/2104.10652 · HTML: https://mlsanity.com/paper/2104.10652/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 23 references that are in the map.
- First posted on arXiv 28 Mar 2021.

## Builds on (most cited first, 13 of 13)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Efficient Estimation of Word Representations in Vector Space](https://mlsanity.com/paper/1301.3781/) (2013)
- [Learning Phrase Representations using RNN Encoder-Decoder for Statistical Machine Translation](https://mlsanity.com/paper/1406.1078/) (2014)
- [BioBERT: a pre-trained biomedical language representation model for biomedical text mining](https://mlsanity.com/paper/1901.08746/) (2019)
- [Convolutional Neural Networks for Sentence Classification](https://mlsanity.com/paper/1408.5882/) (2014)
- [Publicly Available Clinical BERT Embeddings](https://mlsanity.com/paper/1904.03323/) (2019)
- [Learning Imbalanced Datasets with Label-Distribution-Aware Margin Loss](https://mlsanity.com/paper/1906.07413/) (2019)
- [A Structured Self-attentive Sentence Embedding](https://mlsanity.com/paper/1703.03130/) (2017)
- [Explainable Prediction of Medical Codes from Clinical Text](https://mlsanity.com/paper/1802.05695/) (2018)
- [Joint Embedding of Words and Labels for Text Classification](https://mlsanity.com/paper/1805.04174/) (2018)
- [Multi-Label Classification of Patient Notes a Case Study on ICD Code Assignment](https://mlsanity.com/paper/1709.09587/) (2017)
- [Condensed Memory Networks for Clinical Diagnostic Inferencing](https://mlsanity.com/paper/1612.01848/) (2016)

## Built on by (most cited first, 13 of 13)

- [Knowledge Injected Prompt Based Fine-tuning for Multi-label Few-shot ICD Coding](https://mlsanity.com/paper/2210.03304/) (2022)
- [A Unified Review of Deep Learning for Automated Medical Coding](https://mlsanity.com/paper/2201.02797/) (2022)
- [A Survey of Methods for Addressing Class Imbalance in Deep-Learning Based Natural Language Processing](https://mlsanity.com/paper/2210.04675/) (2022)
- [Transformers in Healthcare: A Survey](https://mlsanity.com/paper/2307.00067/) (2023)
- [Hierarchical Label-wise Attention Transformer Model for Explainable ICD Coding](https://mlsanity.com/paper/2204.10716/) (2022)
- [Description-based Label Attention Classifier for Explainable ICD-9 Classification](https://mlsanity.com/paper/2109.12026/) (2021)
- [Automated ICD Coding using Extreme Multi-label Long Text Transformer-based Models](https://mlsanity.com/paper/2212.05857/) (2022)
- [Improving Predictions of Tail-end Labels using Concatenated BioMed-Transformers for Long Medical Documents](https://mlsanity.com/paper/2112.01718/) (2021)
- [An Automatic ICD Coding Network Using Partition-Based Label Attention](https://mlsanity.com/paper/2211.08429/) (2022)
- [Large Language Model in Medical Informatics: Direct Classification and Enhanced Text Representations for Automatic ICD Coding](https://mlsanity.com/paper/2411.06823/) (2024)
- [A Novel ICD Coding Method Based on Associated and Hierarchical Code Description Distillation](https://mlsanity.com/paper/2404.11132/) (2024)
- [A Novel Method to Metigate Demographic and Expert Bias in ICD Coding with Causal Inference](https://mlsanity.com/paper/2410.14236/) (2024)
- [CoLa-ICD: A Knowledge-Enhanced Framework for Long-Tail Automated Medical Coding](https://mlsanity.com/paper/2608.30234/) (2026)

## Abstract (arXiv)

International Classification of Disease (ICD) coding procedure which refers to tagging medical notes with diagnosis codes has been shown to be effective and crucial to the billing system in medical sector. Currently, ICD codes are assigned to a clinical note manually which is likely to cause many errors. Moreover, training skilled coders also requires time and human resources. Therefore, automating the ICD code determination process is an important task. With the advancement of artificial intelligence theory and computational hardware, machine learning approach has emerged as a suitable solution to automate this process. In this project, we apply a transformer-based architecture to capture the interdependence among the tokens of a document and then use a code-wise attention mechanism to learn code-specific representations of the entire document. Finally, they are fed to separate dense layers for corresponding code prediction. Furthermore, to handle the imbalance in the code frequency of clinical datasets, we employ a label distribution aware margin (LDAM) loss function. The experimental results on the MIMIC-III dataset show that our proposed model outperforms other baselines by a significant margin. In particular, our best setting achieves a micro-AUC score of 0.923 compared to 0.868 of bidirectional recurrent neural networks. We also show that by using the code-wise attention mechanism, the model can provide more insights about its prediction, and thus it can support clinicians to make reliable decisions. Our code is available online (https://github.com/biplob1ly/TransICD)