# Automated ICD Coding using Extreme Multi-label Long Text Transformer-based Models

arXiv: https://arxiv.org/abs/2212.05857 · HTML: https://mlsanity.com/paper/2212.05857/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 47 references that are in the map.
- First posted on arXiv 12 Dec 2022.

## Builds on (most cited first, 29 of 29)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Longformer: The Long-Document Transformer](https://mlsanity.com/paper/2004.05150/) (2020)
- [XLNet: Generalized Autoregressive Pretraining for Language Understanding](https://mlsanity.com/paper/1906.08237/) (2019)
- [BioBERT: a pre-trained biomedical language representation model for biomedical text mining](https://mlsanity.com/paper/1901.08746/) (2019)
- [Big Bird: Transformers for Longer Sequences](https://mlsanity.com/paper/2007.14062/) (2020)
- [Domain-Specific Language Model Pretraining for Biomedical Natural Language Processing](https://mlsanity.com/paper/2007.15779/) (2020)
- [Poincar'e Embeddings for Learning Hierarchical Representations](https://mlsanity.com/paper/1705.08039/) (2017)
- [ClinicalBERT: Modeling Clinical Notes and Predicting Hospital Readmission](https://mlsanity.com/paper/1904.05342/) (2019)
- [Asymmetric Loss For Multi-Label Classification](https://mlsanity.com/paper/2009.14119/) (2020)
- [Explainable Prediction of Medical Codes from Clinical Text](https://mlsanity.com/paper/1802.05695/) (2018)
- [ICD Coding from Clinical Text Using Multi-Filter Residual Convolutional Neural Network](https://mlsanity.com/paper/1912.00862/) (2019)
- [Clinical-Longformer and Clinical-BigBird: Transformers for long clinical sequences](https://mlsanity.com/paper/2201.11838/) (2022)
- [Fast Multi-Resolution Transformer Fine-tuning for Extreme Multi-label Text Classification](https://mlsanity.com/paper/2110.00685/) (2021)
- [LightXML: Transformer with Dynamic Negative Sampling for High-Performance Extreme Multi-label Text Classification](https://mlsanity.com/paper/2101.03305/) (2021)
- [PLM-ICD: Automatic ICD Coding with Pretrained Language Models](https://mlsanity.com/paper/2207.05289/) (2022)
- [Explainable Automated Coding of Clinical Notes using Hierarchical Label-wise Attention Networks and Label Embedding Initialisation](https://mlsanity.com/paper/2010.15728/) (2020)
- [Multi-Label Classification of Patient Notes a Case Study on ICD Code Assignment](https://mlsanity.com/paper/1709.09587/) (2017)
- [BERT-XML: Large Scale Automated ICD Coding Using BERT Pretraining](https://mlsanity.com/paper/2006.03685/) (2020)
- [Inferring Degrees from Incomplete Networks and Nonlinear Dynamics](https://mlsanity.com/paper/2004.10546/) (2020)
- [Dilated Convolutional Attention Network for Medical Code Assignment from Clinical Text](https://mlsanity.com/paper/2009.14578/) (2020)
- [Experimental Evaluation and Development of a Silver-Standard for the MIMIC-III Clinical Coding Dataset](https://mlsanity.com/paper/2006.07332/) (2020)
- [TransICD: Transformer Based Code-wise Attention Model for Explainable ICD Coding](https://mlsanity.com/paper/2104.10652/) (2021)
- [Hierarchical Label-wise Attention Transformer Model for Explainable ICD Coding](https://mlsanity.com/paper/2204.10716/) (2022)
- [Description-based Label Attention Classifier for Explainable ICD-9 Classification](https://mlsanity.com/paper/2109.12026/) (2021)
- [An Explainable CNN Approach for Medical Codes Prediction from Clinical Text](https://mlsanity.com/paper/2101.11430/) (2021)
- [Medical code prediction with multi-view convolution and description-regularized label-dependent attention](https://mlsanity.com/paper/1811.01468/) (2018)
- [HiCu: Leveraging Hierarchy for Curriculum Learning in Automated ICD Coding](https://mlsanity.com/paper/2208.02301/) (2022)

## Built on by (most cited first, 5 of 5)

- [Uncertainty-aware abstention in medical diagnosis based on medical texts](https://mlsanity.com/paper/2502.18050/) (2025)
- [Large Language Model in Medical Informatics: Direct Classification and Enhanced Text Representations for Automatic ICD Coding](https://mlsanity.com/paper/2411.06823/) (2024)
- [A Novel Method to Metigate Demographic and Expert Bias in ICD Coding with Causal Inference](https://mlsanity.com/paper/2410.14236/) (2024)
- [MedicalBench: Evaluating Large Language Models Toward Improved Medical Concept Extraction](https://mlsanity.com/paper/2605.20197/) (2026)
- [Automated ICD Classification of Psychiatric Diagnoses: From Classical NLP to Large Language Models](https://mlsanity.com/paper/2605.21154/) (2026)

## Abstract (arXiv)

Background: Encouraged by the success of pretrained Transformer models in many natural language processing tasks, their use for International Classification of Diseases (ICD) coding tasks is now actively being explored. In this study, we investigate three types of Transformer-based models, aiming to address the extreme label set and long text classification challenges that are posed by automated ICD coding tasks. Methods: The Transformer-based model PLM-ICD achieved the current state-of-the-art (SOTA) performance on the ICD coding benchmark dataset MIMIC-III. It was chosen as our baseline model to be further optimised. XR-Transformer, the new SOTA model in the general extreme multi-label text classification domain, and XR-LAT, a novel adaptation of the XR-Transformer model, were also trained on the MIMIC-III dataset. XR-LAT is a recursively trained model chain on a predefined hierarchical code tree with label-wise attention, knowledge transferring and dynamic negative sampling mechanisms. Results: Our optimised PLM-ICD model, which was trained with longer total and chunk sequence lengths, significantly outperformed the current SOTA PLM-ICD model, and achieved the highest micro-F1 score of 60.8%. The XR-Transformer model, although SOTA in the general domain, did not perform well across all metrics. The best XR-LAT based model obtained results that were competitive with the current SOTA PLM-ICD model, including improving the macro-AUC by 2.1%. Conclusion: Our optimised PLM-ICD model is the new SOTA model for automated ICD coding on the MIMIC-III dataset, while our novel XR-LAT model performs competitively with the previous SOTA PLM-ICD model.