# BERT-XML: Large Scale Automated ICD Coding Using BERT Pretraining

arXiv: https://arxiv.org/abs/2006.03685 · HTML: https://mlsanity.com/paper/2006.03685/

## Key facts

- Cited by 37 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 26 May 2020.

## Built on by (most cited first, 30 of 37)

- [R3M: A Universal Visual Representation for Robot Manipulation](https://mlsanity.com/paper/2203.12601/) (2022)
- [Large Language Models are Few-Shot Clinical Information Extractors](https://mlsanity.com/paper/2205.12689/) (2022)
- [Clinical-Longformer and Clinical-BigBird: Transformers for long clinical sequences](https://mlsanity.com/paper/2201.11838/) (2022)
- [PLM-ICD: Automatic ICD Coding with Pretrained Language Models](https://mlsanity.com/paper/2207.05289/) (2022)
- [Automated Medical Coding on MIMIC-III and MIMIC-IV: A Critical Review and Replicability Study](https://mlsanity.com/paper/2304.10909/) (2023)
- [Clinical Outcome Prediction from Admission Notes using Self-Supervised Knowledge Integration](https://mlsanity.com/paper/2102.04110/) (2021)
- [Knowledge Injected Prompt Based Fine-tuning for Multi-label Few-shot ICD Coding](https://mlsanity.com/paper/2210.03304/) (2022)
- [Neural Natural Language Processing for Unstructured Data in Electronic Health Records: a Review](https://mlsanity.com/paper/2107.02975/) (2021)
- [A Comparative Study of Pretrained Language Models for Long Clinical Text](https://mlsanity.com/paper/2301.11847/) (2023)
- [A Unified Review of Deep Learning for Automated Medical Coding](https://mlsanity.com/paper/2201.02797/) (2022)
- [Large Language Models in Drug Discovery and Development: From Disease Mechanisms to Clinical Trials](https://mlsanity.com/paper/2409.04481/) (2024)
- [Read, Attend, and Code: Pushing the Limits of Medical Codes Prediction from Clinical Notes by Machines](https://mlsanity.com/paper/2107.10650/) (2021)
- [Multi-label Few-shot ICD Coding as Autoregressive Generation with Prompt](https://mlsanity.com/paper/2211.13813/) (2022)
- [Transformers in Healthcare: A Survey](https://mlsanity.com/paper/2307.00067/) (2023)
- [Multimodal data matters: language model pre-training over structured and unstructured electronic health records](https://mlsanity.com/paper/2201.10113/) (2022)
- [Hierarchical Label-wise Attention Transformer Model for Explainable ICD Coding](https://mlsanity.com/paper/2204.10716/) (2022)
- [A Review on Knowledge Graphs for Healthcare: Resources, Applications, and Promises](https://mlsanity.com/paper/2306.04802/) (2023)
- [A Systematic Literature Review of Automated ICD Coding and Classification Systems using Discharge Summaries](https://mlsanity.com/paper/2107.10652/) (2021)
- [ICDBigBird: A Contextual Embedding Model for ICD Code Classification](https://mlsanity.com/paper/2204.10408/) (2022)
- [TreeMAN: Tree-enhanced Multimodal Attention Network for ICD Coding](https://mlsanity.com/paper/2305.18576/) (2023)
- [Automated ICD Coding using Extreme Multi-label Long Text Transformer-based Models](https://mlsanity.com/paper/2212.05857/) (2022)
- [GAMedX: Generative AI-based Medical Entity Data Extractor Using Large Language Models](https://mlsanity.com/paper/2405.20585/) (2024)
- [Aligning AI Research with the Needs of Clinical Coding Workflows: Eight Recommendations Based on US Data Analysis and Critical Review](https://mlsanity.com/paper/2412.18043/) (2024)
- [BERT for Long Documents: A Case Study of Automated ICD Coding](https://mlsanity.com/paper/2211.02519/) (2022)
- [Entity Anchored ICD Coding](https://mlsanity.com/paper/2208.07444/) (2022)
- [Contrastive Learning on Multimodal Analysis of Electronic Health Records](https://mlsanity.com/paper/2403.14926/) (2024)
- [A Scalable Workflow to Build Machine Learning Classifiers with Clinician-in-the-Loop to Identify Patients in Specific Diseases](https://mlsanity.com/paper/2205.08891/) (2022)
- [Automated Clinical Coding for Outpatient Departments](https://mlsanity.com/paper/2312.13533/) (2023)
- [RuCCoD: Towards Automated ICD Coding in Russian](https://mlsanity.com/paper/2502.21263/) (2025)
- [CURENet: Combining Unified Representations for Efficient Chronic Disease Prediction](https://mlsanity.com/paper/2511.11423/) (2025)

## Abstract (arXiv)

Clinical interactions are initially recorded and documented in free text medical notes. ICD coding is the task of classifying and coding all diagnoses, symptoms and procedures associated with a patient's visit. The process is often manual and extremely time-consuming and expensive for hospitals. In this paper, we propose a machine learning model, BERT-XML, for large scale automated ICD coding from EHR notes, utilizing recently developed unsupervised pretraining that have achieved state of the art performance on a variety of NLP tasks. We train a BERT model from scratch on EHR notes, learning with vocabulary better suited for EHR tasks and thus outperform off-the-shelf models. We adapt the BERT architecture for ICD coding with multi-label attention. While other works focus on small public medical datasets, we have produced the first large scale ICD-10 classification model using millions of EHR notes to predict thousands of unique ICD codes.