# PLM-ICD: Automatic ICD Coding with Pretrained Language Models

arXiv: https://arxiv.org/abs/2207.05289 · HTML: https://mlsanity.com/paper/2207.05289/

## Key facts

- Cited by 49 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 41 references that are in the map.
- First posted on arXiv 12 Jul 2022.

## Builds on (most cited first, 25 of 25)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Longformer: The Long-Document Transformer](https://mlsanity.com/paper/2004.05150/) (2020)
- [XLNet: Generalized Autoregressive Pretraining for Language Understanding](https://mlsanity.com/paper/1906.08237/) (2019)
- [Deep contextualized word representations](https://mlsanity.com/paper/1802.05365/) (2018)
- [BioBERT: a pre-trained biomedical language representation model for biomedical text mining](https://mlsanity.com/paper/1901.08746/) (2019)
- [Don't Stop Pretraining: Adapt Language Models to Domains and Tasks](https://mlsanity.com/paper/2004.10964/) (2020)
- [Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Books](https://mlsanity.com/paper/1506.06724/) (2015)
- [Domain-Specific Language Model Pretraining for Biomedical Natural Language Processing](https://mlsanity.com/paper/2007.15779/) (2020)
- [Publicly Available Clinical BERT Embeddings](https://mlsanity.com/paper/1904.03323/) (2019)
- [How to Fine-Tune BERT for Text Classification?](https://mlsanity.com/paper/1905.05583/) (2019)
- [Explainable Prediction of Medical Codes from Clinical Text](https://mlsanity.com/paper/1802.05695/) (2018)
- [Neural Legal Judgment Prediction in English](https://mlsanity.com/paper/1906.02059/) (2019)
- [Doctor AI: Predicting Clinical Events via Recurrent Neural Networks](https://mlsanity.com/paper/1511.05942/) (2015)
- [AttentionXML: Label Tree-based Attention-Aware Deep Model for High-Performance Extreme Multi-Label Text Classification](https://mlsanity.com/paper/1811.01727/) (2018)
- [ICD Coding from Clinical Text Using Multi-Filter Residual Convolutional Neural Network](https://mlsanity.com/paper/1912.00862/) (2019)
- [An Empirical Study on Large-Scale Multi-Label Text Classification Including Few and Zero-Shot Labels](https://mlsanity.com/paper/2010.01653/) (2020)
- [Multi-Label Classification of Patient Notes a Case Study on ICD Code Assignment](https://mlsanity.com/paper/1709.09587/) (2017)
- [BERT-XML: Large Scale Automated ICD Coding Using BERT Pretraining](https://mlsanity.com/paper/2006.03685/) (2020)
- [Towards Automated ICD Coding Using Deep Learning](https://mlsanity.com/paper/1711.04075/) (2017)
- [Inferring Degrees from Incomplete Networks and Nonlinear Dynamics](https://mlsanity.com/paper/2004.10546/) (2020)
- [Towards BERT-based Automatic ICD Coding: Limitations and Opportunities](https://mlsanity.com/paper/2104.06709/) (2021)
- [Read, Attend, and Code: Pushing the Limits of Medical Codes Prediction from Clinical Notes by Machines](https://mlsanity.com/paper/2107.10650/) (2021)

## Built on by (most cited first, 30 of 49)

- [A Survey of Large Language Models in Medicine: Progress, Application, and Challenge](https://mlsanity.com/paper/2311.05112/) (2023)
- [Automated Medical Coding on MIMIC-III and MIMIC-IV: A Critical Review and Replicability Study](https://mlsanity.com/paper/2304.10909/) (2023)
- [Knowledge Injected Prompt Based Fine-tuning for Multi-label Few-shot ICD Coding](https://mlsanity.com/paper/2210.03304/) (2022)
- [Large Language Models in Drug Discovery and Development: From Disease Mechanisms to Clinical Trials](https://mlsanity.com/paper/2409.04481/) (2024)
- [Mimic-IV-ICD: A new benchmark for eXtreme MultiLabel Classification](https://mlsanity.com/paper/2304.13998/) (2023)
- [MDACE: MIMIC Documents Annotated with Code Evidence](https://mlsanity.com/paper/2307.03859/) (2023)
- [Automated clinical coding using off-the-shelf large language models](https://mlsanity.com/paper/2310.06552/) (2023)
- [Can GPT-3.5 Generate and Code Discharge Summaries?](https://mlsanity.com/paper/2401.13512/) (2024)
- [Knowledge Graphs for the Life Sciences: Recent Developments, Challenges and Opportunities](https://mlsanity.com/paper/2309.17255/) (2023)
- [A Two-Stage Decoder for Efficient ICD Coding](https://mlsanity.com/paper/2306.00005/) (2023)
- [Almanac Copilot: Towards Autonomous Electronic Health Record Navigation](https://mlsanity.com/paper/2405.07896/) (2024)
- [Modelling Temporal Document Sequences for Clinical ICD Coding](https://mlsanity.com/paper/2302.12666/) (2023)
- [Surpassing GPT-4 Medical Coding with a Two-Stage Approach](https://mlsanity.com/paper/2311.13735/) (2023)
- [Automated ICD Coding using Extreme Multi-label Long Text Transformer-based Models](https://mlsanity.com/paper/2212.05857/) (2022)
- [Aligning AI Research with the Needs of Clinical Coding Workflows: Eight Recommendations Based on US Data Analysis and Critical Review](https://mlsanity.com/paper/2412.18043/) (2024)
- [Toward Reliable Clinical Coding with Language Models: Verification and Lightweight Adaptation](https://mlsanity.com/paper/2510.07629/) (2025)
- [Towards Semi-Structured Automatic ICD Coding via Tree-based Contrastive Learning](https://mlsanity.com/paper/2310.09672/) (2023)
- [Infusing clinical knowledge into tokenisers for language models](https://mlsanity.com/paper/2406.14312/) (2024)
- [Similarity-Dissimilarity Loss for Multi-label Supervised Contrastive Learning](https://mlsanity.com/paper/2410.13439/) (2024)
- [TACL: Threshold-Adaptive Curriculum Learning Strategy for Enhancing Medical Text Understanding](https://mlsanity.com/paper/2510.15269/) (2025)
- [Automated Clinical Coding for Outpatient Departments](https://mlsanity.com/paper/2312.13533/) (2023)
- [Bespoke Large Language Models for Digital Triage Assistance in Mental Health Care](https://mlsanity.com/paper/2403.19790/) (2024)
- [NoteContrast: Contrastive Language-Diagnostic Pretraining for Medical Text](https://mlsanity.com/paper/2412.11477/) (2024)
- [Enhancing Biomedical Relation Extraction with Directionality](https://mlsanity.com/paper/2501.14079/) (2025)
- [RuCCoD: Towards Automated ICD Coding in Russian](https://mlsanity.com/paper/2502.21263/) (2025)
- [Evaluating ChatGPT on Medical Information Extraction Tasks: Performance, Explainability and Beyond](https://mlsanity.com/paper/2601.21767/) (2026)
- [A medical coding language model trained on clinical narratives from a population-wide cohort of 1.8 million patients](https://mlsanity.com/paper/2603.00221/) (2026)
- [Symphony for Medical Coding: A Next-Generation Agentic System for Scalable and Explainable Medical Coding](https://mlsanity.com/paper/2603.29709/) (2026)
- [Retrieve, Then Classify: Corpus-Grounded Automation of Clinical Value Set Authoring](https://mlsanity.com/paper/2604.14616/) (2026)
- [ShifaMind: A Multiplicative Concept Bottleneck for Interpretable ICD-10 Coding](https://mlsanity.com/paper/2605.08482/) (2026)

## Abstract (arXiv)

Automatically classifying electronic health records (EHRs) into diagnostic codes has been challenging to the NLP community. State-of-the-art methods treated this problem as a multilabel classification problem and proposed various architectures to model this problem. However, these systems did not leverage the superb performance of pretrained language models, which achieved superb performance on natural language understanding tasks. Prior work has shown that pretrained language models underperformed on this task with the regular finetuning scheme. Therefore, this paper aims at analyzing the causes of the underperformance and developing a framework for automatic ICD coding with pretrained language models. We spotted three main issues through the experiments: 1) large label space, 2) long input sequences, and 3) domain mismatch between pretraining and fine-tuning. We propose PLMICD, a framework that tackles the challenges with various strategies. The experimental results show that our proposed framework can overcome the challenges and achieves state-of-the-art performance in terms of multiple metrics on the benchmark MIMIC data. The source code is available at https://github.com/MiuLab/PLM-ICD