mlsanityData to 7 Oct 2026

Towards BERT-based Automatic ICD Coding: Limitations and Opportunities

arXiv 2104.06709 ↗First posted 14 Apr 2021cs.CL cs.AIcited by 25

Not on the map. Why?

Abstract (arXiv)

Automatic ICD coding is the task of assigning codes from the International Classification of Diseases (ICD) to medical notes. These codes describe the state of the patient and have multiple applications, e.g., computer-assisted diagnosis or epidemiological studies. ICD coding is a challenging task due to the complexity and length of medical notes. Unlike the general trend in language processing, no transformer model has been reported to reach high performance on this task. Here, we investigate in detail ICD coding using PubMedBERT, a state-of-the-art transformer model for biomedical language understanding. We find that the difficulty of fine-tuning the model on long pieces of text is the main limitation for BERT-based models on ICD coding. We run extensive experiments and show that despite the gap with current state-of-the-art, pretrained transformers can reach competitive performance using relatively small portions of text. We point at better methods to aggregate information from long texts as the main need for improving BERT-based ICD coding.

Key facts. Cited by 25 papers in the map (data through 7 Oct 2026). Builds on 25 of its 41 references that are in the map. First posted on arXiv 14 Apr 2021.

Builds on

25 of 41 references.

References · 25

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  6. 2016Layer Normalizationcited by 5,579
  7. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  8. 2014Neural Machine Translation by Jointly Learning to Align and Translatecited by 4,530
  9. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,590
  10. 2019BioBERT: a pre-trained biomedical language representation model for biomedical text miningcited by 1,935
  11. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,718
  12. 2019Publicly Available Clinical BERT Embeddingscited by 995
13 more
  1. 2020Quantifying Attention Flow in Transformerscited by 759
  2. 2019Transfer Learning in Biomedical Natural Language Processing: An Evaluation of BERT and ELMo on Ten Benchmarking Datasetscited by 298
  3. 2018Explainable Prediction of Medical Codes from Clinical Textcited by 231
  4. 2015Doctor AI: Predicting Clinical Events via Recurrent Neural Networkscited by 205
  5. 2019On Identifiability in Transformerscited by 115
  6. 2019Learning to Deceive with Attention-Based Explanationscited by 111
  7. 2019ICD Coding from Clinical Text Using Multi-Filter Residual Convolutional Neural Networkcited by 73
  8. 2018Joint Embedding of Words and Labels for Text Classificationcited by 69
  9. 2017Multi-Label Classification of Patient Notes a Case Study on ICD Code Assignmentcited by 37
  10. 2020Inferring Degrees from Incomplete Networks and Nonlinear Dynamicscited by 33
  11. 2017Improving Palliative Care with Deep Learningcited by 28
  12. 2020Telling BERT's full story: from Local Attention to Global Aggregationcited by 15
  13. 2016Condensed Memory Networks for Clinical Diagnostic Inferencingcited by 14

Built on by

Cited by year

Citing papers per year: 2021: 1; 2022: 12; 2023: 6; 2024: 2; 2025: 4.

25 papers.

Sort:

By area:

  1. 2022Code Synonyms Do Matter: Multiple Synonyms Matching Network for Automatic ICD Codingcited by 52
  2. 2022PLM-ICD: Automatic ICD Coding with Pretrained Language Modelscited by 49
  3. 2022Revisiting Transformer-based Models for Long Document Classificationcited by 48
  4. 2023Automated Medical Coding on MIMIC-III and MIMIC-IV: A Critical Review and Replicability Studycited by 48
  5. 2022Knowledge Injected Prompt Based Fine-tuning for Multi-label Few-shot ICD Codingcited by 40
  6. 2022A Unified Review of Deep Learning for Automated Medical Codingcited by 26
  7. 2023MDACE: MIMIC Documents Annotated with Code Evidencecited by 19
  8. 2024CoRelation: Boosting Automatic ICD Coding Through Contextualized Code Relation Learningcited by 17
  9. 2022Multi-label Few-shot ICD Coding as Autoregressive Generation with Promptcited by 16
  10. 2022Hierarchical Label-wise Attention Transformer Model for Explainable ICD Codingcited by 13
  11. 2021Description-based Label Attention Classifier for Explainable ICD-9 Classificationcited by 12
  12. 2025Code Like Humans: A Multi-Agent Solution for Medical Codingcited by 10
13 more
  1. 2023Modelling Temporal Document Sequences for Clinical ICD Codingcited by 7
  2. 2022HiCu: Leveraging Hierarchy for Curriculum Learning in Automated ICD Codingcited by 6
  3. 2022Hierarchical BERT for Medical Document Understandingcited by 4
  4. 2022BERT for Long Documents: A Case Study of Automated ICD Codingcited by 4
  5. 2023Towards Semi-Structured Automatic ICD Coding via Tree-based Contrastive Learningcited by 4
  6. 2022Entity Anchored ICD Codingcited by 3
  7. 2024Optimized Biomedical Question-Answering Services with LLM and Multi-BERT Integrationcited by 2
  8. 2023Automated Clinical Coding for Outpatient Departmentscited by 1
  9. 2025RuCCoD: Towards Automated ICD Coding in Russiancited by 1
  10. 2022Semi-self-supervised Automated ICD Codingcited by 0
  11. 2023A Multi-View Joint Learning Framework for Embedding Clinical Codes and Text Using Graph Neural Networkscited by 0
  12. 2025MKE-Coder: Multi-Axial Knowledge with Evidence Verification in ICD Coding for Chinese EMRscited by 0
  13. 2025Model selection meets clinical semantics: Optimizing ICD-10-CM prediction via LLM-as-Judge evaluation, redundancy-aware sampling, and section-aware fine-tuningcited by 0