mlsanityData to 7 Oct 2026

Does the Magic of BERT Apply to Medical Code Assignment? A Quantitative Study

arXiv 2103.06511 ↗First posted 11 Mar 2021cs.CLcited by 39

Not on the map. Why?

Abstract (arXiv)

Unsupervised pretraining is an integral part of many natural language processing systems, and transfer learning with language models has achieved remarkable results in many downstream tasks. In the clinical application of medical code assignment, diagnosis and procedure codes are inferred from lengthy clinical notes such as hospital discharge summaries. However, it is not clear if pretrained models are useful for medical code prediction without further architecture engineering. This paper conducts a comprehensive quantitative analysis of various contextualized language models' performance, pretrained in different domains, for medical code assignment from clinical notes. We propose a hierarchical fine-tuning architecture to capture interactions between distant words and adopt label-wise attention to exploit label information. Contrary to current trends, we demonstrate that a carefully trained classical CNN outperforms attention-based models on a MIMIC-III subset with frequent codes. Our empirical findings suggest directions for improving the medical code assignment application.

Key facts. Cited by 39 papers in the map (data through 7 Oct 2026). Builds on 27 of its 48 references that are in the map. First posted on arXiv 11 Mar 2021.

Builds on

27 of 48 references.

References · 27

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  5. 2013Distributed Representations of Words and Phrases and their Compositionalitycited by 3,462
  6. 2020Longformer: The Long-Document Transformercited by 3,362
  7. 2019BioBERT: a pre-trained biomedical language representation model for biomedical text miningcited by 1,935
  8. 2020Big Bird: Transformers for Longer Sequencescited by 1,673
  9. 2020Don't Stop Pretraining: Adapt Language Models to Domains and Taskscited by 1,548
  10. 2020Linformer: Self-Attention with Linear Complexitycited by 1,462
  11. 2014Convolutional Neural Networks for Sentence Classificationcited by 1,124
  12. 2015Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Bookscited by 1,041
15 more
  1. 2019Publicly Available Clinical BERT Embeddingscited by 995
  2. 2020Pre-trained Models for Natural Language Processing: A Surveycited by 504
  3. 2019ClinicalBERT: Modeling Clinical Notes and Predicting Hospital Readmissioncited by 487
  4. 2019Transfer Learning in Biomedical Natural Language Processing: An Evaluation of BERT and ELMo on Ten Benchmarking Datasetscited by 298
  5. 2020Self-Alignment Pretraining for Biomedical Entity Representationscited by 233
  6. 2018Explainable Prediction of Medical Codes from Clinical Textcited by 231
  7. 2019ICD Coding from Clinical Text Using Multi-Filter Residual Convolutional Neural Networkcited by 73
  8. 2020COMETA: A Corpus for Medical Entity Linking in the Social Mediacited by 49
  9. 2020Explainable Automated Coding of Clinical Notes using Hierarchical Label-wise Attention Networks and Label Embedding Initialisationcited by 38
  10. 2017Multi-Label Classification of Patient Notes a Case Study on ICD Code Assignmentcited by 37
  11. 2017Towards Automated ICD Coding Using Deep Learningcited by 35
  12. 2020Dilated Convolutional Attention Network for Medical Code Assignment from Clinical Textcited by 21
  13. 2019Phenotyping of Clinical Notes with Improved Document Classification Models Using Contextualized Neural Language Modelscited by 20
  14. 2016Condensed Memory Networks for Clinical Diagnostic Inferencingcited by 14
  15. 2020Medical Code Assignment with Gated Convolution and Note-Code Interactioncited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 5; 2022: 11; 2023: 9; 2024: 4; 2025: 6; 2026: 4.

39 papers.

Sort:

  1. 2022Code Synonyms Do Matter: Multiple Synonyms Matching Network for Automatic ICD Codingcited by 52
  2. 2022Revisiting Transformer-based Models for Long Document Classificationcited by 48
  3. 2023Automated Medical Coding on MIMIC-III and MIMIC-IV: A Critical Review and Replicability Studycited by 48
  4. 2022Knowledge Injected Prompt Based Fine-tuning for Multi-label Few-shot ICD Codingcited by 40
  5. 2023A Comprehensive Evaluation of Large Language Models on Benchmark Biomedical Text Processing Taskscited by 36
  6. 2022Automated Clinical Coding: What, Why, and Where We Are?cited by 33
  7. 2022A Unified Review of Deep Learning for Automated Medical Codingcited by 26
  8. 2024CoRelation: Boosting Automatic ICD Coding Through Contextualized Code Relation Learningcited by 17
  9. 2022Multi-label Few-shot ICD Coding as Autoregressive Generation with Promptcited by 16
  10. 2021Description-based Label Attention Classifier for Explainable ICD-9 Classificationcited by 12
  11. 2023Knowledge Graphs for the Life Sciences: Recent Developments, Challenges and Opportunitiescited by 10
  12. 2021Multitask Balanced and Recalibrated Network for Medical Code Predictioncited by 9
27 more
  1. 2021CoPHE: A Count-Preserving Hierarchical Evaluation Metric in Large-Scale Multi-Label Text Classificationcited by 9
  2. 2022Developing a general-purpose clinical language inference model from a large corpus of clinical notescited by 8
  3. 2023DKEC: Domain Knowledge Enhanced Multi-Label Classification for Diagnosis Predictioncited by 8
  4. 2023Modelling Temporal Document Sequences for Clinical ICD Codingcited by 7
  5. 2021Multitask Recalibrated Aggregation Network for Medical Code Predictioncited by 5
  6. 2021Efficacy of BERT embeddings on predicting disaster from Twitter datacited by 5
  7. 2024TempoFormer: A Transformer for Temporally-aware Representations in Change Detectioncited by 5
  8. 2022BERT for Long Documents: A Case Study of Automated ICD Codingcited by 4
  9. 2023Towards Semi-Structured Automatic ICD Coding via Tree-based Contrastive Learningcited by 4
  10. 2024Examining Imbalance Effects on Performance and Demographic Fairness of Clinical Language Modelscited by 4
  11. 2022Entity Anchored ICD Codingcited by 3
  12. 2022An Automatic ICD Coding Network Using Partition-Based Label Attentioncited by 2
  13. 2023A Novel Collaborative Self-Supervised Learning Method for Radiomic Datacited by 2
  14. 2022How Long Is Enough? Exploring the Optimal Intervals of Long-Range Clinical Note Language Modelingcited by 1
  15. 2023Content Reduction, Surprisal and Information Density Estimation for Long Documentscited by 1
  16. 2023An Eye on Clinical BERT: Investigating Language Model Generalization for Diabetic Eye Disease Phenotypingcited by 1
  17. 2024Continuous Predictive Modeling of Clinical Notes and ICD Codes in Patient Health Recordscited by 1
  18. 2025RuCCoD: Towards Automated ICD Coding in Russiancited by 1
  19. 2025MKE-Coder: Multi-Axial Knowledge with Evidence Verification in ICD Coding for Chinese EMRscited by 0
  20. 2025Task as Context Prompting for Accurate Medical Symptom Coding Using Large Language Modelscited by 0
  21. 2025Predictive Multimodal Modeling of Diagnoses and Treatments in EHRcited by 0
  22. 2025Structured Information Matters: Explainable ICD Coding with Patient-Level Knowledge Graphscited by 0
  23. 2025Model selection meets clinical semantics: Optimizing ICD-10-CM prediction via LLM-as-Judge evaluation, redundancy-aware sampling, and section-aware fine-tuningcited by 0
  24. 2026Transparency-First Medical Language Models: Datasheets, Model Cards, and End-to-End Data Provenance for Clinical NLPcited by 0
  25. 2026MedicalBench: Evaluating Large Language Models Toward Improved Medical Concept Extractioncited by 0
  26. 2026Generalistic or Specific Embeddings, Which is Better? An Empirical Study on Search for Clinical Coding in Non-English Languagescited by 0
  27. 2026Bag of Tricks or Bag of Myths? Reducing Modeling Complexity with Task Knowledge in Explainable Suicide Risk Assessmentcited by 0