# Does the Magic of BERT Apply to Medical Code Assignment? A Quantitative Study

arXiv: https://arxiv.org/abs/2103.06511 · HTML: https://mlsanity.com/paper/2103.06511/

## Key facts

- Cited by 39 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 48 references that are in the map.
- First posted on arXiv 11 Mar 2021.

## Builds on (most cited first, 27 of 27)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Distributed Representations of Words and Phrases and their Compositionality](https://mlsanity.com/paper/1310.4546/) (2013)
- [Longformer: The Long-Document Transformer](https://mlsanity.com/paper/2004.05150/) (2020)
- [BioBERT: a pre-trained biomedical language representation model for biomedical text mining](https://mlsanity.com/paper/1901.08746/) (2019)
- [Big Bird: Transformers for Longer Sequences](https://mlsanity.com/paper/2007.14062/) (2020)
- [Don't Stop Pretraining: Adapt Language Models to Domains and Tasks](https://mlsanity.com/paper/2004.10964/) (2020)
- [Linformer: Self-Attention with Linear Complexity](https://mlsanity.com/paper/2006.04768/) (2020)
- [Convolutional Neural Networks for Sentence Classification](https://mlsanity.com/paper/1408.5882/) (2014)
- [Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Books](https://mlsanity.com/paper/1506.06724/) (2015)
- [Publicly Available Clinical BERT Embeddings](https://mlsanity.com/paper/1904.03323/) (2019)
- [Pre-trained Models for Natural Language Processing: A Survey](https://mlsanity.com/paper/2003.08271/) (2020)
- [ClinicalBERT: Modeling Clinical Notes and Predicting Hospital Readmission](https://mlsanity.com/paper/1904.05342/) (2019)
- [Transfer Learning in Biomedical Natural Language Processing: An Evaluation of BERT and ELMo on Ten Benchmarking Datasets](https://mlsanity.com/paper/1906.05474/) (2019)
- [Self-Alignment Pretraining for Biomedical Entity Representations](https://mlsanity.com/paper/2010.11784/) (2020)
- [Explainable Prediction of Medical Codes from Clinical Text](https://mlsanity.com/paper/1802.05695/) (2018)
- [ICD Coding from Clinical Text Using Multi-Filter Residual Convolutional Neural Network](https://mlsanity.com/paper/1912.00862/) (2019)
- [COMETA: A Corpus for Medical Entity Linking in the Social Media](https://mlsanity.com/paper/2010.03295/) (2020)
- [Explainable Automated Coding of Clinical Notes using Hierarchical Label-wise Attention Networks and Label Embedding Initialisation](https://mlsanity.com/paper/2010.15728/) (2020)
- [Multi-Label Classification of Patient Notes a Case Study on ICD Code Assignment](https://mlsanity.com/paper/1709.09587/) (2017)
- [Towards Automated ICD Coding Using Deep Learning](https://mlsanity.com/paper/1711.04075/) (2017)
- [Dilated Convolutional Attention Network for Medical Code Assignment from Clinical Text](https://mlsanity.com/paper/2009.14578/) (2020)
- [Phenotyping of Clinical Notes with Improved Document Classification Models Using Contextualized Neural Language Models](https://mlsanity.com/paper/1910.13664/) (2019)
- [Condensed Memory Networks for Clinical Diagnostic Inferencing](https://mlsanity.com/paper/1612.01848/) (2016)
- [Medical Code Assignment with Gated Convolution and Note-Code Interaction](https://mlsanity.com/paper/2010.06975/) (2020)

## Built on by (most cited first, 30 of 39)

- [Code Synonyms Do Matter: Multiple Synonyms Matching Network for Automatic ICD Coding](https://mlsanity.com/paper/2203.01515/) (2022)
- [Revisiting Transformer-based Models for Long Document Classification](https://mlsanity.com/paper/2204.06683/) (2022)
- [Automated Medical Coding on MIMIC-III and MIMIC-IV: A Critical Review and Replicability Study](https://mlsanity.com/paper/2304.10909/) (2023)
- [Knowledge Injected Prompt Based Fine-tuning for Multi-label Few-shot ICD Coding](https://mlsanity.com/paper/2210.03304/) (2022)
- [A Comprehensive Evaluation of Large Language Models on Benchmark Biomedical Text Processing Tasks](https://mlsanity.com/paper/2310.04270/) (2023)
- [Automated Clinical Coding: What, Why, and Where We Are?](https://mlsanity.com/paper/2203.11092/) (2022)
- [A Unified Review of Deep Learning for Automated Medical Coding](https://mlsanity.com/paper/2201.02797/) (2022)
- [CoRelation: Boosting Automatic ICD Coding Through Contextualized Code Relation Learning](https://mlsanity.com/paper/2402.15700/) (2024)
- [Multi-label Few-shot ICD Coding as Autoregressive Generation with Prompt](https://mlsanity.com/paper/2211.13813/) (2022)
- [Description-based Label Attention Classifier for Explainable ICD-9 Classification](https://mlsanity.com/paper/2109.12026/) (2021)
- [Knowledge Graphs for the Life Sciences: Recent Developments, Challenges and Opportunities](https://mlsanity.com/paper/2309.17255/) (2023)
- [Multitask Balanced and Recalibrated Network for Medical Code Prediction](https://mlsanity.com/paper/2109.02418/) (2021)
- [CoPHE: A Count-Preserving Hierarchical Evaluation Metric in Large-Scale Multi-Label Text Classification](https://mlsanity.com/paper/2109.04853/) (2021)
- [Developing a general-purpose clinical language inference model from a large corpus of clinical notes](https://mlsanity.com/paper/2210.06566/) (2022)
- [DKEC: Domain Knowledge Enhanced Multi-Label Classification for Diagnosis Prediction](https://mlsanity.com/paper/2310.07059/) (2023)
- [Modelling Temporal Document Sequences for Clinical ICD Coding](https://mlsanity.com/paper/2302.12666/) (2023)
- [Multitask Recalibrated Aggregation Network for Medical Code Prediction](https://mlsanity.com/paper/2104.00952/) (2021)
- [Efficacy of BERT embeddings on predicting disaster from Twitter data](https://mlsanity.com/paper/2108.10698/) (2021)
- [TempoFormer: A Transformer for Temporally-aware Representations in Change Detection](https://mlsanity.com/paper/2408.15689/) (2024)
- [BERT for Long Documents: A Case Study of Automated ICD Coding](https://mlsanity.com/paper/2211.02519/) (2022)
- [Towards Semi-Structured Automatic ICD Coding via Tree-based Contrastive Learning](https://mlsanity.com/paper/2310.09672/) (2023)
- [Examining Imbalance Effects on Performance and Demographic Fairness of Clinical Language Models](https://mlsanity.com/paper/2412.17803/) (2024)
- [Entity Anchored ICD Coding](https://mlsanity.com/paper/2208.07444/) (2022)
- [An Automatic ICD Coding Network Using Partition-Based Label Attention](https://mlsanity.com/paper/2211.08429/) (2022)
- [A Novel Collaborative Self-Supervised Learning Method for Radiomic Data](https://mlsanity.com/paper/2302.09807/) (2023)
- [How Long Is Enough? Exploring the Optimal Intervals of Long-Range Clinical Note Language Modeling](https://mlsanity.com/paper/2211.07713/) (2022)
- [Content Reduction, Surprisal and Information Density Estimation for Long Documents](https://mlsanity.com/paper/2309.06009/) (2023)
- [An Eye on Clinical BERT: Investigating Language Model Generalization for Diabetic Eye Disease Phenotyping](https://mlsanity.com/paper/2311.08687/) (2023)
- [Continuous Predictive Modeling of Clinical Notes and ICD Codes in Patient Health Records](https://mlsanity.com/paper/2405.11622/) (2024)
- [RuCCoD: Towards Automated ICD Coding in Russian](https://mlsanity.com/paper/2502.21263/) (2025)

## Abstract (arXiv)

Unsupervised pretraining is an integral part of many natural language processing systems, and transfer learning with language models has achieved remarkable results in many downstream tasks. In the clinical application of medical code assignment, diagnosis and procedure codes are inferred from lengthy clinical notes such as hospital discharge summaries. However, it is not clear if pretrained models are useful for medical code prediction without further architecture engineering. This paper conducts a comprehensive quantitative analysis of various contextualized language models' performance, pretrained in different domains, for medical code assignment from clinical notes. We propose a hierarchical fine-tuning architecture to capture interactions between distant words and adopt label-wise attention to exploit label information. Contrary to current trends, we demonstrate that a carefully trained classical CNN outperforms attention-based models on a MIMIC-III subset with frequent codes. Our empirical findings suggest directions for improving the medical code assignment application.