# Automated Medical Coding on MIMIC-III and MIMIC-IV: A Critical Review and Replicability Study

arXiv: https://arxiv.org/abs/2304.10909 · HTML: https://mlsanity.com/paper/2304.10909/

## Key facts

- Cited by 49 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 52 references that are in the map.
- First posted on arXiv 21 Apr 2023.

## Builds on (most cited first, 23 of 23)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [wav2vec 2.0: A Framework for Self-Supervised Learning of Speech Representations](https://mlsanity.com/paper/2006.11477/) (2020)
- [Pretrained Transformers for Text Ranking: BERT and Beyond](https://mlsanity.com/paper/2010.06467/) (2020)
- [Self-Supervised Speech Representation Learning: A Review](https://mlsanity.com/paper/2205.10643/) (2022)
- [Explainable Prediction of Medical Codes from Clinical Text](https://mlsanity.com/paper/1802.05695/) (2018)
- [ICD Coding from Clinical Text Using Multi-Filter Residual Convolutional Neural Network](https://mlsanity.com/paper/1912.00862/) (2019)
- [Macro F1 and Macro F1](https://mlsanity.com/paper/1911.03347/) (2019)
- [Code Synonyms Do Matter: Multiple Synonyms Matching Network for Automatic ICD Coding](https://mlsanity.com/paper/2203.01515/) (2022)
- [PLM-ICD: Automatic ICD Coding with Pretrained Language Models](https://mlsanity.com/paper/2207.05289/) (2022)
- [Knowledge Injected Prompt Based Fine-tuning for Multi-label Few-shot ICD Coding](https://mlsanity.com/paper/2210.03304/) (2022)
- [Does the Magic of BERT Apply to Medical Code Assignment? A Quantitative Study](https://mlsanity.com/paper/2103.06511/) (2021)
- [Explainable Automated Coding of Clinical Notes using Hierarchical Label-wise Attention Networks and Label Embedding Initialisation](https://mlsanity.com/paper/2010.15728/) (2020)
- [BERT-XML: Large Scale Automated ICD Coding Using BERT Pretraining](https://mlsanity.com/paper/2006.03685/) (2020)
- [Inferring Degrees from Incomplete Networks and Nonlinear Dynamics](https://mlsanity.com/paper/2004.10546/) (2020)
- [Automated Clinical Coding: What, Why, and Where We Are?](https://mlsanity.com/paper/2203.11092/) (2022)
- [A Unified Review of Deep Learning for Automated Medical Coding](https://mlsanity.com/paper/2201.02797/) (2022)
- [Towards BERT-based Automatic ICD Coding: Limitations and Opportunities](https://mlsanity.com/paper/2104.06709/) (2021)
- [Read, Attend, and Code: Pushing the Limits of Medical Codes Prediction from Clinical Notes by Machines](https://mlsanity.com/paper/2107.10650/) (2021)
- [Experimental Evaluation and Development of a Silver-Standard for the MIMIC-III Clinical Coding Dataset](https://mlsanity.com/paper/2006.07332/) (2020)
- [Description-based Label Attention Classifier for Explainable ICD-9 Classification](https://mlsanity.com/paper/2109.12026/) (2021)
- [ICDBigBird: A Contextual Embedding Model for ICD Code Classification](https://mlsanity.com/paper/2204.10408/) (2022)

## Built on by (most cited first, 30 of 49)

- [Self-Verification Improves Few-Shot Clinical Information Extraction](https://mlsanity.com/paper/2306.00024/) (2023)
- [MedEval: A Multi-Level, Multi-Task, and Multi-Domain Medical Benchmark for Language Model Evaluation](https://mlsanity.com/paper/2310.14088/) (2023)
- [Automated clinical coding using off-the-shelf large language models](https://mlsanity.com/paper/2310.06552/) (2023)
- [Can GPT-3.5 Generate and Code Discharge Summaries?](https://mlsanity.com/paper/2401.13512/) (2024)
- [An Unsupervised Approach to Achieve Supervised-Level Explainability in Healthcare Records](https://mlsanity.com/paper/2406.08958/) (2024)
- [Taming the Sigmoid Bottleneck: Provably Argmaxable Sparse Multi-Label Classification](https://mlsanity.com/paper/2310.10443/) (2023)
- [MedCodER: A Generative AI Assistant for Medical Coding](https://mlsanity.com/paper/2409.15368/) (2024)
- [Code Like Humans: A Multi-Agent Solution for Medical Coding](https://mlsanity.com/paper/2509.05378/) (2025)
- [Knowledge Graphs for the Life Sciences: Recent Developments, Challenges and Opportunities](https://mlsanity.com/paper/2309.17255/) (2023)
- [Abstraction Alignment: Comparing Model-Learned and Human-Encoded Conceptual Relationships](https://mlsanity.com/paper/2407.12543/) (2024)
- [A General Knowledge Injection Framework for ICD Coding](https://mlsanity.com/paper/2505.18708/) (2025)
- [Beyond Label Attention: Transparency in Language Models for Automated Medical Coding via Dictionary Learning](https://mlsanity.com/paper/2411.00173/) (2024)
- [Aligning AI Research with the Needs of Clinical Coding Workflows: Eight Recommendations Based on US Data Analysis and Critical Review](https://mlsanity.com/paper/2412.18043/) (2024)
- [DILA: Dictionary Label Attention for Mechanistic Interpretability in High-dimensional Multi-label Medical Coding Prediction](https://mlsanity.com/paper/2409.10504/) (2024)
- [The Anatomy of Evidence: An Investigation Into Explainable ICD Coding](https://mlsanity.com/paper/2507.01802/) (2025)
- [Similarity-Dissimilarity Loss for Multi-label Supervised Contrastive Learning](https://mlsanity.com/paper/2410.13439/) (2024)
- [MedSyn: Enhancing Diagnostics with Human-AI Collaboration](https://mlsanity.com/paper/2506.14774/) (2025)
- [TACL: Threshold-Adaptive Curriculum Learning Strategy for Enhancing Medical Text Understanding](https://mlsanity.com/paper/2510.15269/) (2025)
- [Automated Clinical Coding for Outpatient Departments](https://mlsanity.com/paper/2312.13533/) (2023)
- [Bespoke Large Language Models for Digital Triage Assistance in Mental Health Care](https://mlsanity.com/paper/2403.19790/) (2024)
- [Energy-Based Model for Accurate Estimation of Shapley Values in Feature Attribution](https://mlsanity.com/paper/2404.01078/) (2024)
- [Automatic identification of diagnosis from hospital discharge letters via weakly supervised Natural Language Processing](https://mlsanity.com/paper/2410.15051/) (2024)
- [RuCCoD: Towards Automated ICD Coding in Russian](https://mlsanity.com/paper/2502.21263/) (2025)
- [MedGellan: LLM-Generated Medical Guidance to Support Physicians](https://mlsanity.com/paper/2507.04431/) (2025)
- [Evaluation and LLM-Guided Learning of ICD Coding Rationales](https://mlsanity.com/paper/2508.16777/) (2025)
- [A medical coding language model trained on clinical narratives from a population-wide cohort of 1.8 million patients](https://mlsanity.com/paper/2603.00221/) (2026)
- [Symphony for Medical Coding: A Next-Generation Agentic System for Scalable and Explainable Medical Coding](https://mlsanity.com/paper/2603.29709/) (2026)
- [ShifaMind: A Multiplicative Concept Bottleneck for Interpretable ICD-10 Coding](https://mlsanity.com/paper/2605.08482/) (2026)
- [Can Post-Training Turn LLMs into Good Medical Coders? An Empirical Study of Generative ICD Coding](https://mlsanity.com/paper/2606.13940/) (2026)
- [Coding historical causes of death data with Large Language Models](https://mlsanity.com/paper/2405.07560/) (2024)

## Abstract (arXiv)

Medical coding is the task of assigning medical codes to clinical free-text documentation. Healthcare professionals manually assign such codes to track patient diagnoses and treatments. Automated medical coding can considerably alleviate this administrative burden. In this paper, we reproduce, compare, and analyze state-of-the-art automated medical coding machine learning models. We show that several models underperform due to weak configurations, poorly sampled train-test splits, and insufficient evaluation. In previous work, the macro F1 score has been calculated sub-optimally, and our correction doubles it. We contribute a revised model comparison using stratified sampling and identical experimental setups, including hyperparameters and decision boundary tuning. We analyze prediction errors to validate and falsify assumptions of previous works. The analysis confirms that all models struggle with rare codes, while long documents only have a negligible impact. Finally, we present the first comprehensive results on the newly released MIMIC-IV dataset using the reproduced models. We release our code, model parameters, and new MIMIC-III and MIMIC-IV training and evaluation pipelines to accommodate fair future comparisons.