# Surpassing GPT-4 Medical Coding with a Two-Stage Approach

arXiv: https://arxiv.org/abs/2311.13735 · HTML: https://mlsanity.com/paper/2311.13735/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 39 of its 67 references that are in the map.
- First posted on arXiv 22 Nov 2023.

## Builds on (most cited first, 30 of 39)

- [GPT-4 Technical Report](https://mlsanity.com/paper/2303.08774/) (2023)
- [Lost in the Middle: How Language Models Use Long Contexts](https://mlsanity.com/paper/2307.03172/) (2023)
- [Calibrate Before Use: Improving Few-Shot Performance of Language Models](https://mlsanity.com/paper/2102.09690/) (2021)
- [Attention is not Explanation](https://mlsanity.com/paper/1902.10186/) (2019)
- [MetaICL: Learning to Learn In Context](https://mlsanity.com/paper/2110.15943/) (2021)
- [Large Language Models are Better Reasoners with Self-Verification](https://mlsanity.com/paper/2212.09561/) (2022)
- [MedAlpaca -- An Open-Source Collection of Medical Conversational AI Models and Training Data](https://mlsanity.com/paper/2304.08247/) (2023)
- [Explainable Prediction of Medical Codes from Clinical Text](https://mlsanity.com/paper/1802.05695/) (2018)
- [Large Language Models are Few-Shot Clinical Information Extractors](https://mlsanity.com/paper/2205.12689/) (2022)
- [Explainable Artificial Intelligence: Understanding, Visualizing and Interpreting Deep Learning Models](https://mlsanity.com/paper/1708.08296/) (2017)
- [Simple and Effective Multi-Paragraph Reading Comprehension](https://mlsanity.com/paper/1710.10723/) (2017)
- [Thinking about GPT-3 In-Context Learning for Biomedical IE? Think Again](https://mlsanity.com/paper/2203.08410/) (2022)
- [A Discrete Hard EM Approach for Weakly Supervised Question Answering](https://mlsanity.com/paper/1909.04849/) (2019)
- [Improving In-Context Few-Shot Learning via Self-Supervised Training](https://mlsanity.com/paper/2205.01703/) (2022)
- [Code Synonyms Do Matter: Multiple Synonyms Matching Network for Automatic ICD Coding](https://mlsanity.com/paper/2203.01515/) (2022)
- [Self-Verification Improves Few-Shot Clinical Information Extraction](https://mlsanity.com/paper/2306.00024/) (2023)
- [PLM-ICD: Automatic ICD Coding with Pretrained Language Models](https://mlsanity.com/paper/2207.05289/) (2022)
- [Knowledge Injected Prompt Based Fine-tuning for Multi-label Few-shot ICD Coding](https://mlsanity.com/paper/2210.03304/) (2022)
- [Explainable Automated Coding of Clinical Notes using Hierarchical Label-wise Attention Networks and Label Embedding Initialisation](https://mlsanity.com/paper/2010.15728/) (2020)
- [Perturbing Inputs for Fragile Interpretations in Deep Natural Language Processing](https://mlsanity.com/paper/2108.04990/) (2021)
- [Clinical Prompt Learning with Frozen Language Models](https://mlsanity.com/paper/2205.05535/) (2022)
- [Mimic-IV-ICD: A new benchmark for eXtreme MultiLabel Classification](https://mlsanity.com/paper/2304.13998/) (2023)
- [MDACE: MIMIC Documents Annotated with Code Evidence](https://mlsanity.com/paper/2307.03859/) (2023)
- [Identifying and Extracting Rare Disease Phenotypes with Large Language Models](https://mlsanity.com/paper/2306.12656/) (2023)
- [Multi-label Few-shot ICD Coding as Autoregressive Generation with Prompt](https://mlsanity.com/paper/2211.13813/) (2022)
- [Why do you think that? Exploring Faithful Sentence-Level Rationales Without Supervision](https://mlsanity.com/paper/2010.03384/) (2020)
- [CHiLL: Zero-shot Custom Interpretable Feature Extraction from Clinical Notes with Large Language Models](https://mlsanity.com/paper/2302.12343/) (2023)
- [Fooling Explanations in Text Classifiers](https://mlsanity.com/paper/2206.03178/) (2022)
- [ICDBigBird: A Contextual Embedding Model for ICD Code Classification](https://mlsanity.com/paper/2204.10408/) (2022)
- [A Two-Stage Decoder for Efficient ICD Coding](https://mlsanity.com/paper/2306.00005/) (2023)

## Built on by (most cited first, 6 of 6)

- [Reinforcement Learning for Out-of-Distribution Reasoning in LLMs: An Empirical Study on Diagnosis-Related Group Coding](https://mlsanity.com/paper/2505.21908/) (2025)
- [DILA: Dictionary Label Attention for Mechanistic Interpretability in High-dimensional Multi-label Medical Coding Prediction](https://mlsanity.com/paper/2409.10504/) (2024)
- [Stochastic Parrots or ICU Experts? Large Language Models in Critical Care Medicine: A Scoping Review](https://mlsanity.com/paper/2407.19256/) (2024)
- [Don't Pay Attention, PLANT It: Pretraining Attention via Learning-to-Rank](https://mlsanity.com/paper/2410.23066/) (2024)
- [Automated ICD Classification of Psychiatric Diagnoses: From Classical NLP to Large Language Models](https://mlsanity.com/paper/2605.21154/) (2026)
- [LearnActCoder: Role-Aware Error Memory for Adaptive Clinical Coding Agents](https://mlsanity.com/paper/2609.19721/) (2026)

## Abstract (arXiv)

Recent advances in large language models (LLMs) show potential for clinical applications, such as clinical decision support and trial recommendations. However, the GPT-4 LLM predicts an excessive number of ICD codes for medical coding tasks, leading to high recall but low precision. To tackle this challenge, we introduce LLM-codex, a two-stage approach to predict ICD codes that first generates evidence proposals using an LLM and then employs an LSTM-based verification stage. The LSTM learns from both the LLM's high recall and human expert's high precision, using a custom loss function. Our model is the only approach that simultaneously achieves state-of-the-art results in medical coding accuracy, accuracy on rare codes, and sentence-level evidence identification to support coding decisions without training on human-annotated evidence according to experiments on the MIMIC dataset.