# Knowledge Injected Prompt Based Fine-tuning for Multi-label Few-shot ICD Coding

arXiv: https://arxiv.org/abs/2210.03304 · HTML: https://mlsanity.com/paper/2210.03304/

## Key facts

- Cited by 40 papers in the map (data through 14 Oct 2026).
- Builds on 47 of its 75 references that are in the map.
- First posted on arXiv 7 Oct 2022.

## Builds on (most cited first, 30 of 47)

- [Efficient Estimation of Word Representations in Vector Space](https://mlsanity.com/paper/1301.3781/) (2013)
- [Longformer: The Long-Document Transformer](https://mlsanity.com/paper/2004.05150/) (2020)
- [Making Pre-trained Language Models Better Few-shot Learners](https://mlsanity.com/paper/2012.15723/) (2020)
- [BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-models](https://mlsanity.com/paper/2106.10199/) (2021)
- [What Makes for Good Views for Contrastive Learning?](https://mlsanity.com/paper/2005.10243/) (2020)
- [It's Not Just Size That Matters: Small Language Models Are Also Few-Shot Learners](https://mlsanity.com/paper/2009.07118/) (2020)
- [Passage Re-ranking with BERT](https://mlsanity.com/paper/1901.04085/) (2019)
- [Self-Alignment Pretraining for Biomedical Entity Representations](https://mlsanity.com/paper/2010.11784/) (2020)
- [Explainable Prediction of Medical Codes from Clinical Text](https://mlsanity.com/paper/1802.05695/) (2018)
- [How Many Data Points is a Prompt Worth?](https://mlsanity.com/paper/2103.08493/) (2021)
- [CLEAR: Contrastive Learning for Sentence Representation](https://mlsanity.com/paper/2012.15466/) (2020)
- [Knowledgeable Prompt-tuning: Incorporating Knowledge into Prompt Verbalizer for Text Classification](https://mlsanity.com/paper/2108.02035/) (2021)
- [Open Domain Question Answering Using Early Fusion of Knowledge Bases and Text](https://mlsanity.com/paper/1809.00782/) (2018)
- [Cutting Down on Prompts and Parameters: Simple Few-Shot Learning with Language Models](https://mlsanity.com/paper/2106.13353/) (2021)
- [Deep Metric Learning with Hierarchical Triplet Loss](https://mlsanity.com/paper/1810.06951/) (2018)
- [Inferring Networks of Substitutable and Complementary Products](https://mlsanity.com/paper/1506.08839/) (2015)
- [Can Language Models be Biomedical Knowledge Bases?](https://mlsanity.com/paper/2109.07154/) (2021)
- [ICD Coding from Clinical Text Using Multi-Filter Residual Convolutional Neural Network](https://mlsanity.com/paper/1912.00862/) (2019)
- [Clinical-Longformer and Clinical-BigBird: Transformers for long clinical sequences](https://mlsanity.com/paper/2201.11838/) (2022)
- [Biomedical Entity Representations with Synonym Marginalization](https://mlsanity.com/paper/2005.00239/) (2020)
- [Code Synonyms Do Matter: Multiple Synonyms Matching Network for Automatic ICD Coding](https://mlsanity.com/paper/2203.01515/) (2022)
- [Revisiting Transformer-based Models for Long Document Classification](https://mlsanity.com/paper/2204.06683/) (2022)
- [PLM-ICD: Automatic ICD Coding with Pretrained Language Models](https://mlsanity.com/paper/2207.05289/) (2022)
- [A Label Attention Model for ICD Coding from Clinical Text](https://mlsanity.com/paper/2007.06351/) (2020)
- [Phrase-BERT: Improved Phrase Embeddings from BERT with an Application to Corpus Exploration](https://mlsanity.com/paper/2109.06304/) (2021)
- [Does the Magic of BERT Apply to Medical Code Assignment? A Quantitative Study](https://mlsanity.com/paper/2103.06511/) (2021)
- [Explainable Automated Coding of Clinical Notes using Hierarchical Label-wise Attention Networks and Label Embedding Initialisation](https://mlsanity.com/paper/2010.15728/) (2020)
- [BERT-XML: Large Scale Automated ICD Coding Using BERT Pretraining](https://mlsanity.com/paper/2006.03685/) (2020)
- [Knowledge-Rich Self-Supervision for Biomedical Entity Linking](https://mlsanity.com/paper/2112.07887/) (2021)
- [Towards BERT-based Automatic ICD Coding: Limitations and Opportunities](https://mlsanity.com/paper/2104.06709/) (2021)

## Built on by (most cited first, 30 of 40)

- [Automated Medical Coding on MIMIC-III and MIMIC-IV: A Critical Review and Replicability Study](https://mlsanity.com/paper/2304.10909/) (2023)
- [ClinicalMamba: A Generative Clinical Language Model on Longitudinal Clinical Notes](https://mlsanity.com/paper/2403.05795/) (2024)
- [Mimic-IV-ICD: A new benchmark for eXtreme MultiLabel Classification](https://mlsanity.com/paper/2304.13998/) (2023)
- [CoRelation: Boosting Automatic ICD Coding Through Contextualized Code Relation Learning](https://mlsanity.com/paper/2402.15700/) (2024)
- [Multi-label Few-shot ICD Coding as Autoregressive Generation with Prompt](https://mlsanity.com/paper/2211.13813/) (2022)
- [Automated clinical coding using off-the-shelf large language models](https://mlsanity.com/paper/2310.06552/) (2023)
- [Exploring LLM Multi-Agents for ICD Coding](https://mlsanity.com/paper/2406.15363/) (2024)
- [PULSAR at MEDIQA-Sum 2023: Large Language Models Augmented by Synthetic Dialogue Convert Patient Dialogues to Medical Records](https://mlsanity.com/paper/2307.02006/) (2023)
- [From Explainable to Interpretable Deep Learning for Natural Language Processing in Healthcare: How Far from Reality?](https://mlsanity.com/paper/2403.11894/) (2024)
- [A Two-Stage Decoder for Efficient ICD Coding](https://mlsanity.com/paper/2306.00005/) (2023)
- [Accurate and Well-Calibrated ICD Code Assignment Through Attention Over Diverse Label Embeddings](https://mlsanity.com/paper/2402.03172/) (2024)
- [Multi-stage Retrieve and Re-rank Model for Automatic Medical Coding Recommendation](https://mlsanity.com/paper/2405.19093/) (2024)
- [A General Knowledge Injection Framework for ICD Coding](https://mlsanity.com/paper/2505.18708/) (2025)
- [Automated Identification of Eviction Status from Electronic Health Record Notes](https://mlsanity.com/paper/2212.02762/) (2022)
- [Surpassing GPT-4 Medical Coding with a Two-Stage Approach](https://mlsanity.com/paper/2311.13735/) (2023)
- [Aligning AI Research with the Needs of Clinical Coding Workflows: Eight Recommendations Based on US Data Analysis and Critical Review](https://mlsanity.com/paper/2412.18043/) (2024)
- [Towards Semi-Structured Automatic ICD Coding via Tree-based Contrastive Learning](https://mlsanity.com/paper/2310.09672/) (2023)
- [ODD: A Benchmark Dataset for the Natural Language Processing based Opioid Related Aberrant Behavior Detection](https://mlsanity.com/paper/2307.02591/) (2023)
- [Auxiliary Knowledge-Induced Learning for Automatic Multi-Label Medical Document Classification](https://mlsanity.com/paper/2405.19084/) (2024)
- [Your Pretrained Model Tells the Difficulty Itself: A Self-Adaptive Curriculum Learning Paradigm for Natural Language Understanding](https://mlsanity.com/paper/2507.09758/) (2025)
- [Similarity-Dissimilarity Loss for Multi-label Supervised Contrastive Learning](https://mlsanity.com/paper/2410.13439/) (2024)
- [Segmented Harmonic Loss: Handling Class-Imbalanced Multi-Label Clinical Data for Medical Coding with Large Language Models](https://mlsanity.com/paper/2310.04595/) (2023)
- [Keyword-optimized Template Insertion for Clinical Information Extraction via Prompt-based Learning](https://mlsanity.com/paper/2310.20089/) (2023)
- [Text2Tree: Aligning Text Representation to the Label Tree Hierarchy for Imbalanced Medical Classification](https://mlsanity.com/paper/2311.16650/) (2023)
- [Automated Clinical Coding for Outpatient Departments](https://mlsanity.com/paper/2312.13533/) (2023)
- [NoteContrast: Contrastive Language-Diagnostic Pretraining for Medical Text](https://mlsanity.com/paper/2412.11477/) (2024)
- [RuCCoD: Towards Automated ICD Coding in Russian](https://mlsanity.com/paper/2502.21263/) (2025)
- [Evaluation and LLM-Guided Learning of ICD Coding Rationales](https://mlsanity.com/paper/2508.16777/) (2025)
- [ShifaMind: A Multiplicative Concept Bottleneck for Interpretable ICD-10 Coding](https://mlsanity.com/paper/2605.08482/) (2026)
- [Hierarchical Pretraining for Biomedical Term Embeddings](https://mlsanity.com/paper/2307.00266/) (2023)

## Abstract (arXiv)

Automatic International Classification of Diseases (ICD) coding aims to assign multiple ICD codes to a medical note with average length of 3,000+ tokens. This task is challenging due to a high-dimensional space of multi-label assignment (tens of thousands of ICD codes) and the long-tail challenge: only a few codes (common diseases) are frequently assigned while most codes (rare diseases) are infrequently assigned. This study addresses the long-tail challenge by adapting a prompt-based fine-tuning technique with label semantics, which has been shown to be effective under few-shot setting. To further enhance the performance in medical domain, we propose a knowledge-enhanced longformer by injecting three domain-specific knowledge: hierarchy, synonym, and abbreviation with additional pretraining using contrastive learning. Experiments on MIMIC-III-full, a benchmark dataset of code assignment, show that our proposed method outperforms previous state-of-the-art method in 14.5% in marco F1 (from 10.3 to 11.8, P<0.001). To further test our model on few-shot setting, we created a new rare diseases coding dataset, MIMIC-III-rare50, on which our model improves marco F1 from 17.1 to 30.4 and micro F1 from 17.2 to 32.6 compared to previous method.