# Modelling Temporal Document Sequences for Clinical ICD Coding

arXiv: https://arxiv.org/abs/2302.12666 · HTML: https://mlsanity.com/paper/2302.12666/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 20 of its 28 references that are in the map.
- First posted on arXiv 24 Feb 2023.

## Builds on (most cited first, 20 of 20)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Longformer: The Long-Document Transformer](https://mlsanity.com/paper/2004.05150/) (2020)
- [BioBERT: a pre-trained biomedical language representation model for biomedical text mining](https://mlsanity.com/paper/1901.08746/) (2019)
- [Transformer-XL: Attentive Language Models Beyond a Fixed-Length Context](https://mlsanity.com/paper/1901.02860/) (2019)
- [Big Bird: Transformers for Longer Sequences](https://mlsanity.com/paper/2007.14062/) (2020)
- [Don't Stop Pretraining: Adapt Language Models to Domains and Tasks](https://mlsanity.com/paper/2004.10964/) (2020)
- [Domain-Specific Language Model Pretraining for Biomedical Natural Language Processing](https://mlsanity.com/paper/2007.15779/) (2020)
- [Publicly Available Clinical BERT Embeddings](https://mlsanity.com/paper/1904.03323/) (2019)
- [Transfer Learning in Biomedical Natural Language Processing: An Evaluation of BERT and ELMo on Ten Benchmarking Datasets](https://mlsanity.com/paper/1906.05474/) (2019)
- [Explainable Prediction of Medical Codes from Clinical Text](https://mlsanity.com/paper/1802.05695/) (2018)
- [Super-Convergence: Very Fast Training of Neural Networks Using Large Learning Rates](https://mlsanity.com/paper/1708.07120/) (2017)
- [HIBERT: Document Level Pre-training of Hierarchical Bidirectional Transformers for Document Summarization](https://mlsanity.com/paper/1905.06566/) (2019)
- [Code Synonyms Do Matter: Multiple Synonyms Matching Network for Automatic ICD Coding](https://mlsanity.com/paper/2203.01515/) (2022)
- [Revisiting Transformer-based Models for Long Document Classification](https://mlsanity.com/paper/2204.06683/) (2022)
- [PLM-ICD: Automatic ICD Coding with Pretrained Language Models](https://mlsanity.com/paper/2207.05289/) (2022)
- [Does the Magic of BERT Apply to Medical Code Assignment? A Quantitative Study](https://mlsanity.com/paper/2103.06511/) (2021)
- [Inferring Degrees from Incomplete Networks and Nonlinear Dynamics](https://mlsanity.com/paper/2004.10546/) (2020)
- [Towards BERT-based Automatic ICD Coding: Limitations and Opportunities](https://mlsanity.com/paper/2104.06709/) (2021)
- [Learning to Write Notes in Electronic Health Records](https://mlsanity.com/paper/1808.02622/) (2018)

## Built on by (most cited first, 7 of 7)

- [CoRelation: Boosting Automatic ICD Coding Through Contextualized Code Relation Learning](https://mlsanity.com/paper/2402.15700/) (2024)
- [A Survey on Responsible Generative AI: What to Generate and What Not](https://mlsanity.com/paper/2404.05783/) (2024)
- [TempoFormer: A Transformer for Temporally-aware Representations in Change Detection](https://mlsanity.com/paper/2408.15689/) (2024)
- [Sig-Networks Toolkit: Signature Networks for Longitudinal Language Modelling](https://mlsanity.com/paper/2312.03523/) (2023)
- [Continuous Predictive Modeling of Clinical Notes and ICD Codes in Patient Health Records](https://mlsanity.com/paper/2405.11622/) (2024)
- [Don't Pay Attention, PLANT It: Pretraining Attention via Learning-to-Rank](https://mlsanity.com/paper/2410.23066/) (2024)
- [Predictive Multimodal Modeling of Diagnoses and Treatments in EHR](https://mlsanity.com/paper/2508.11092/) (2025)

## Abstract (arXiv)

Past studies on the ICD coding problem focus on predicting clinical codes primarily based on the discharge summary. This covers only a small fraction of the notes generated during each hospital stay and leaves potential for improving performance by analysing all the available clinical notes. We propose a hierarchical transformer architecture that uses text across the entire sequence of clinical notes in each hospital stay for ICD coding, and incorporates embeddings for text metadata such as their position, time, and type of note. While using all clinical notes increases the quantity of data substantially, superconvergence can be used to reduce training costs. We evaluate the model on the MIMIC-III dataset. Our model exceeds the prior state-of-the-art when using only discharge summaries as input, and achieves further performance improvements when all clinical notes are used as input.