mlsanityData to 7 Oct 2026

Modelling Temporal Document Sequences for Clinical ICD Coding

arXiv 2302.12666 ↗First posted 24 Feb 2023cs.LG cs.AI cs.CLcited by 7

Not on the map. Why?

Abstract (arXiv)

Past studies on the ICD coding problem focus on predicting clinical codes primarily based on the discharge summary. This covers only a small fraction of the notes generated during each hospital stay and leaves potential for improving performance by analysing all the available clinical notes. We propose a hierarchical transformer architecture that uses text across the entire sequence of clinical notes in each hospital stay for ICD coding, and incorporates embeddings for text metadata such as their position, time, and type of note. While using all clinical notes increases the quantity of data substantially, superconvergence can be used to reduce training costs. We evaluate the model on the MIMIC-III dataset. Our model exceeds the prior state-of-the-art when using only discharge summaries as input, and achieves further performance improvements when all clinical notes are used as input.

Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Builds on 20 of its 28 references that are in the map. First posted on arXiv 24 Feb 2023.

Builds on

20 of 28 references.

References · 20

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  3. 2020Longformer: The Long-Document Transformercited by 3,362
  4. 2019BioBERT: a pre-trained biomedical language representation model for biomedical text miningcited by 1,935
  5. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,830
  6. 2020Big Bird: Transformers for Longer Sequencescited by 1,673
  7. 2020Don't Stop Pretraining: Adapt Language Models to Domains and Taskscited by 1,548
  8. 2020Domain-Specific Language Model Pretraining for Biomedical Natural Language Processingcited by 996
  9. 2019Publicly Available Clinical BERT Embeddingscited by 995
  10. 2019Transfer Learning in Biomedical Natural Language Processing: An Evaluation of BERT and ELMo on Ten Benchmarking Datasetscited by 298
  11. 2018Explainable Prediction of Medical Codes from Clinical Textcited by 231
  12. 2017Super-Convergence: Very Fast Training of Neural Networks Using Large Learning Ratescited by 196
8 more
  1. 2019HIBERT: Document Level Pre-training of Hierarchical Bidirectional Transformers for Document Summarizationcited by 120
  2. 2022Code Synonyms Do Matter: Multiple Synonyms Matching Network for Automatic ICD Codingcited by 52
  3. 2022PLM-ICD: Automatic ICD Coding with Pretrained Language Modelscited by 49
  4. 2022Revisiting Transformer-based Models for Long Document Classificationcited by 48
  5. 2021Does the Magic of BERT Apply to Medical Code Assignment? A Quantitative Studycited by 39
  6. 2020Inferring Degrees from Incomplete Networks and Nonlinear Dynamicscited by 33
  7. 2021Towards BERT-based Automatic ICD Coding: Limitations and Opportunitiescited by 25
  8. 2018Learning to Write Notes in Electronic Health Recordscited by 5

Built on by

Cited by year

Citing papers per year: 2023: 1; 2024: 5; 2025: 1.

7 papers.

  1. 2024CoRelation: Boosting Automatic ICD Coding Through Contextualized Code Relation Learningcited by 17
  2. 2024A Survey on Responsible Generative AI: What to Generate and What Notcited by 8
  3. 2024TempoFormer: A Transformer for Temporally-aware Representations in Change Detectioncited by 5
  4. 2023Sig-Networks Toolkit: Signature Networks for Longitudinal Language Modellingcited by 2
  5. 2024Continuous Predictive Modeling of Clinical Notes and ICD Codes in Patient Health Recordscited by 1
  6. 2024Don't Pay Attention, PLANT It: Pretraining Attention via Learning-to-Rankcited by 0
  7. 2025Predictive Multimodal Modeling of Diagnoses and Treatments in EHRcited by 0