mlsanityData to 7 Oct 2026

BERT for Long Documents: A Case Study of Automated ICD Coding

arXiv 2211.02519 ↗First posted 4 Nov 2022cs.CL cs.LGcited by 4

Not on the map. Why?

Abstract (arXiv)

Transformer models have achieved great success across many NLP problems. However, previous studies in automated ICD coding concluded that these models fail to outperform some of the earlier solutions such as CNN-based models. In this paper we challenge this conclusion. We present a simple and scalable method to process long text with the existing transformer models such as BERT. We show that this method significantly improves the previous results reported for transformer models in ICD coding, and is able to outperform one of the prominent CNN-based methods.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Builds on 16 of its 28 references that are in the map. First posted on arXiv 4 Nov 2022.

Builds on

16 of 28 references.

References · 16

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  4. 2020Longformer: The Long-Document Transformercited by 3,362
  5. 2019BioBERT: a pre-trained biomedical language representation model for biomedical text miningcited by 1,935
  6. 2020Big Bird: Transformers for Longer Sequencescited by 1,673
  7. 2020Efficient Transformers: A Surveycited by 784
  8. 2021LongT5: Efficient Text-To-Text Transformer for Long Sequencescited by 232
  9. 2018Explainable Prediction of Medical Codes from Clinical Textcited by 231
  10. 2019ICD Coding from Clinical Text Using Multi-Filter Residual Convolutional Neural Networkcited by 73
  11. 2021Does the Magic of BERT Apply to Medical Code Assignment? A Quantitative Studycited by 39
  12. 2017Multi-Label Classification of Patient Notes a Case Study on ICD Code Assignmentcited by 37
4 more
  1. 2020BERT-XML: Large Scale Automated ICD Coding Using BERT Pretrainingcited by 37
  2. 2021Towards BERT-based Automatic ICD Coding: Limitations and Opportunitiescited by 25
  3. 2020Multi-label natural language processing to identify diagnosis and procedure codes from MIMIC-III inpatient notescited by 4
  4. 2020Prediction of ICD Codes with Clinical BERT Embeddings and Text Augmentation with Label Balancing using MIMIC-IIIcited by 4

Built on by

Cited by year

Citing papers per year: 2024: 3; 2025: 1.

4 papers.

  1. 2024Can GPT-3.5 Generate and Code Discharge Summaries?cited by 12
  2. 2024CiMaTe: Citation Count Prediction Effectively Leveraging the Main Textcited by 3
  3. 2024Inverse Probability of Treatment Weighting with Deep Sequence Models Enables Accurate treatment effect Estimation from Electronic Health Recordscited by 0
  4. 2025MKE-Coder: Multi-Axial Knowledge with Evidence Verification in ICD Coding for Chinese EMRscited by 0