mlsanityData to 7 Oct 2026

Transformers in Healthcare: A Survey

arXiv 2307.00067 ↗First posted 30 Jun 2023cs.AI cs.CY cs.LGcited by 17

Not on the map. Why?

Abstract (arXiv)

With Artificial Intelligence (AI) increasingly permeating various aspects of society, including healthcare, the adoption of the Transformers neural network architecture is rapidly changing many applications. Transformer is a type of deep learning architecture initially developed to solve general-purpose Natural Language Processing (NLP) tasks and has subsequently been adapted in many fields, including healthcare. In this survey paper, we provide an overview of how this architecture has been adopted to analyze various forms of data, including medical imaging, structured and unstructured Electronic Health Records (EHR), social media, physiological signals, and biomolecular sequences. Those models could help in clinical diagnosis, report generation, data reconstruction, and drug/protein synthesis. We identified relevant studies using the Preferred Reporting Items for Systematic Reviews and Meta-Analyses (PRISMA) guidelines. We also discuss the benefits and limitations of using transformers in healthcare and examine issues such as computational cost, model interpretability, fairness, alignment with human values, ethical implications, and environmental impact.

Key facts. Cited by 17 papers in the map (data through 7 Oct 2026). Builds on 223 of its 495 references that are in the map. First posted on arXiv 30 Jun 2023.

Builds on

223 of 495 references.

References · 223

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  6. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  7. 2023GPT-4 Technical Reportcited by 18,132
  8. 2022Training language models to follow instructions with human feedbackcited by 15,712
  9. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  10. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  11. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  12. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
38 more
  1. 2016Densely Connected Convolutional Networkscited by 6,691
  2. 2022Generative Adversarial Networkscited by 5,603
  3. 2016Layer Normalizationcited by 5,579
  4. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  5. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  6. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  7. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  8. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  9. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  10. 2021SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformerscited by 2,923
  11. 2015Neural Machine Translation of Rare Words with Subword Unitscited by 2,898
  12. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,801
  13. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
  14. 2022Emergent Abilities of Large Language Modelscited by 2,483
  15. 2018Deep contextualized word representationscited by 2,285
  16. 2016V-Net: Fully Convolutional Neural Networks for Volumetric Medical Image Segmentationcited by 2,284
  17. 2022Survey of Hallucination in Natural Language Generationcited by 2,056
  18. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,027
  19. 2021Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutionscited by 1,960
  20. 2019BioBERT: a pre-trained biomedical language representation model for biomedical text miningcited by 1,935
  21. 2022Large Language Models Encode Clinical Knowledgecited by 1,807
  22. 2020Rethinking Attention with Performerscited by 1,584
  23. 2016Enriching Word Vectors with Subword Informationcited by 1,578
  24. 2021TransUNet: Transformers Make Strong Encoders for Medical Image Segmentationcited by 1,552
  25. 2018Know What You Don't Know: Unanswerable Questions for SQuADcited by 1,538
  26. 2019CheXpert: A Large Chest Radiograph Dataset with Uncertainty Labels and Expert Comparisoncited by 1,410
  27. 2019SciBERT: A Pretrained Language Model for Scientific Textcited by 1,404
  28. 2019PubMedQA: A Dataset for Biomedical Research Question Answeringcited by 1,271
  29. 2019Energy and Policy Considerations for Deep Learning in NLPcited by 1,242
  30. 2020RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Modelscited by 1,235
  31. 2021Swin-Unet: Unet-like Pure Transformer for Medical Image Segmentationcited by 1,218
  32. 2023PaLM 2 Technical Reportcited by 1,154
  33. 2017Deep Image Priorcited by 1,097
  34. 2019Cross-lingual Language Model Pretrainingcited by 1,088
  35. 2021Transformers in Vision: A Surveycited by 1,009
  36. 2020Domain-Specific Language Model Pretraining for Biomedical Natural Language Processingcited by 996
  37. 2019Publicly Available Clinical BERT Embeddingscited by 995
  38. 2019What Does BERT Look At? An Analysis of BERT's Attentioncited by 994

50 most cited of 223 All 223 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 5; 2025: 8; 2026: 3.

17 papers.

Sort:

By area:

  1. 2024Get More with LESS: Synthesizing Recurrence with KV Cache Compression for Efficient LLM Inferencecited by 75
  2. 2024Chain-of-Interaction: Enhancing Large Language Models for Psychiatric Behavior Understanding by Dyadic Contextscited by 19
  3. 2025CogniAlign: Word-Level Multimodal Speech Alignment with Gated Cross-Attention for Alzheimer's Detectioncited by 6
  4. 2024Length-Aware Multi-Kernel Transformer for Long Document Classificationcited by 5
  5. 2025Advanced Deep Learning and Large Language Models: Comprehensive Insights for Cancer Detectioncited by 4
  6. 2025Interpretable Auto Window Setting for Deep-Learning-Based CT Analysiscited by 2
  7. 2025ChestGPT: Integrating Large Language Models and Vision Transformers for Disease Detection and Localization in Chest X-Rayscited by 1
  8. 2025MedBayes-Lite: A Clinical Uncertainty Governance Layer for Risk-Aware Medical Decision Supportcited by 1
  9. 2026On the Expressive Power of Transformers for Maxout Networks and Continuous Piecewise Linear Functionscited by 1
  10. 2023The Potential of Wearable Sensors for Assessing Patient Acuity in Intensive Care Unit (ICU)cited by 0
  11. 2024COMFORT: A Continual Fine-Tuning Framework for Foundation Models Targeted at Consumer Healthcarecited by 0
  12. 2024Don't Pay Attention, PLANT It: Pretraining Attention via Learning-to-Rankcited by 0
5 more
  1. 2025Do LLMs Triage Like Clinicians? A Dynamic Study of Outpatient Referralcited by 0
  2. 2025A Laplace diffusion-based transformer model for heart rate forecasting within daily activity contextcited by 0
  3. 2025An Information-Theoretic Framework for Robust Large Language Model Editingcited by 0
  4. 2026CaresAI at CT-DEB26: Detecting Dosing Errors In Clinical Trials Using Domain-Specific Transformer Embeddings and Classification Modelscited by 0
  5. 2026MedGate-Fusion: Integrating First-Encounter Semantic Narratives and Physiological Biomarkers for Prospective Stroke Risk Stratificationcited by 0