Large AI Models in Health Informatics: Applications, Challenges, and the Future
Not on the map. Why?
Abstract (arXiv)
Large AI models, or foundation models, are models recently emerging with massive scales both parameter-wise and data-wise, the magnitudes of which can reach beyond billions. Once pretrained, large AI models demonstrate impressive performance in various downstream tasks. A prime example is ChatGPT, whose capability has compelled people's imagination about the far-reaching influence that large AI models can have and their potential to transform different domains of our lives. In health informatics, the advent of large AI models has brought new paradigms for the design of methodologies. The scale of multi-modal data in the biomedical and health domain has been ever-expanding especially since the community embraced the era of deep learning, which provides the ground to develop, validate, and advance large AI models for breakthroughs in health-related areas. This article presents a comprehensive review of large AI models, from background to their applications. We identify seven key sectors in which large AI models are applicable and might have substantial influence, including 1) bioinformatics; 2) medical diagnosis; 3) medical imaging; 4) medical informatics; 5) medical education; 6) public health; and 7) medical robotics. We examine their challenges, followed by a critical discussion about potential future directions and pitfalls of large AI models in transforming the field of health informatics.
Key facts. Cited by 61 papers in the map (data through 7 Oct 2026). Builds on 191 of its 348 references that are in the map. First posted on arXiv 21 Mar 2023.
Builds on
191 of 348 references.
References · 191
- 2017Attention Is All You Needcited by 59,932
- 2015Deep Residual Learning for Image Recognitioncited by 51,809
- 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
- 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
- 2020Language Models are Few-Shot Learnerscited by 32,199
- 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
- 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
- 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
- 2023GPT-4 Technical Reportcited by 18,132
- 2022Training language models to follow instructions with human feedbackcited by 15,712
- 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
- 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
38 more
- 2017Proximal Policy Optimization Algorithmscited by 14,441
- 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
- 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
- 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
- 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
- 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
- 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
- 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
- 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
- 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
- 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
- 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,415
- 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
- 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
- 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
- 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
- 2022BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generationcited by 4,310
- 2021Zero-Shot Text-to-Image Generationcited by 4,276
- 2014Going Deeper with Convolutionscited by 4,231
- 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
- 2022Robust Speech Recognition via Large-Scale Weak Supervisioncited by 3,972
- 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
- 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
- 2017Deep reinforcement learning from human preferencescited by 3,846
- 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,595
- 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,590
- 2022A ConvNet for the 2020scited by 3,545
- 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
- 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
- 2022Training Compute-Optimal Large Language Modelscited by 2,927
- 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
- 2016Aggregated Residual Transformations for Deep Neural Networkscited by 2,540
- 2015Trust Region Policy Optimizationcited by 2,503
- 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
- 2022Emergent Abilities of Large Language Modelscited by 2,483
- 2022Do As I Can, Not As I Say: Grounding Language in Robotic Affordancescited by 2,302
- 2021HuBERT: Self-Supervised Speech Representation Learning by Masked Prediction of Hidden Unitscited by 2,301
- 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,222
50 most cited of 191 All 191 (JSON)
Built on by
Citing papers per year: 2023: 25; 2024: 28; 2025: 8.
61 papers.
Sort:
By area:
- 2023Medical SAM Adapter: Adapting Segment Anything Model for Medical Image Segmentationcited by 485
- 2024A Survey on Large Language Model Acceleration based on KV Cache Managementcited by 125
- 2023A Survey of Large Language Models in Medicine: Progress, Application, and Challengecited by 123
- 2023Large Models for Time Series and Spatio-Temporal Data: A Survey and Outlookcited by 109
- 2023LEGO-Prover: Neural Theorem Proving with Growing Librariescited by 108
- 2023EHRSHOT: An EHR Benchmark for Few-Shot Evaluation of Foundation Modelscited by 101
- 2024Foundation Model for Advancing Healthcare: Challenges, Opportunities, and Future Directionscited by 77
- 2024Detecting Multimedia Generated by Large AI Models: A Surveycited by 73
- 2023Generalist Vision Foundation Models for Medical Imaging: A Case Study of Segment Anything Model on Zero-Shot Medical Segmentationcited by 51
- 2024One Prompt Word is Enough to Boost Adversarial Robustness for Pre-trained Vision-Language Modelscited by 48
- 2024A Comprehensive Survey of Large Language Models and Multimodal Large Language Models in Medicinecited by 45
- 2024BitDelta: Your Fine-Tune May Only Be Worth One Bitcited by 38
38 more
- 2023ChatCAD+: Towards a Universal and Reliable Interactive CAD using LLMscited by 35
- 2024MEDCO: Medical Education Copilots Based on A Multi-Agent Frameworkcited by 33
- 2024VLMPC: Vision-Language Model Predictive Control for Robotic Manipulationcited by 32
- 2023Evaluation of GPT-3.5 and GPT-4 for supporting real-world information needs in healthcare deliverycited by 30
- 2024Effectiveness Assessment of Recent Large Vision-Language Modelscited by 29
- 2023Lyra: Orchestrating Dual Correction in Automated Theorem Provingcited by 24
- 2023BioBridge: Bridging Biomedical Foundation Models via Knowledge Graphscited by 24
- 2023VisionFM: a Multi-Modal Multi-Task Vision Foundation Model for Generalist Ophthalmic Artificial Intelligencecited by 24
- 2023Qilin-Med: Multi-stage Knowledge Injection Advanced Medical Large Language Modelcited by 24
- 2023Label-Efficient Deep Learning in Medical Image Analysis: Challenges and Future Directionscited by 14
- 2023PULSAR at MEDIQA-Sum 2023: Large Language Models Augmented by Synthetic Dialogue Convert Patient Dialogues to Medical Recordscited by 11
- 2023Dietary Assessment with Multimodal ChatGPT: A Systematic Analysiscited by 10
- 2024Multimodal Data Integration for Precision Oncology: Challenges and Future Directionscited by 10
- 2024Almanac Copilot: Towards Autonomous Electronic Health Record Navigationcited by 9
- 2024PhenoFlow: A Human-LLM Driven Visual Analytics System for Exploring Large and Complex Stroke Datasetscited by 9
- 2023Large Language Models Illuminate a Progressive Pathway to Artificial Healthcare Assistant: A Reviewcited by 8
- 2024LLaVA-Ultra: Large Chinese Language and Vision Assistant for Ultrasoundcited by 8
- 2025Emerging Cyber Attack Risks of Medical AI Agentscited by 8
- 2024Generalist embedding models are better at short-context clinical semantic search than specialized embedding modelscited by 7
- 2025SynapseRoute: An Auto-Route Switching Framework on Dual-State Large Language Modelcited by 7
- 2023AROID: Improving Adversarial Robustness Through Online Instance-Wise Data Augmentationcited by 6
- 2024On the Workflows and Smells of Leaderboard Operations (LBOps): An Exploratory Study of Foundation Model Leaderboardscited by 6
- 2024AIGC for Industrial Time Series: From Deep Generative Models to Large Generative Modelscited by 6
- 2023Humans are Still Better than ChatGPT: Case of the IEEEXtreme Competitioncited by 5
- 2023CauDR: A Causality-inspired Domain Generalization Framework for Fundus-based Diabetic Retinopathy Gradingcited by 5
- 2024Retrieval Augmented Thought Process for Private Data Handling in Healthcarecited by 5
- 2024CathAction: A Benchmark for Endovascular Intervention Understandingcited by 5
- 2025Science Is Falling Behind the Frontier: Foundation Model Adoption Across Half a Million Paperscited by 5
- 2023Utilizing ChatGPT to Enhance Clinical Trial Enrollmentcited by 4
- 2023A Survey on Image-text Multimodal Modelscited by 4
- 2023Aria-NeRF: Multimodal Egocentric View Synthesiscited by 4
- 2023GlanceSeg: Real-time microaneurysm lesion segmentation with gaze-map-guided foundation model for early detection of diabetic retinopathycited by 4
- 2024Bioinformatics and Biomedical Informatics with ChatGPT: Year One Reviewcited by 4
- 2024DEeR: Deviation Eliminating and Noise Regulating for Privacy-preserving Federated Low-rank Adaptationcited by 4
- 2024High-Throughput Phenotyping of Clinical Text Using Large Language Modelscited by 3
- 2025SingLEM: Single-Channel Large EEG Modelcited by 3
- 2023AI-SAM: Automatic and Interactive Segment Anything Modelcited by 2
- 2024Leveraging Large Language Models for Patient Engagement: The Power of Conversational AI in Digital Healthcited by 2
50 most cited of 61 All 61 (JSON)
- 2025Science Is Falling Behind the Frontier: Foundation Model Adoption Across Half a Million Papers21 Nov 2025
- 2025Diagnosing Shoulder Disorders Using Multimodal Large Language Models and Consumer-Grade Cameras10 Oct 2025
- 2025SingLEM: Single-Channel Large EEG Model22 Sep 2025
- 2025SynapseRoute: An Auto-Route Switching Framework on Dual-State Large Language Model3 Jul 2025
- 2025Brain Imaging Foundation Models, Are We There Yet? A Systematic Review of Foundation Models for Brain Imaging and Biomedical Research16 Jun 2025
- 2025RHealthTwin: Towards Responsible and Multimodal Digital Twins for Personalized Well-being10 Jun 2025
- 2025Emerging Cyber Attack Risks of Medical AI Agents2 Apr 2025
- 2025Rapid Bone Scintigraphy Enhancement via Semantic Prior Distillation from Segment Anything Model4 Mar 2025
- 2024A Survey on Large Language Model Acceleration based on KV Cache Management27 Dec 2024
- 2024A Hybrid Artificial Intelligence System for Automated EEG Background Analysis and Report Generation15 Nov 2024
- 2024Artificial Intelligence for Biomedical Video Generation12 Nov 2024
- 2024The Potential of LLMs in Medical Education: Generating Questions and Answers for Qualification Exams31 Oct 2024
38 more
- 2024LLaVA-Ultra: Large Chinese Language and Vision Assistant for Ultrasound19 Oct 2024
- 2024DEeR: Deviation Eliminating and Noise Regulating for Privacy-preserving Federated Low-rank Adaptation16 Oct 2024
- 2024Patch is Enough: Naturalistic Adversarial Patch against Vision-Language Pre-training Models7 Oct 2024
- 2024CathAction: A Benchmark for Endovascular Intervention Understanding23 Aug 2024
- 2024MEDCO: Medical Education Copilots Based on A Multi-Agent Framework22 Aug 2024
- 2024High-Throughput Phenotyping of Clinical Text Using Large Language Models2 Aug 2024
- 2024PhenoFlow: A Human-LLM Driven Visual Analytics System for Exploring Large and Complex Stroke Datasets23 Jul 2024
- 2024AIGC for Industrial Time Series: From Deep Generative Models to Large Generative Models16 Jul 2024
- 2024VLMPC: Vision-Language Model Predictive Control for Robotic Manipulation13 Jul 2024
- 2024On the Workflows and Smells of Leaderboard Operations (LBOps): An Exploratory Study of Foundation Model Leaderboards4 Jul 2024
- 2024Multimodal Data Integration for Precision Oncology: Challenges and Future Directions28 Jun 2024
- 2024Leveraging Large Language Models for Patient Engagement: The Power of Conversational AI in Digital Health19 Jun 2024
- 2024A Comprehensive Survey of Large Language Models and Multimodal Large Language Models in Medicine14 May 2024
- 2024Almanac Copilot: Towards Autonomous Electronic Health Record Navigation30 Apr 2024
- 2024BC-MRI-SEG: A Breast Cancer MRI Tumor Segmentation Benchmark21 Apr 2024
- 2024Foundation Model for Advancing Healthcare: Challenges, Opportunities, and Future Directions4 Apr 2024
- 2024Bioinformatics and Biomedical Informatics with ChatGPT: Year One Review22 Mar 2024
- 2024Effectiveness Assessment of Recent Large Vision-Language Models7 Mar 2024
- 2024One Prompt Word is Enough to Boost Adversarial Robustness for Pre-trained Vision-Language Models4 Mar 2024
- 2024BitDelta: Your Fine-Tune May Only Be Worth One Bit15 Feb 2024
- 2024Retrieval Augmented Thought Process for Private Data Handling in Healthcare12 Feb 2024
- 2024Evaluation of General Large Language Models in Contextually Assessing Semantic Concepts Extracted from Adult Critical Care Electronic Health Record Notes24 Jan 2024
- 2024Detecting Multimedia Generated by Large AI Models: A Survey22 Jan 2024
- 2024Generalist embedding models are better at short-context clinical semantic search than specialized embedding models3 Jan 2024
- 2023Dietary Assessment with Multimodal ChatGPT: A Systematic Analysis14 Dec 2023
- 2023AI-SAM: Automatic and Interactive Segment Anything Model5 Dec 2023
- 2023GlanceSeg: Real-time microaneurysm lesion segmentation with gaze-map-guided foundation model for early detection of diabetic retinopathy14 Nov 2023
- 2023Aria-NeRF: Multimodal Egocentric View Synthesis11 Nov 2023
- 2023A Survey of Large Language Models in Medicine: Progress, Application, and Challenge9 Nov 2023
- 2023Large Language Models Illuminate a Progressive Pathway to Artificial Healthcare Assistant: A Review3 Nov 2023
- 2023Large Models for Time Series and Spatio-Temporal Data: A Survey and Outlook16 Oct 2023
- 2023Qilin-Med: Multi-stage Knowledge Injection Advanced Medical Large Language Model13 Oct 2023
- 2023Multi-domain improves out-of-distribution and data-limited scenarios for medical image analysis10 Oct 2023
- 2023VisionFM: a Multi-Modal Multi-Task Vision Foundation Model for Generalist Ophthalmic Artificial Intelligence8 Oct 2023
- 2023BioBridge: Bridging Biomedical Foundation Models via Knowledge Graphs5 Oct 2023
- 2023LEGO-Prover: Neural Theorem Proving with Growing Libraries1 Oct 2023
- 2023Lyra: Orchestrating Dual Correction in Automated Theorem Proving27 Sep 2023
- 2023CauDR: A Causality-inspired Domain Generalization Framework for Fundus-based Diabetic Retinopathy Grading27 Sep 2023
50 newest of 61 All 61 (JSON)
- 2024One Prompt Word is Enough to Boost Adversarial Robustness for Pre-trained Vision-Language Modelscited by 48
- 2024A Comprehensive Survey of Large Language Models and Multimodal Large Language Models in Medicinecited by 45
- 2024VLMPC: Vision-Language Model Predictive Control for Robotic Manipulationcited by 32
- 2024Effectiveness Assessment of Recent Large Vision-Language Modelscited by 29
- 2024LLaVA-Ultra: Large Chinese Language and Vision Assistant for Ultrasoundcited by 8
- 2024Patch is Enough: Naturalistic Adversarial Patch against Vision-Language Pre-training Modelscited by 0
- 2024LLaVA-Ultra: Large Chinese Language and Vision Assistant for Ultrasound19 Oct 2024
- 2024Patch is Enough: Naturalistic Adversarial Patch against Vision-Language Pre-training Models7 Oct 2024
- 2024VLMPC: Vision-Language Model Predictive Control for Robotic Manipulation13 Jul 2024
- 2024A Comprehensive Survey of Large Language Models and Multimodal Large Language Models in Medicine14 May 2024
- 2024Effectiveness Assessment of Recent Large Vision-Language Models7 Mar 2024
- 2024One Prompt Word is Enough to Boost Adversarial Robustness for Pre-trained Vision-Language Models4 Mar 2024
- 2024MEDCO: Medical Education Copilots Based on A Multi-Agent Frameworkcited by 33
- 2025Emerging Cyber Attack Risks of Medical AI Agentscited by 8
- 2025Emerging Cyber Attack Risks of Medical AI Agents2 Apr 2025
- 2024MEDCO: Medical Education Copilots Based on A Multi-Agent Framework22 Aug 2024
- 2024Detecting Multimedia Generated by Large AI Models: A Surveycited by 73
- 2024Detecting Multimedia Generated by Large AI Models: A Survey22 Jan 2024
- 2023Medical SAM Adapter: Adapting Segment Anything Model for Medical Image Segmentationcited by 485
- 2023A Survey of Large Language Models in Medicine: Progress, Application, and Challengecited by 123
- 2023Large Models for Time Series and Spatio-Temporal Data: A Survey and Outlookcited by 109
- 2023LEGO-Prover: Neural Theorem Proving with Growing Librariescited by 108
- 2023EHRSHOT: An EHR Benchmark for Few-Shot Evaluation of Foundation Modelscited by 101
- 2024Foundation Model for Advancing Healthcare: Challenges, Opportunities, and Future Directionscited by 77
- 2023Generalist Vision Foundation Models for Medical Imaging: A Case Study of Segment Anything Model on Zero-Shot Medical Segmentationcited by 51
- 2024BitDelta: Your Fine-Tune May Only Be Worth One Bitcited by 38
- 2023ChatCAD+: Towards a Universal and Reliable Interactive CAD using LLMscited by 35
- 2023Evaluation of GPT-3.5 and GPT-4 for supporting real-world information needs in healthcare deliverycited by 30
- 2023Lyra: Orchestrating Dual Correction in Automated Theorem Provingcited by 24
- 2023BioBridge: Bridging Biomedical Foundation Models via Knowledge Graphscited by 24
12 most cited of 49 All 49 (JSON)
- 2025Science Is Falling Behind the Frontier: Foundation Model Adoption Across Half a Million Papers21 Nov 2025
- 2025Diagnosing Shoulder Disorders Using Multimodal Large Language Models and Consumer-Grade Cameras10 Oct 2025
- 2025SingLEM: Single-Channel Large EEG Model22 Sep 2025
- 2025Brain Imaging Foundation Models, Are We There Yet? A Systematic Review of Foundation Models for Brain Imaging and Biomedical Research16 Jun 2025
- 2025RHealthTwin: Towards Responsible and Multimodal Digital Twins for Personalized Well-being10 Jun 2025
- 2025Rapid Bone Scintigraphy Enhancement via Semantic Prior Distillation from Segment Anything Model4 Mar 2025
- 2024A Hybrid Artificial Intelligence System for Automated EEG Background Analysis and Report Generation15 Nov 2024
- 2024Artificial Intelligence for Biomedical Video Generation12 Nov 2024
- 2024The Potential of LLMs in Medical Education: Generating Questions and Answers for Qualification Exams31 Oct 2024
- 2024DEeR: Deviation Eliminating and Noise Regulating for Privacy-preserving Federated Low-rank Adaptation16 Oct 2024
- 2024CathAction: A Benchmark for Endovascular Intervention Understanding23 Aug 2024
- 2024High-Throughput Phenotyping of Clinical Text Using Large Language Models2 Aug 2024
12 newest of 49 All 49 (JSON)