mlsanityData to 7 Oct 2026

Has Multimodal Learning Delivered Universal Intelligence in Healthcare? A Comprehensive Survey

arXiv 2408.12880 ↗First posted 23 Aug 2024cs.AI cs.MMcited by 20

Not on the map. Why?

Abstract (arXiv)

The rapid development of artificial intelligence has constantly reshaped the field of intelligent healthcare and medicine. As a vital technology, multimodal learning has increasingly garnered interest due to data complementarity, comprehensive modeling form, and great application potential. Currently, numerous researchers are dedicating their attention to this field, conducting extensive studies and constructing abundant intelligent systems. Naturally, an open question arises that has multimodal learning delivered universal intelligence in healthcare? To answer the question, we adopt three unique viewpoints for a holistic analysis. Firstly, we conduct a comprehensive survey of the current progress of medical multimodal learning from the perspectives of datasets, task-oriented methods, and universal foundation models. Based on them, we further discuss the proposed question from five issues to explore the real impacts of advanced techniques in healthcare, from data and technologies to performance and ethics. The answer is that current technologies have NOT achieved universal intelligence and there remains a significant journey to undertake. Finally, in light of the above reviews and discussions, we point out ten potential directions for exploration towards the goal of universal intelligence in healthcare.

Key facts. Cited by 20 papers in the map (data through 7 Oct 2026). Builds on 128 of its 208 references that are in the map. First posted on arXiv 23 Aug 2024.

Builds on

128 of 208 references.

References · 128

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  6. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  7. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  8. 2017Proximal Policy Optimization Algorithmscited by 14,441
  9. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  10. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  11. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  12. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
38 more
  1. 2016Semi-Supervised Classification with Graph Convolutional Networkscited by 11,282
  2. 2023Visual Instruction Tuningcited by 7,792
  3. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  4. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  5. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,466
  6. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  7. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  8. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  9. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  10. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,229
  11. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,155
  12. 2023Mistral 7Bcited by 3,065
  13. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  14. 2023MiniGPT-4: Enhancing Vision-Language Understanding with Advanced Large Language Modelscited by 2,689
  15. 2018YOLOv3: An Incremental Improvementcited by 2,652
  16. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,881
  17. 2024Mixtral of Expertscited by 1,785
  18. 20163D U-Net: Learning Dense Volumetric Segmentation from Sparse Annotationcited by 1,471
  19. 2023LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Daycited by 1,287
  20. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,265
  21. 2017Multimodal Machine Learning: A Survey and Taxonomycited by 1,004
  22. 2020Domain-Specific Language Model Pretraining for Biomedical Natural Language Processingcited by 996
  23. 2023A Survey on Multimodal Large Language Modelscited by 898
  24. 2023Graph of Thoughts: Solving Elaborate Problems with Large Language Modelscited by 873
  25. 2023Multimodal Chain-of-Thought Reasoning in Language Modelscited by 740
  26. 2022EVA: Exploring the Limits of Masked Visual Representation Learning at Scalecited by 685
  27. 2020Contrastive Learning of Medical Visual Representations from Paired Images and Textcited by 636
  28. 2023NExT-GPT: Any-to-Any Multimodal LLMcited by 610
  29. 2022MedCLIP: Contrastive Learning from Unpaired Medical Images and Textcited by 564
  30. 2022BioGPT: Generative Pre-trained Transformer for Biomedical Text Generation and Miningcited by 531
  31. 2023BiomedCLIP: a multimodal biomedical foundation model pretrained from fifteen million scientific image-text pairscited by 506
  32. 2020PathVQA: 30000+ Questions for Medical Visual Question Answeringcited by 467
  33. 2021SLAKE: A Semantically-Labeled Knowledge-Enhanced Dataset for Medical Visual Question Answeringcited by 444
  34. 2023Med-Flamingo: a Multimodal Medical Few-shot Learnercited by 387
  35. 2023Towards Expert-Level Medical Question Answering with Large Language Modelscited by 382
  36. 2024Hallucination of Multimodal Large Language Models: A Surveycited by 380
  37. 2024Mini-Gemini: Mining the Potential of Multi-modality Vision Language Modelscited by 361
  38. 2021High-Performance Large-Scale Image Recognition Without Normalizationcited by 359

50 most cited of 128 All 128 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 11; 2026: 8.

20 papers.

Sort:

By area:

  1. 2024Look Twice Before You Answer: Memory-Space Visual Retracing for Hallucination Mitigation in Multimodal Large Language Modelscited by 72
  2. 2025CodeBrain: Bridging Decoupled Tokenizer and Multi-Scale Architecture for EEG Foundation Modelcited by 23
  3. 2025Self-supervised Quantized Representation for Seamlessly Integrating Knowledge Graphs with Large Language Modelscited by 11
  4. 2025GKG-LLM: A Unified Framework for Generalized Knowledge Graph Constructioncited by 6
  5. 2025Toward Clinically Explainable AI for Medical Diagnosis: A Foundation Model with Human-Compatible Reasoning via Reinforcement Learningcited by 5
  6. 2025Affective-ROPTester: Capability and Bias Analysis of LLMs in Predicting Retinopathy of Prematuritycited by 3
  7. 2025Generalist Models in Medical Image Segmentation: A Survey and Performance Comparison with Task-Specific Approachescited by 2
  8. 2026An Empirical Analysis of Calibration and Selective Prediction in Multimodal Clinical Condition Classificationcited by 2
  9. 2025COMMA: Coordinate-aware Modulated Mamba Network for 3D Dispersed Vessel Segmentationcited by 1
  10. 2025Enhancing Multi-task Learning Capability of Medical Generalist Foundation Model via Image-centric Multi-annotation Datacited by 1
  11. 2025Robust Multimodal Sentiment Analysis via Double Information Bottleneckcited by 1
  12. 2026When Looking Is Not Enough: Visual Attention Structure Reveals Hallucination in MLLMscited by 1
8 more
  1. 2025SEPS: Semantic-enhanced Patch Slimming Framework for fine-grained cross-modal alignmentcited by 0
  2. 2025Transferring Clinical Knowledge into ECGs Representationcited by 0
  3. 2026Towards Efficient and Robust Linguistic Emotion Diagnosis for Mental Health via Multi-Agent Instruction Refinementcited by 0
  4. 2026MedForge: Interpretable Medical Deepfake Detection via Forgery-aware Reasoningcited by 0
  5. 2026Instruction-Free Tuning of Large Vision Language Models for Medical Instruction Followingcited by 0
  6. 2026Relaxing Anchor-Frame Dominance for Mitigating Hallucinations in Video Large Language Modelscited by 0
  7. 2026From Static Risk to Dynamic Trajectories: Toward World-Model-Inspired Clinical Predictioncited by 0
  8. 2026Dynamic Alignment Compensation for Hallucination Mitigation in Large Vision-Language Modelscited by 0