mlsanityData to 7 Oct 2026

Towards Generalist Biomedical AI

arXiv 2307.14334 ↗First posted 26 Jul 2023cs.CL cs.CVcited by 286

Not on the map. Why?

Cited by 80 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Medicine is inherently multimodal, with rich data modalities spanning text, imaging, genomics, and more. Generalist biomedical artificial intelligence (AI) systems that flexibly encode, integrate, and interpret this data at scale can potentially enable impactful applications ranging from scientific discovery to care delivery. To enable the development of these models, we first curate MultiMedBench, a new multimodal biomedical benchmark. MultiMedBench encompasses 14 diverse tasks such as medical question answering, mammography and dermatology image interpretation, radiology report generation and summarization, and genomic variant calling. We then introduce Med-PaLM Multimodal (Med-PaLM M), our proof of concept for a generalist biomedical AI system. Med-PaLM M is a large multimodal generative model that flexibly encodes and interprets biomedical data including clinical language, imaging, and genomics with the same set of model weights. Med-PaLM M reaches performance competitive with or exceeding the state of the art on all MultiMedBench tasks, often surpassing specialist models by a wide margin. We also report examples of zero-shot generalization to novel medical concepts and tasks, positive transfer learning across tasks, and emergent zero-shot medical reasoning. To further probe the capabilities and limitations of Med-PaLM M, we conduct a radiologist evaluation of model-generated (and human) chest X-ray reports and observe encouraging performance across model scales. In a side-by-side ranking on 246 retrospective chest X-rays, clinicians express a pairwise preference for Med-PaLM M reports over those produced by radiologists in up to 40.50% of cases, suggesting potential clinical utility. While considerable work is needed to validate these models in real-world use cases, our results represent a milestone towards the development of generalist biomedical AI systems.

Key facts. Cited by 286 papers in the map (data through 7 Oct 2026). Builds on 71 of its 131 references that are in the map. First posted on arXiv 26 Jul 2023.

Builds on

71 of 131 references.

References · 71

  1. 2017Attention Is All You Needcited by 59,932
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  6. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  7. 2020Scaling Laws for Neural Language Modelscited by 6,411
  8. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  9. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  10. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  11. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  12. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,072
38 more
  1. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  2. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  3. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,579
  4. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  5. 2022Emergent Abilities of Large Language Modelscited by 2,483
  6. 2019RandAugment: Practical automated data augmentation with a reduced search spacecited by 2,261
  7. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,037
  8. 2014CIDEr: Consensus-based Image Description Evaluationcited by 2,027
  9. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,970
  10. 2022Large Language Models Encode Clinical Knowledgecited by 1,807
  11. 2019CheXpert: A Large Chest Radiograph Dataset with Uncertainty Labels and Expert Comparisoncited by 1,410
  12. 2020What Disease does this Patient Have? A Large-scale Open Domain Question Answering Dataset from Medical Examscited by 1,324
  13. 2023LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Daycited by 1,287
  14. 2019PubMedQA: A Dataset for Biomedical Research Question Answeringcited by 1,271
  15. 2019OK-VQA: A Visual Question Answering Benchmark Requiring External Knowledgecited by 1,114
  16. 2021Perceiver: General Perception with Iterative Attentioncited by 1,089
  17. 2021Scaling Vision Transformerscited by 993
  18. 2017Revisiting Unreasonable Effectiveness of Data in Deep Learning Eracited by 916
  19. 2018Adafactor: Adaptive Learning Rates with Sublinear Memory Costcited by 896
  20. 2022A Generalist Agentcited by 802
  21. 2017An Overview of Multi-Task Learning in Deep Neural Networkscited by 791
  22. 2023LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attentioncited by 776
  23. 2022PaLI: A Jointly-Scaled Multilingual Language-Image Modelcited by 762
  24. 2023Scaling Vision Transformers to 22 Billion Parameterscited by 690
  25. 2021Multimodal Few-Shot Learning with Frozen Language Modelscited by 657
  26. 2022AudioLM: a Language Modeling Approach to Audio Generationcited by 611
  27. 2022MedMCQA : A Large-scale Multi-Subject Multi-Choice Dataset for Medical domain Question Answeringcited by 525
  28. 2020PathVQA: 30000+ Questions for Medical Visual Question Answeringcited by 467
  29. 2021SLAKE: A Semantically-Labeled Knowledge-Enhanced Dataset for Medical Visual Question Answeringcited by 444
  30. 2023MusicLM: Generating Music From Textcited by 442
  31. 2022Unified-IO: A Unified Model for Vision, Language, and Multi-Modal Taskscited by 420
  32. 2023Towards Expert-Level Medical Question Answering with Large Language Modelscited by 382
  33. 2020Generating Radiology Reports via Memory-driven Transformercited by 363
  34. 2021Pix2seq: A Language Modeling Framework for Object Detectioncited by 345
  35. 2020CheXbert: Combining Automatic Labelers and Expert Annotations for Accurate Radiology Report Labeling Using BERTcited by 329
  36. 2021RadGraph: Extracting Clinical Entities and Relations from Radiology Reportscited by 310
  37. 2023Multimodal Deep Learningcited by 257
  38. 2021ViTAE: Vision Transformer Advanced by Exploring Intrinsic Inductive Biascited by 216

50 most cited of 71 All 71 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 33; 2024: 91; 2025: 90; 2026: 72.

286 papers.

Sort:

By area:

  1. 2024VideoLLaMA 2: Advancing Spatial-Temporal Modeling and Audio Understanding in Video-LLMscited by 749
  2. 2025HealthBench: Evaluating Large Language Models Towards Improved Human Healthcited by 385
  3. 2023MedAgents: Large Language Models as Collaborators for Zero-shot Medical Reasoningcited by 311
  4. 2023Multimodal Foundation Models: From Specialists to General-Purpose Assistantscited by 247
  5. 2023Adapted Large Language Models Can Outperform Medical Experts in Clinical Text Summarizationcited by 244
  6. 2023Towards Generalist Foundation Model for Radiology by Leveraging Web-scale 2D&3D Medical Datacited by 176
  7. 2024M3D: Advancing 3D Medical Image Analysis with Multi-Modal Large Language Modelscited by 154
  8. 2024MAIRA-2: Grounded Radiology Report Generationcited by 140
  9. 2024Exploring scalable medical image encoders beyond text supervisioncited by 129
  10. 2023A Survey of Large Language Models in Medicine: Progress, Application, and Challengecited by 123
  11. 2023Towards Accurate Differential Diagnosis with Large Language Modelscited by 115
  12. 2025Baichuan-Omni-1.5 Technical Reportcited by 107
38 more
  1. 2024On-Device Language Models: A Comprehensive Reviewcited by 101
  2. 2024Efficient Multimodal Large Language Models: A Surveycited by 95
  3. 2023Conversational Health Agents: A Personalized LLM-Powered Agent Frameworkcited by 91
  4. 2024Towards Building Multilingual Language Model for Medicinecited by 91
  5. 2024MMed-RAG: Versatile Multimodal RAG System for Medical Vision Language Modelscited by 84
  6. 2024MedTrinity-25M: A Large-scale Multimodal Dataset with Multigranular Annotations for Medicinecited by 79
  7. 2023Can GPT-4V(ision) Serve Medical Applications? Case Studies on GPT-4V for Multimodal Medical Diagnosiscited by 77
  8. 2024Foundation Model for Advancing Healthcare: Challenges, Opportunities, and Future Directionscited by 77
  9. 2024Towards Modular LLMs by Building and Reusing a Library of LoRAscited by 72
  10. 2024CARES: A Comprehensive Benchmark of Trustworthiness in Medical Vision Language Modelscited by 64
  11. 2024Towards a clinically accessible radiology foundation model: open-access and lightweight, with automated evaluationcited by 51
  12. 2025Rewarding Doubt: A Reinforcement Learning Approach to Calibrated Confidence Expression of Large Language Modelscited by 48
  13. 2024A Comprehensive Survey of Large Language Models and Multimodal Large Language Models in Medicinecited by 45
  14. 2024MedImageInsight: An Open-Source Embedding Model for General Domain Medical Imagingcited by 45
  15. 2025Med3DVLM: An Efficient Vision-Language Model for 3D Medical Image Analysiscited by 44
  16. 2024SmallToLarge (S2L): Scalable Data Selection for Fine-tuning Large Language Models by Summarizing Training Trajectories of Small Modelscited by 43
  17. 2024RadGenome-Chest CT: A Grounded Vision-Language Dataset for Chest CT Analysiscited by 41
  18. 2024RudolfV: A Foundation Model by Pathologists for Pathologistscited by 40
  19. 2023Multimodal ChatGPT for Medical Applications: an Experimental Study of GPT-4Vcited by 38
  20. 2025EndoChat: Grounded Multimodal Large Language Model for Endoscopic Surgerycited by 37
  21. 2025MMedAgent-RL: Optimizing Multi-Agent Collaboration for Multimodal Medical Reasoningcited by 37
  22. 2025RadGPT: Constructing 3D Image-Text Tumor Datasetscited by 34
  23. 2025MedFrameQA: A Multi-Image Medical VQA Benchmark for Clinical Reasoningcited by 33
  24. 2023Computational Pathology at Health System Scale -- Self-Supervised Foundation Models from Three Billion Imagescited by 30
  25. 2024Towards a Holistic Framework for Multimodal Large Language Models in Three-dimensional Brain CT Report Generationcited by 30
  26. 2023DRG-LLaMA : Tuning LLaMA Model to Predict Diagnosis-related Group for Hospitalized Patientscited by 29
  27. 2024MultiMedEval: A Benchmark and a Toolkit for Evaluating Medical Vision-Language Modelscited by 28
  28. 2024Foundation Models in Radiology: What, How, When, Why and Why Notcited by 26
  29. 2025Evidence-based diagnostic reasoning with multi-agent copilot for human pathologycited by 26
  30. 2024Towards a Personal Health Large Language Modelcited by 25
  31. 2024Large Language Models in Drug Discovery and Development: From Disease Mechanisms to Clinical Trialscited by 24
  32. 2024Interpretable Bilingual Multimodal Large Language Model for Diverse Biomedical Taskscited by 24
  33. 2023MarineGPT: Unlocking Secrets of Ocean to the Publiccited by 23
  34. 2024AutoRG-Brain: Grounded Report Generation for Brain MRIcited by 23
  35. 2024An X-Ray Is Worth 15 Features: Sparse Autoencoders for Interpretable Radiology Report Generationcited by 23
  36. 2024GMAI-VL & GMAI-VL-5.5M: A Large Vision-Language Model and A Comprehensive Multimodal Dataset Towards General Medical AIcited by 23
  37. 2025Triad: Vision Foundation Model for 3D Magnetic Resonance Imagingcited by 23
  38. 2023Large language models in medicine: the potentials and pitfallscited by 21

50 most cited of 286 All 286 (JSON)