mlsanityData to 7 Oct 2026

Multimodal LLMs for health grounded in individual-specific data

arXiv 2307.09018 ↗First posted 18 Jul 2023q-bio.QM cs.LGcited by 48

On the map: Multimodal LLMs / vision-language models → Medical vision-language models

Abstract (arXiv)

Foundation large language models (LLMs) have shown an impressive ability to solve tasks across a wide range of fields including health. To effectively solve personalized health tasks, LLMs need the ability to ingest a diversity of data modalities that are relevant to an individual's health status. In this paper, we take a step towards creating multimodal LLMs for health that are grounded in individual-specific data by developing a framework (HeLM: Health Large Language Model for Multimodal Understanding) that enables LLMs to use high-dimensional clinical modalities to estimate underlying disease risk. HeLM encodes complex data modalities by learning an encoder that maps them into the LLM's token embedding space and for simple modalities like tabular data by serializing the data into text. Using data from the UK Biobank, we show that HeLM can effectively use demographic and clinical features in addition to high-dimensional time-series data to estimate disease risk. For example, HeLM achieves an AUROC of 0.75 for asthma prediction when combining tabular and spirogram data modalities compared with 0.49 when only using tabular data. Overall, we find that HeLM outperforms or performs at parity with classical machine learning approaches across a selection of eight binary traits. Furthermore, we investigate the downstream uses of this model such as its generalizability to out-of-distribution traits and its ability to power conversations around individual health and wellness.

Key facts. Cited by 48 papers in the map (data through 7 Oct 2026). Placed in Medical vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 27 of its 40 references that are in the map. First posted on arXiv 18 Jul 2023.

Builds on

27 of 40 references.

References · 27

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2023GPT-4 Technical Reportcited by 18,132
  5. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  6. 2016XGBoost: A Scalable Tree Boosting Systemcited by 5,120
  7. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  8. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  9. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  10. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
  11. 2022Emergent Abilities of Large Language Modelscited by 2,483
  12. 2023PaLM-E: An Embodied Multimodal Language Modelcited by 2,037
15 more
  1. 2022Large Language Models Encode Clinical Knowledgecited by 1,807
  2. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,265
  3. 2023ImageBind: One Embedding Space To Bind Them Allcited by 1,162
  4. 2023PaLM 2 Technical Reportcited by 1,154
  5. 2018Bag of Tricks for Image Classification with Convolutional Neural Networkscited by 430
  6. 2023LaMP: When Large Language Models Meet Personalizationcited by 425
  7. 2022Unified-IO: A Unified Model for Vision, Language, and Multi-Modal Taskscited by 420
  8. 2023Towards Expert-Level Medical Question Answering with Large Language Modelscited by 382
  9. 2022TabLLM: Few-shot Classification of Tabular Data with Large Language Modelscited by 273
  10. 2022GatorTron: A Large Clinical Language Model to Unlock Patient Information from Unstructured Electronic Health Recordscited by 238
  11. 2018Do CIFAR-10 Classifiers Generalize to CIFAR-10?cited by 197
  12. 2022LIFT: Language-Interfaced Fine-Tuning for Non-Language Machine Learning Taskscited by 146
  13. 2023Personalisation within bounds: A risk taxonomy and policy framework for the alignment of large language models with personalised feedbackcited by 86
  14. 2020Language Models Are An Effective Patient Representation Learning Technique For Electronic Health Record Datacited by 85
  15. 2021Generalized Radiograph Representation Learning via Cross-supervision between Images and Free-text Radiology Reportscited by 80

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 6; 2024: 17; 2025: 15; 2026: 10.

48 papers.

Sort:

By area:

  1. 2024Large Language Models(LLMs) on Tabular Data: Prediction, Generation, and Understanding -- A Surveycited by 157
  2. 2023Conversational Health Agents: A Personalized LLM-Powered Agent Frameworkcited by 91
  3. 2024Health-LLM: Large Language Models for Health Prediction via Wearable Sensor Datacited by 65
  4. 2024PaPaGei: Open Foundation Models for Optical Physiological Signalscited by 64
  5. 2024Towards Time Series Reasoning with LLMscited by 36
  6. 2023The first step is the hardest: Pitfalls of Representing and Tokenizing Temporal Data for Large Language Modelscited by 34
  7. 2025Visual Attention Never Fades: Selective Progressive Attention ReCalibration for Detailed Image Captioning in Multimodal Large Language Modelscited by 21
  8. 2025A Survey of Reasoning and Agentic Systems in Time Series with Large Language Modelscited by 21
  9. 2024Can Large Language Models abstract Medical Coded Language?cited by 20
  10. 2025How Can Time Series Analysis Benefit From Multiple Modalities? A Survey and Outlookcited by 20
  11. 2024Language Modeling on Tabular Data: A Survey of Foundations, Techniques and Evolutioncited by 19
  12. 2023User Modeling in the Era of Large Language Models: Current Research and Future Directionscited by 18
36 more
  1. 2024Evaluating Large Language Models as Virtual Annotators for Time-series Physical Sensing Datacited by 17
  2. 2024Text Serialization and Their Relationship with the Conventional Paradigms of Tabular Machine Learningcited by 17
  3. 2024Lived Experience Not Found: LLMs Struggle to Align with Experts on Addressing Adverse Drug Reactions from Psychiatric Medication Usecited by 17
  4. 2025Valid Survey Simulations with Limited Human Data: The Roles of Prompting, Fine-Tuning, and Rectificationcited by 17
  5. 2024A Spectrum Evaluation Benchmark for Medical Multi-Modal Large Language Modelscited by 16
  6. 2024HeAR -- Health Acoustic Representationscited by 15
  7. 2024Data Imputation using Large Language Model to Accelerate Recommendation Systemcited by 15
  8. 2023Why LLMs Hallucinate, and How to Get (Evidential) Closure: Perceptual, Intensional, and Extensional Learning for Faithful Natural Language Generationcited by 13
  9. 2024Prompt engineering paradigms for medical applications: scoping review and recommendations for better practicescited by 13
  10. 2023Large Language Models Illuminate a Progressive Pathway to Artificial Healthcare Assistant: A Reviewcited by 8
  11. 2025Time2Lang: Bridging Time-Series Foundation Models and Large Language Models for Health Sensing Beyond Promptingcited by 7
  12. 2025Advancing and Benchmarking Personalized Tool Invocation for LLMscited by 7
  13. 2025Mask-GCG: Are All Tokens in Adversarial Suffixes Necessary for Jailbreak Attacks?cited by 6
  14. 2025MDD-LLM: Towards Accuracy Large Language Models for Major Depressive Disorder Diagnosiscited by 3
  15. 2025DocCHA: Towards LLM-Augmented Interactive Online diagnosis Systemcited by 3
  16. 2024Integrating Physiological Data with Large Language Models for Empathic Human-AI Interactioncited by 2
  17. 2024Data-Centric AI in the Age of Large Language Modelscited by 2
  18. 2024Transfer Learning with Clinical Concept Embeddings from Large Language Modelscited by 2
  19. 2025Adaptable Cardiovascular Disease Risk Prediction from Heterogeneous Data using Large Language Modelscited by 2
  20. 2025Delta-KNN: Improving Demonstration Selection in In-Context Learning for Alzheimer's Disease Detectioncited by 2
  21. 2026Learning Clinical Representations Under Systematic Distribution Shiftcited by 2
  22. 2026Discriminative Representation Learning for Clinical Predictioncited by 2
  23. 2026MILM: Large Language Models for Multimodal Irregular Time Series with Informative Samplingcited by 2
  24. 2025SpiroLLM: Finetuning Pretrained LLMs to Understand Spirogram Time Series with Clinical Validation in COPD Reportingcited by 1
  25. 2026Uncertainty-Aware Foundation Models for Clinical Datacited by 1
  26. 2023An AI-Guided Data Centric Strategy to Detect and Mitigate Biases in Healthcare Datasetscited by 0
  27. 2024Towards Holistic Disease Risk Prediction using Small Language Modelscited by 0
  28. 2025Predictive Multimodal Modeling of Diagnoses and Treatments in EHRcited by 0
  29. 2025Mitigating Clinician Information Overload: Generative AI for Integrated EHR and RPM Data Analysiscited by 0
  30. 2025Multimodal Health Risk Prediction System for Chronic Diseases via Vision-Language Fusion and Large Language Modelscited by 0
  31. 2026Dense Feature Learning via Linear Structure Preservation in Medical Datacited by 0
  32. 2026Evidence-Based Intelligent Diagnostic and Therapeutic Visualization System with Large Language Models: Multi-Turn Interaction and Multimodal Treatment Plan Generationcited by 0
  33. 2026TRIAGE: Dialectical Reasoning for Explainable Risk Prediction on Irregularly Sampled Medical Time Series with LLMscited by 0
  34. 2026On-Device Language Models for Privacy-Preserving Stress Prediction: A Multimodal Evaluation on Mobile Healthcited by 0
  35. 2026M2G-LLM: Enhancing Clinical Prediction via Multimodal Graph Reasoning and LLM Context Injectioncited by 0
  36. 2026GLoC-EHR: Evidence-Cited Clinical Reasoning over Global Context and Local EHR Eventscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024From Text to Multimodality: Exploring the Evolution and Impact of Large Language Models in Medical Practice
  2. 2024Health-LLM: Personalized Retrieval-Augmented Disease Prediction System
  3. 2024RespLLM: Unifying Audio and Text with Multimodal LLMs for Generalized Respiratory Health Prediction
  4. 2024A Comprehensive Survey of Large Language Models and Multimodal Large Language Models in Medicine
  5. 2025Multimodal Large Language Models for Medicine: A Comprehensive Survey
  6. 2024MedTsLLM: Leveraging LLMs for Multimodal Medical Time Series Analysis
  7. 2024Biomedical Large Languages Models Seem not to be Superior to Generalist Models on Unseen Medical Data
  8. 2024Evaluating LLM -- Generated Multimodal Diagnosis from Medical Images and Symptom Analysis