mlsanityData to 14 Oct 2026

Retrieval-Augmented Multimodal Language Model with Heterogeneous Evidence Integration for Clinical Prognosis

arXiv 2610.13510 ↗First posted 9 Oct 2026cs.AI cs.CLcited by 0

On the map: Multimodal LLMs / vision-language models → Medical vision-language models

also in 1 more
Abstract (arXiv)

Clinical prognosis requires integrating structured measurements, physiological waveforms, longitudinal history, and population context. We introduce PRISM (Population-Retrieved Inference for Structured Multimodal Prognosis), a multimodal language model that combines per-feature numerical tokenization, recency-weighted aggregation of longitudinal ECGs, and a dual-modality population memory bank. Retrieval operates in representation space: the model receives mean-pooled embeddings of similar training-cohort encounters, while no individual retrieved record enters the prompt. On the 1,443-task MDS-ED benchmark, PRISM achieves AUROCs of 86.2% for diagnosis, 93.1% for deterioration, 91.8% for ICU admission, and 92.8% for mortality. These results use a 96-hour pre-arrival window unavailable to reported baselines. With benchmark-window inputs only, PRISM reaches 0.8895 AUROC, compared with 0.8937 for UniPACT and 0.8889 for MDS-ED. A matched-input compact fusion model using the same five evidence streams and frozen encoders reaches 0.7652 AUROC, versus 0.9092 for PRISM. Replacing numerical EHR encoding with text serialization reduces AUROC by 3.66 points; removing pre-arrival history reduces it by 1.97 points; and zeroing retrieval reduces it by 0.99 points. Genuine retrieval outperforms random-neighbor retrieval by 0.97 macro-average points, although a paired task-level sign test does not establish a uniform advantage (p=0.067). On the external MC-MED cohort, supervised fine-tuning reaches 0.8611 AUROC, compared with 0.8227 for locally trained gradient boosting and 0.7841 for source-only PRISM. These results support structure-aware multimodal integration and supervised external adaptation.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Medical vision-language models, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 24 of its 35 references that are in the map. First posted on arXiv 9 Oct 2026.

Builds on

24 of 35 references.

Closest ancestors · 1

Same direction, newest first

  1. 2023LLaVA-Med: Training a Large Language-and-Vision Assistant for Biomedicine in One Day1 Jun

Other references · 23

  1. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  2. 2023Visual Instruction Tuningcited by 7,869
  3. 2025Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilitiescited by 3,796
  4. 2022Large Language Models Encode Clinical Knowledgecited by 1,830
  5. 2020On Layer Normalization in the Transformer Architecturecited by 922
  6. 2021Revisiting Deep Learning Models for Tabular Datacited by 740
  7. 2025MedGemma Technical Reportcited by 436
  8. 2020Med-BERT: pre-trained contextualized embeddings on large-scale structured electronic health records for disease predictioncited by 373
  9. 2019BEHRT: Transformer for Electronic Health Recordscited by 263
  10. 2025GEM: Empowering MLLM for Grounded ECG Understanding with Time Series and Imagescited by 39
  11. 2024ECG-Chat: A Large ECG-Language Model for Cardiac Disease Diagnosiscited by 38
  12. 2024Teach Multimodal LLMs to Comprehend Electrocardiographic Imagescited by 28
11 more
  1. 2024ECG Semantic Integrator (ESI): A Foundation ECG Model Pretrained with LLM-Enhanced Cardiological Textcited by 26
  2. 2019Deep neural networks can predict mortality from 12-lead electrocardiogram voltage datacited by 21
  3. 2025TimeMaster: Training Time-Series Multimodal LLMs to Reason via Reinforcement Learningcited by 18
  4. 2024EMERGE: Enhancing Multimodal Electronic Health Records Predictive Modeling with Retrieval-Augmented Generationcited by 17
  5. 2024Boosting Masked ECG-Text Auto-Encoders as Discriminative Learnerscited by 17
  6. 2024Enhancing clinical decision support with physiological waveforms -- a multimodal benchmark in emergency carecited by 15
  7. 2025Q-Heart: ECG Question Answering via Knowledge-Informed Multimodal LLMscited by 13
  8. 2024Electrocardiogram-Language Model for Few-Shot Question Answering with Meta Learningcited by 5
  9. 2024HeartBERT: A Self-Supervised ECG Embedding Model for Efficient and Effective Medical Signal Analysiscited by 3
  10. 2025High-Accuracy ECG Image Interpretation using Parameter-Efficient LoRA Fine-Tuning with Multimodal LLaMA 3.2cited by 2
  11. 2026UniPACT: A Multimodal Framework for Prognostic Question Answering on Raw ECG and Structured EHRcited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026ViSTA: A Simple Bridge Extends Visual Alignment to Clinical Time-Series Understanding in Multimodal LLMs
  2. 2026Multimodal Survival Analysis with Locally Deployable Large Language Models
  3. 2026Large Language Models as Unified Multimodal Learners for Clinical Prediction
  4. 2024Enhancing In-Hospital Mortality Prediction Using Multi-Representational Learning with LLM-Generated Expert Summaries
  5. 2025Large Language Models with Temporal Reasoning for Longitudinal Clinical Summarization and Prediction
  6. 2025ECG-LLM -- training and evaluation of domain-specific large language models for electrocardiography
  7. 2026CURE: A Multimodal Benchmark for Clinical Understanding and Retrieval Evaluation
  8. 2026EHR-RAG: Bridging Long-Horizon Structured Electronic Health Records and Large Language Models via Enhanced Retrieval-Augmented Generation