# A General Multimodal Probing Framework for Data Fusion and Prediction with Frozen Large Language Models: Evidence from Multimodal EHR Data

arXiv: https://arxiv.org/abs/2606.28798 · HTML: https://mlsanity.com/paper/2606.28798/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 22 of its 34 references that are in the map.
- First posted on arXiv 27 Jun 2026.

## Builds on (most cited first, 22 of 22)

- [XGBoost: A Scalable Tree Boosting System](https://mlsanity.com/paper/1603.02754/) (2016)
- [Understanding intermediate layers using linear classifier probes](https://mlsanity.com/paper/1610.01644/) (2016)
- [Designing and Interpreting Probes with Control Tasks](https://mlsanity.com/paper/1909.03368/) (2019)
- [RETAIN: An Interpretable Predictive Model for Healthcare using Reverse Time Attention Mechanism](https://mlsanity.com/paper/1608.05745/) (2016)
- [Multitask learning and benchmarking with clinical time series data](https://mlsanity.com/paper/1703.07771/) (2017)
- [Scalable and accurate deep learning for electronic health records](https://mlsanity.com/paper/1801.07860/) (2018)
- [Explainable Prediction of Medical Codes from Clinical Text](https://mlsanity.com/paper/1802.05695/) (2018)
- [Doctor AI: Predicting Clinical Events via Recurrent Neural Networks](https://mlsanity.com/paper/1511.05942/) (2015)
- [PLM-ICD: Automatic ICD Coding with Pretrained Language Models](https://mlsanity.com/paper/2207.05289/) (2022)
- [A Label Attention Model for ICD Coding from Clinical Text](https://mlsanity.com/paper/2007.06351/) (2020)
- [A Multimodal Transformer: Fusing Clinical Notes with Structured EHR Data for Interpretable In-Hospital Mortality Prediction](https://mlsanity.com/paper/2208.10240/) (2022)
- [Multimodal Machine Learning for Automated ICD Coding](https://mlsanity.com/paper/1810.13348/) (2018)
- [Large Language Models are Powerful Electronic Health Record Encoders](https://mlsanity.com/paper/2502.17403/) (2025)
- [Multimodal data matters: language model pre-training over structured and unstructured electronic health records](https://mlsanity.com/paper/2201.10113/) (2022)
- [ClinicRealm: Re-evaluating Large Language Models with Conventional Machine Learning for Non-Generative Clinical Prediction Tasks](https://mlsanity.com/paper/2407.18525/) (2024)
- [TreeMAN: Tree-enhanced Multimodal Attention Network for ICD Coding](https://mlsanity.com/paper/2305.18576/) (2023)
- [Clinical Risk Prediction Using Language Models: Benefits And Considerations](https://mlsanity.com/paper/2312.03742/) (2023)
- [MedRep: Medical Concept Representation for General Electronic Health Record Foundation Models](https://mlsanity.com/paper/2504.08329/) (2025)
- [MEDFuse: Multimodal EHR Data Fusion with Masked Lab-Test Modeling and Large Language Models](https://mlsanity.com/paper/2407.12309/) (2024)
- [Zero-shot Medical Event Prediction Using a Generative Pre-trained Transformer on Electronic Health Records](https://mlsanity.com/paper/2503.05893/) (2025)
- [A Multimodal Data Processing Pipeline for MIMIC-IV Dataset](https://mlsanity.com/paper/2601.11606/) (2026)
- [P-Transformer: A Prompt-based Multimodal Transformer Architecture For Medical Tabular Data](https://mlsanity.com/paper/2303.17408/) (2023)

## Abstract (arXiv)

The main objective of this paper is to propose a general framework for prediction based on different sources of multimodal data in the healthcare domain. We evaluated whether frozen medical large language model (LLM) representations can serve as a shared embedding space for multimodal primary diagnosis category prediction. We propose a backbone-flexible probing framework that serializes structured electronic health record (EHR) variables and leakage-pruned discharge-note sections into a shared frozen-LLM representation space and recovers primary diagnosis categories with linear probes, without fine-tuning the LLM. On a MIMIC-IV cohort of 13,645 admissions (seven diagnosis categories), we probed inputs of different modalities (Structured-only, Unstructured-only, and Combined) across transformer layers and six frozen backbones, compared against XGBoost and three text baselines (CAML, LAAT, PLM-ICD) under a matched split and context budget, and tested robustness across datasets on MIMIC-III. The Combined terminal-layer probe reached 88.86% top-1 accuracy on MIMIC-IV, significantly outperforming CAML and PLM-ICD and statistically comparable to LAAT; Combined accuracy ranged from 87.69% to 89.25% across six backbones. On MIMIC-III, in-domain Combined probing reached 74.18% accuracy and significantly exceeded all three baselines, and in a transfer-learning test a 2.8M-parameter adapter on frozen MIMIC-IV representations reached 92.51% accuracy, exceeding adapted LAAT (85.54%). Frozen LLM embeddings unify structured and narrative EHR information: the Combined configuration was strongest in every setting, and its representations transferred across datasets through a compact adapter. Multimodal probing of frozen LLM representations provides a practical, backbone-flexible approach for studying EHR modalities and adapting clinical representations across datasets.