# Multimodal Training to Unimodal Deployment: Leveraging Unstructured Data During Training to Optimize Structured Data Only Deployment

arXiv: https://arxiv.org/abs/2603.22530 · HTML: https://mlsanity.com/paper/2603.22530/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 16 of its 38 references that are in the map.
- First posted on arXiv 23 Mar 2026.

## Builds on (most cited first, 16 of 16)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [BioBERT: a pre-trained biomedical language representation model for biomedical text mining](https://mlsanity.com/paper/1901.08746/) (2019)
- [Knowledge Distillation: A Survey](https://mlsanity.com/paper/2006.05525/) (2020)
- [Publicly Available Clinical BERT Embeddings](https://mlsanity.com/paper/1904.03323/) (2019)
- [MedCLIP: Contrastive Learning from Unpaired Medical Images and Text](https://mlsanity.com/paper/2210.10163/) (2022)
- [Med-BERT: pre-trained contextualized embeddings on large-scale structured electronic health records for disease prediction](https://mlsanity.com/paper/2005.12833/) (2020)
- [Explainable Prediction of Medical Codes from Clinical Text](https://mlsanity.com/paper/1802.05695/) (2018)
- [Deep Multimodal Learning with Missing Modality: A Survey](https://mlsanity.com/paper/2409.07825/) (2024)
- [Deep Partial Multi-View Learning](https://mlsanity.com/paper/2011.06170/) (2020)
- [Joint Embedding of Words and Labels for Text Classification](https://mlsanity.com/paper/1805.04174/) (2018)
- [Deep Representation Learning of Patient Data from Electronic Health Records (EHR): A Systematic Review](https://mlsanity.com/paper/2010.02809/) (2020)
- [Learnable Cross-modal Knowledge Distillation for Multi-modal Learning with Missing Modality](https://mlsanity.com/paper/2310.01035/) (2023)
- [CL4CTR: A Contrastive Learning Framework for CTR Prediction](https://mlsanity.com/paper/2212.00522/) (2022)
- [Machine Learning for Multimodal Electronic Health Records-based Research: Challenges and Perspectives](https://mlsanity.com/paper/2111.04898/) (2021)
- [CKD: Contrastive Knowledge Distillation from A Sample-wise Perspective](https://mlsanity.com/paper/2404.14109/) (2024)

## Abstract (arXiv)

Unstructured Electronic Health Record (EHR) data, such as clinical notes, contain clinical contextual observations that are not directly reflected in structured data fields. This additional information can substantially improve model learning. However, due to their unstructured nature, these data are often unavailable or impractical to use when deploying a model. We introduce a multimodal learning framework that leverages unstructured EHR data during training while producing a model that can be deployed using only structured EHR data. Using a cohort of 3,466 children evaluated for late talking, we generated note embeddings with BioClinicalBERT and encoded structured embeddings from demographics and medical codes. A note-based teacher model and a structured-only student model were jointly trained using contrastive learning and contrastive knowledge distillation loss, producing a strong classifier (AUROC = 0.985). Our proposed model reached an AUROC of 0.705, outperforming the structured-only baseline of 0.656. These results demonstrate that incorporating unstructured data during training enhances the model's capacity to identify task-relevant information within structured EHR data, enabling a deployable structured-only phenotype model.