# OC-Distill: Ontology-aware Contrastive Learning with Cross-Modal Distillation for ICU Risk Prediction

arXiv: https://arxiv.org/abs/2604.16878 · HTML: https://mlsanity.com/paper/2604.16878/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 48 of its 97 references that are in the map.
- First posted on arXiv 18 Apr 2026.

## Builds on (most cited first, 30 of 48)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Distilling the Knowledge in a Neural Network](https://mlsanity.com/paper/1503.02531/) (2015)
- [A Unified Approach to Interpreting Model Predictions](https://mlsanity.com/paper/1705.07874/) (2017)
- [Supervised Contrastive Learning](https://mlsanity.com/paper/2004.11362/) (2020)
- [Publicly Available Clinical BERT Embeddings](https://mlsanity.com/paper/1904.03323/) (2019)
- [A Survey on Contrastive Self-supervised Learning](https://mlsanity.com/paper/2011.00362/) (2020)
- [Multitask learning and benchmarking with clinical time series data](https://mlsanity.com/paper/1703.07771/) (2017)
- [Adapted Large Language Models Can Outperform Medical Experts in Clinical Text Summarization](https://mlsanity.com/paper/2309.07430/) (2023)
- [CLOCS: Contrastive Learning of Cardiac Signals Across Space, Time, and Patients](https://mlsanity.com/paper/2005.13249/) (2020)
- [FutureSightDrive: Thinking Visually with Spatio-Temporal CoT for Autonomous Driving](https://mlsanity.com/paper/2505.17685/) (2025)
- [MIMIC-Extract: A Data Extraction, Preprocessing, and Representation Pipeline for MIMIC-III](https://mlsanity.com/paper/1907.08322/) (2019)
- [Rethinking Interpretability in the Era of Large Language Models](https://mlsanity.com/paper/2402.01761/) (2024)
- [Attend and Diagnose: Clinical Time Series Analysis using Attention Models](https://mlsanity.com/paper/1711.03905/) (2017)
- [JanusVLN: Decoupling Semantics and Spatiality with Dual Implicit Memory for Vision-Language Navigation](https://mlsanity.com/paper/2509.22548/) (2025)
- [MedFuse: Multi-modal fusion with clinical time-series data and chest X-ray images](https://mlsanity.com/paper/2207.07027/) (2022)
- [A Multimodal Transformer: Fusing Clinical Notes with Structured EHR Data for Interpretable In-Hospital Mortality Prediction](https://mlsanity.com/paper/2208.10240/) (2022)
- [3KG: Contrastive Learning of 12-Lead Electrocardiograms using Physiologically-Inspired Augmentations](https://mlsanity.com/paper/2106.04452/) (2021)
- [Clinical Intervention Prediction and Understanding using Deep Networks](https://mlsanity.com/paper/1705.08498/) (2017)
- [Toward Engineering AGI: Benchmarking the Engineering Design Capabilities of LLMs](https://mlsanity.com/paper/2509.16204/) (2025)
- [A Learning Based Hypothesis Test for Harmful Covariate Shift](https://mlsanity.com/paper/2212.02742/) (2022)
- [ACAM-KD: Adaptive and Cooperative Attention Masking for Knowledge Distillation](https://mlsanity.com/paper/2503.06307/) (2025)
- [Sequential Multi-Dimensional Self-Supervised Learning for Clinical Time Series](https://mlsanity.com/paper/2307.10923/) (2023)
- [Efficient Fine-Tuning of Large Language Models for Automated Medical Documentation](https://mlsanity.com/paper/2409.09324/) (2024)
- [Retrieval-Augmented Dynamic Prompt Tuning for Incomplete Multimodal Learning](https://mlsanity.com/paper/2501.01120/) (2025)
- [Integrating Physiological Time Series and Clinical Notes with Transformer for Early Prediction of Sepsis](https://mlsanity.com/paper/2203.14469/) (2022)
- [Do Sparse Autoencoders Identify Reasoning Features in Language Models?](https://mlsanity.com/paper/2601.05679/) (2026)
- [ReCo-KD: Region- and Context-Aware Knowledge Distillation for Efficient 3D Medical Image Segmentation](https://mlsanity.com/paper/2601.08301/) (2026)
- [Temporal Supervised Contrastive Learning for Modeling Patient Risk Progression](https://mlsanity.com/paper/2312.05933/) (2023)

## Built on by (most cited first, 2 of 2)

- [SP-Mind: An Autonomous Reasoning Agent for Spatial Proteomics Analysis](https://mlsanity.com/paper/2606.24235/) (2026)
- [MMOOC: A Comprehensive Benchmark for Out-of-Context Evaluation in Multimodal Large Language Models](https://mlsanity.com/paper/2607.27637/) (2026)

## Abstract (arXiv)

Early prediction of severe clinical deterioration and remaining length of stay can enable timely intervention and better resource allocation in high-acuity settings such as the ICU. This has driven the development of machine learning models that leverage continuous streams of vital signs and other physiological signals for real-time risk prediction. Despite their promise, existing methods have important limitations. Contrastive pretraining treats all patients as equally strong negatives, failing to capture clinically meaningful similarity between patients with related diagnoses. Meanwhile, downstream fine-tuning typically ignores complementary modalities such as clinical notes, which provide rich contextual information unavailable in physiological signals alone. To address these challenges, we propose OC-Distill, a two-stage framework that leverages multimodal supervision during training while requiring only vital signs at inference. In the first stage, we introduce an ontology-aware contrastive objective that exploits the ICD hierarchy to quantify patient similarity and learn clinically grounded representations. In the second stage, we fine-tune the pretrained encoder via cross-modal knowledge distillation, transferring complementary information from clinical notes into the model. Across multiple ICU prediction tasks on MIMIC, OC-Distill demonstrates improved label efficiency and achieves state-of-the-art performance among methods that use only vital signs at inference.