# Multimodal Learning for Scalable Representation of High-Dimensional Medical Data

arXiv: https://arxiv.org/abs/2409.13115 · HTML: https://mlsanity.com/paper/2409.13115/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 9 of its 39 references that are in the map.
- First posted on arXiv 19 Sep 2024.

## Builds on (most cited first, 9 of 9)

- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [Pathomic Fusion: An Integrated Framework for Fusing Histopathology and Genomic Features for Cancer Diagnosis and Prognosis](https://mlsanity.com/paper/1912.08937/) (2019)
- [Transcriptomics-guided Slide Representation Learning in Computational Pathology](https://mlsanity.com/paper/2405.11618/) (2024)
- [Learning to Predict RNA Sequence Expressions from Whole Slide Images with Applications for Search and Classification](https://mlsanity.com/paper/2203.13997/) (2022)
- [On Image Search in Histopathology](https://mlsanity.com/paper/2401.08699/) (2024)
- [MinMax Radon Barcodes for Medical Image Retrieval](https://mlsanity.com/paper/1610.00318/) (2016)
- [Learning Binary and Sparse Permutation-Invariant Representations for Fast and Memory Efficient Whole Slide Image Search](https://mlsanity.com/paper/2208.13653/) (2022)
- [Analysis and Validation of Image Search Engines in Histopathology](https://mlsanity.com/paper/2401.03271/) (2024)
- [SPLICE -- Streamlining Digital Pathology Image Processing](https://mlsanity.com/paper/2404.17704/) (2024)

## Built on by (most cited first, 1 of 1)

- [Feature-level Site Leakage Reduction for Cross-Hospital Chest X-ray Transfer via Self-Supervised Learning](https://mlsanity.com/paper/2604.00263/) (2026)

## Abstract (arXiv)

Integrating artificial intelligence (AI) with healthcare data is rapidly transforming medical diagnostics and driving progress toward precision medicine. However, effectively leveraging multimodal data, particularly digital pathology whole slide images (WSIs) and genomic sequencing, remains a significant challenge due to the intrinsic heterogeneity of these modalities and the need for scalable and interpretable frameworks. Existing diagnostic models typically operate on unimodal data, overlooking critical cross-modal interactions that can yield richer clinical insights. We introduce MarbliX (Multimodal Association and Retrieval with Binary Latent Indexed matriX), a self-supervised framework that learns to embed WSIs and immunogenomic profiles into compact, scalable binary codes, termed ``monogram.'' By optimizing a triplet contrastive objective across modalities, MarbliX captures high-resolution patient similarity in a unified latent space, enabling efficient retrieval of clinically relevant cases and facilitating case-based reasoning. textcolorblackIn lung cancer, MarbliX achieves 85-89% across all evaluation metrics, outperforming histopathology (69-71%) and immunogenomics (73-76%). In kidney cancer, real-valued monograms yield the strongest performance (F1: 80-83%, Accuracy: 87-90%), with binary monograms slightly lower (F1: 78-82%).