# Best of Both Worlds: Multimodal Contrastive Learning with Tabular and Imaging Data

arXiv: https://arxiv.org/abs/2303.14080 · HTML: https://mlsanity.com/paper/2303.14080/

## Key facts

- Cited by 69 papers in the map (data through 14 Oct 2026).
- Builds on 41 of its 83 references that are in the map.
- First posted on arXiv 24 Mar 2023.

## Builds on (most cited first, 30 of 41)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Representation Learning with Contrastive Predictive Coding](https://mlsanity.com/paper/1807.03748/) (2018)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Emerging Properties in Self-Supervised Vision Transformers](https://mlsanity.com/paper/2104.14294/) (2021)
- [Bootstrap your own latent: A new approach to self-supervised Learning](https://mlsanity.com/paper/2006.07733/) (2020)
- [Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervision](https://mlsanity.com/paper/2102.05918/) (2021)
- [Axiomatic Attribution for Deep Networks](https://mlsanity.com/paper/1703.01365/) (2017)
- [Exploring Simple Siamese Representation Learning](https://mlsanity.com/paper/2011.10566/) (2020)
- [Unsupervised Learning of Visual Features by Contrasting Cluster Assignments](https://mlsanity.com/paper/2006.09882/) (2020)
- [Supervised Contrastive Learning](https://mlsanity.com/paper/2004.11362/) (2020)
- [Barlow Twins: Self-Supervised Learning via Redundancy Reduction](https://mlsanity.com/paper/2103.03230/) (2021)
- [Context Encoders: Feature Learning by Inpainting](https://mlsanity.com/paper/1604.07379/) (2016)
- [Big Self-Supervised Models are Strong Semi-Supervised Learners](https://mlsanity.com/paper/2006.10029/) (2020)
- [Colorful Image Colorization](https://mlsanity.com/paper/1603.08511/) (2016)
- [Unsupervised Visual Representation Learning by Context Prediction](https://mlsanity.com/paper/1505.05192/) (2015)
- [Florence: A New Foundation Model for Computer Vision](https://mlsanity.com/paper/2111.11432/) (2021)
- [TabPFN: A Transformer That Solves Small Tabular Classification Problems in a Second](https://mlsanity.com/paper/2207.01848/) (2022)
- [TabNet: Attentive Interpretable Tabular Learning](https://mlsanity.com/paper/1908.07442/) (2019)
- [VideoCLIP: Contrastive Pre-training for Zero-shot Video-Text Understanding](https://mlsanity.com/paper/2109.14084/) (2021)
- [Tabular Data: Deep Learning is Not All You Need](https://mlsanity.com/paper/2106.03253/) (2021)
- [Deep Neural Networks and Tabular Data: A Survey](https://mlsanity.com/paper/2110.01889/) (2021)
- [Big Self-Supervised Models Advance Medical Image Classification](https://mlsanity.com/paper/2101.05224/) (2021)
- [Learning Representations for Automatic Colorization](https://mlsanity.com/paper/1603.06668/) (2016)
- [Unified Contrastive Learning in Image-Text-Label Space](https://mlsanity.com/paper/2204.03610/) (2022)
- [Tracking Emerges by Colorizing Videos](https://mlsanity.com/paper/1806.09594/) (2018)
- [SCARF: Self-Supervised Contrastive Learning using Random Feature Corruption](https://mlsanity.com/paper/2106.15147/) (2021)
- [3D Self-Supervised Methods for Medical Imaging](https://mlsanity.com/paper/2006.03829/) (2020)

## Built on by (most cited first, 30 of 69)

- [A Review of Modern Recommender Systems Using Generative Models (Gen-RecSys)](https://mlsanity.com/paper/2404.00579/) (2024)
- [SNIP: Bridging Mathematical Symbolic and Numeric Realms with Unified Pre-training](https://mlsanity.com/paper/2310.02227/) (2023)
- [Representation Learning for Tabular Data: A Comprehensive Survey](https://mlsanity.com/paper/2504.16109/) (2025)
- [TIP: Tabular-Image Pre-training for Multimodal Classification with Incomplete Data](https://mlsanity.com/paper/2407.07582/) (2024)
- [A review of deep learning-based information fusion techniques for multimodal medical image classification](https://mlsanity.com/paper/2404.15022/) (2024)
- [Multi-modal Time Series Analysis: A Tutorial and Survey](https://mlsanity.com/paper/2503.13709/) (2025)
- [HyperFusion: A Hypernetwork Approach to Multimodal Integration of Tabular and Medical Imaging Data for Predictive Modeling](https://mlsanity.com/paper/2403.13319/) (2024)
- [Towards Cardiac MRI Foundation Models: Comprehensive Visual-Tabular Representations for Whole-Heart Assessment and Beyond](https://mlsanity.com/paper/2504.13037/) (2025)
- [Multimodal Contrastive Learning and Tabular Attention for Automated Alzheimer's Disease Prediction](https://mlsanity.com/paper/2308.15469/) (2023)
- [STiL: Semi-supervised Tabular-Image Learning for Comprehensive Task-Relevant Information Exploration in Multimodal Classification](https://mlsanity.com/paper/2503.06277/) (2025)
- [TIME: TabPFN-Integrated Multimodal Engine for Robust Tabular-Image Learning](https://mlsanity.com/paper/2506.00813/) (2025)
- [Hierarchical Pretraining on Multimodal Electronic Health Records](https://mlsanity.com/paper/2310.07871/) (2023)
- [MultiModalPFN: Extending Prior-Data Fitted Networks for Multimodal Tabular Learning](https://mlsanity.com/paper/2602.20223/) (2026)
- [MolBind: Multimodal Alignment of Language, Molecules, and Proteins](https://mlsanity.com/paper/2403.08167/) (2024)
- [Learning Contrastive Multimodal Fusion with Improved Modality Dropout for Disease Detection and Prediction](https://mlsanity.com/paper/2509.18284/) (2025)
- [RegBN: Batch Normalization of Multimodal Data with Regularization](https://mlsanity.com/paper/2310.00641/) (2023)
- [Scalable Representation Learning for Multimodal Tabular Transactions](https://mlsanity.com/paper/2410.07851/) (2024)
- [A Tale of Two Classes: Adapting Supervised Contrastive Learning to Binary Imbalanced Datasets](https://mlsanity.com/paper/2503.17024/) (2025)
- [Unleashing the Power of Image-Tabular Self-Supervised Learning via Breaking Cross-Tabular Barriers](https://mlsanity.com/paper/2512.14026/) (2025)
- [Tables Guide Vision: Learning to See the Heart through Tabular Data](https://mlsanity.com/paper/2503.14998/) (2025)
- [Inference-Time Dynamic Modality Selection for Incomplete Multimodal Classification](https://mlsanity.com/paper/2601.22853/) (2026)
- [Fusing Echocardiography Images and Medical Records for Continuous Patient Stratification](https://mlsanity.com/paper/2401.07796/) (2024)
- [Tree-Regularized Tabular Embeddings](https://mlsanity.com/paper/2403.00963/) (2024)
- [No Data? No Problem: Robust Vision-Tabular Learning with Missing Values](https://mlsanity.com/paper/2512.19602/) (2025)
- [VT-Bench: A Unified Benchmark for Visual-Tabular Multi-Modal Learning](https://mlsanity.com/paper/2605.08146/) (2026)
- [Contrastive Federated Learning with Tabular Data Silos](https://mlsanity.com/paper/2409.06123/) (2024)
- [IMAN: An Adaptive Network for Robust NPC Mortality Prediction with Missing Modalities](https://mlsanity.com/paper/2410.18551/) (2024)
- [A Cross Modal Knowledge Distillation & Data Augmentation Recipe for Improving Transcriptomics Representations through Morphological Features](https://mlsanity.com/paper/2505.21317/) (2025)
- [CARDIUM: Congenital Anomaly Recognition with Diagnostic Images and Unified Medical records](https://mlsanity.com/paper/2510.15208/) (2025)
- [Orthogonalized Multimodal Contrastive Learning with Asymmetric Masking for Structured Representations](https://mlsanity.com/paper/2602.14983/) (2026)

## Abstract (arXiv)

Medical datasets and especially biobanks, often contain extensive tabular data with rich clinical information in addition to images. In practice, clinicians typically have less data, both in terms of diversity and scale, but still wish to deploy deep learning solutions. Combined with increasing medical dataset sizes and expensive annotation costs, the necessity for unsupervised methods that can pretrain multimodally and predict unimodally has risen. To address these needs, we propose the first self-supervised contrastive learning framework that takes advantage of images and tabular data to train unimodal encoders. Our solution combines SimCLR and SCARF, two leading contrastive learning strategies, and is simple and effective. In our experiments, we demonstrate the strength of our framework by predicting risks of myocardial infarction and coronary artery disease (CAD) using cardiac MR images and 120 clinical features from 40,000 UK Biobank subjects. Furthermore, we show the generalizability of our approach to natural images using the DVM car advertisement dataset. We take advantage of the high interpretability of tabular data and through attribution and ablation experiments find that morphometric tabular features, describing size and shape, have outsized importance during the contrastive learning process and improve the quality of the learned embeddings. Finally, we introduce a novel form of supervised contrastive learning, label as a feature (LaaF), by appending the ground truth label as a tabular feature during multimodal pretraining, outperforming all supervised contrastive baselines.