# Unsupervised Multimodal Representation Learning across Medical Images and Reports

arXiv: https://arxiv.org/abs/1811.08615 · HTML: https://mlsanity.com/paper/1811.08615/

## Key facts

- Cited by 15 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 21 Nov 2018.

## Built on by (most cited first, 15 of 15)

- [Making the Most of Text Semantics to Improve Biomedical Vision--Language Processing](https://mlsanity.com/paper/2204.09817/) (2022)
- [Foundations and Trends in Multimodal Machine Learning: Principles, Challenges, and Open Questions](https://mlsanity.com/paper/2209.03430/) (2022)
- [Multi-modal Understanding and Generation for Medical Images and Text via Vision-Language Pre-Training](https://mlsanity.com/paper/2105.11333/) (2021)
- [Pre-trained Language Models in Biomedical Domain: A Systematic Survey](https://mlsanity.com/paper/2110.05006/) (2021)
- [S-CLIP: Semi-supervised Vision-Language Learning using Few Specialist Captions](https://mlsanity.com/paper/2305.14095/) (2023)
- [MLIP: Enhancing Medical Visual Representation with Divergence Encoder and Knowledge-guided Contrastive Learning](https://mlsanity.com/paper/2402.02045/) (2024)
- [Multimodal Learning Without Labeled Multimodal Data: Guarantees and Applications](https://mlsanity.com/paper/2306.04539/) (2023)
- [A scoping review on multimodal deep learning in biomedical images and texts](https://mlsanity.com/paper/2307.07362/) (2023)
- [Multi-Modal Learning Using Physicians Diagnostics for Optical Coherence Tomography Classification](https://mlsanity.com/paper/2203.10622/) (2022)
- [Learning by Hallucinating: Vision-Language Pre-training with Weak Supervision](https://mlsanity.com/paper/2210.13591/) (2022)
- [Multi-modal Pre-training for Medical Vision-language Understanding and Generation: An Empirical Study with A New Benchmark](https://mlsanity.com/paper/2306.06494/) (2023)
- [Deep representation learning: Fundamentals, Perspectives, Applications, and Open Challenges](https://mlsanity.com/paper/2211.14732/) (2022)
- [MIDAS: Misalignment-based Data Augmentation Strategy for Imbalanced Multimodal Learning](https://mlsanity.com/paper/2509.25831/) (2025)
- [Foundations of Multisensory Artificial Intelligence](https://mlsanity.com/paper/2404.18976/) (2024)
- [VSF-Med:A Vulnerability Scoring Framework for Medical Vision-Language Models](https://mlsanity.com/paper/2507.00052/) (2025)

## Abstract (arXiv)

Joint embeddings between medical imaging modalities and associated radiology reports have the potential to offer significant benefits to the clinical community, ranging from cross-domain retrieval to conditional generation of reports to the broader goals of multimodal representation learning. In this work, we establish baseline joint embedding results measured via both local and global retrieval methods on the soon to be released MIMIC-CXR dataset consisting of both chest X-ray images and the associated radiology reports. We examine both supervised and unsupervised methods on this task and show that for document retrieval tasks with the learned representations, only a limited amount of supervision is needed to yield results comparable to those of fully-supervised methods.