# Self-Supervised Multi-Modal Alignment for Whole Body Medical Imaging

arXiv: https://arxiv.org/abs/2107.06652 · HTML: https://mlsanity.com/paper/2107.06652/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 40 references that are in the map.
- First posted on arXiv 14 Jul 2021.

## Builds on (most cited first, 22 of 22)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [U-Net: Convolutional Networks for Biomedical Image Segmentation](https://mlsanity.com/paper/1505.04597/) (2015)
- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Bootstrap your own latent: A new approach to self-supervised Learning](https://mlsanity.com/paper/2006.07733/) (2020)
- [Distributed Representations of Words and Phrases and their Compositionality](https://mlsanity.com/paper/1310.4546/) (2013)
- [Unsupervised Learning of Visual Features by Contrasting Cluster Assignments](https://mlsanity.com/paper/2006.09882/) (2020)
- [Data-Efficient Image Recognition with Contrastive Predictive Coding](https://mlsanity.com/paper/1905.09272/) (2019)
- [Self-labelling via simultaneous clustering and representation learning](https://mlsanity.com/paper/1911.05371/) (2019)
- [Look, Listen and Learn](https://mlsanity.com/paper/1705.08168/) (2017)
- [Spatiotemporal Contrastive Video Representation Learning](https://mlsanity.com/paper/2008.03800/) (2020)
- [Audio-Visual Scene Analysis with Self-Supervised Multisensory Features](https://mlsanity.com/paper/1804.03641/) (2018)
- [Self-Supervised Learning by Cross-Modal Audio-Video Clustering](https://mlsanity.com/paper/1911.12667/) (2019)
- [Objects that Sound](https://mlsanity.com/paper/1712.06651/) (2017)
- [Cooperative Learning of Audio and Video Models from Self-Supervised Synchronization](https://mlsanity.com/paper/1807.00230/) (2018)
- [Self-supervised Co-training for Video Representation Learning](https://mlsanity.com/paper/2010.09709/) (2020)
- [3D Self-Supervised Methods for Medical Imaging](https://mlsanity.com/paper/2006.03829/) (2020)
- [Multimodal Self-Supervised Learning for Medical Image Analysis](https://mlsanity.com/paper/1912.05396/) (2019)
- [DermGAN: Synthetic Generation of Clinical Skin Images with Pathology](https://mlsanity.com/paper/1911.08716/) (2019)
- [Self-Supervised Learning for Spinal MRIs](https://mlsanity.com/paper/1708.00367/) (2017)
- [A Convolutional Approach to Vertebrae Detection and Labelling in Whole Spine MRI](https://mlsanity.com/paper/2007.02606/) (2020)

## Built on by (most cited first, 6 of 6)

- [Bootstrapping Autonomous Driving Radars with Self-Supervised Learning](https://mlsanity.com/paper/2312.04519/) (2023)
- [X-Diffusion: Generating Detailed 3D MRI Volumes From a Single Image Using Cross-Sectional Diffusion Models](https://mlsanity.com/paper/2404.19604/) (2024)
- [UKBOB: One Billion MRI Labeled Masks for Generalizable 3D Medical Image Segmentation](https://mlsanity.com/paper/2504.06908/) (2025)
- [Look, Radiate, and Learn: Self-Supervised Localisation via Radio-Visual Correspondence](https://mlsanity.com/paper/2206.06424/) (2022)
- [MixCL: Pixel label matters to contrastive learning](https://mlsanity.com/paper/2203.02114/) (2022)
- [3D Spine Shape Estimation from Single 2D DXA](https://mlsanity.com/paper/2412.01504/) (2024)

## Abstract (arXiv)

This paper explores the use of self-supervised deep learning in medical imaging in cases where two scan modalities are available for the same subject. Specifically, we use a large publicly-available dataset of over 20,000 subjects from the UK Biobank with both whole body Dixon technique magnetic resonance (MR) scans and also dual-energy x-ray absorptiometry (DXA) scans. We make three contributions: (i) We introduce a multi-modal image-matching contrastive framework, that is able to learn to match different-modality scans of the same subject with high accuracy. (ii) Without any adaption, we show that the correspondences learnt during this contrastive training step can be used to perform automatic cross-modal scan registration in a completely unsupervised manner. (iii) Finally, we use these registrations to transfer segmentation maps from the DXA scans to the MR scans where they are used to train a network to segment anatomical regions without requiring ground-truth MR examples. To aid further research, our code will be made publicly available.