# Self-Supervised Multimodal Domino: in Search of Biomarkers for Alzheimer's Disease

arXiv: https://arxiv.org/abs/2012.13623 · HTML: https://mlsanity.com/paper/2012.13623/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 25 Dec 2020.

## Built on by (most cited first, 6 of 6)

- [Joint Self-Supervised and Supervised Contrastive Learning for Multimodal MRI Data: Towards Predicting Abnormal Neurodevelopment](https://mlsanity.com/paper/2312.15064/) (2023)
- [Fusing multimodal neuroimaging data with a variational autoencoder](https://mlsanity.com/paper/2105.01128/) (2021)
- [Pipeline-Invariant Representation Learning for Neuroimaging](https://mlsanity.com/paper/2208.12909/) (2022)
- [A Survey of the Impact of Self-Supervised Pretraining for Diagnostic Tasks with Radiological Images](https://mlsanity.com/paper/2309.02555/) (2023)
- [Thyroid ultrasound diagnosis improvement via multi-view self-supervised learning and two-stage pre-training](https://mlsanity.com/paper/2402.11497/) (2024)
- [Self-supervised multimodal neuroimaging yields predictive representations for a spectrum of Alzheimer's phenotypes](https://mlsanity.com/paper/2209.02876/) (2022)

## Abstract (arXiv)

Sensory input from multiple sources is crucial for robust and coherent human perception. Different sources contribute complementary explanatory factors. Similarly, research studies often collect multimodal imaging data, each of which can provide shared and unique information. This observation motivated the design of powerful multimodal self-supervised representation-learning algorithms. In this paper, we unify recent work on multimodal self-supervised learning under a single framework. Observing that most self-supervised methods optimize similarity metrics between a set of model components, we propose a taxonomy of all reasonable ways to organize this process. We first evaluate models on toy multimodal MNIST datasets and then apply them to a multimodal neuroimaging dataset with Alzheimer's disease patients. We find that (1) multimodal contrastive learning has significant benefits over its unimodal counterpart, (2) the specific composition of multiple contrastive objectives is critical to performance on a downstream task, (3) maximization of the similarity between representations has a regularizing effect on a neural network, which can sometimes lead to reduced downstream performance but still reveal multimodal relations. Results show that the proposed approach outperforms previous self-supervised encoder-decoder methods based on canonical correlation analysis (CCA) or the mixture-of-experts multimodal variational autoEncoder (MMVAE) on various datasets with a linear evaluation protocol. Importantly, we find a promising solution to uncover connections between modalities through a jointly shared subspace that can help advance work in our search for neuroimaging biomarkers.