# Multi-modal Variational Autoencoders for normative modelling across multiple imaging modalities

arXiv: https://arxiv.org/abs/2303.12706 · HTML: https://mlsanity.com/paper/2303.12706/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 5 of its 15 references that are in the map.
- First posted on arXiv 16 Mar 2023.

## Builds on (most cited first, 5 of 5)

- [Multimodal Generative Models for Scalable Weakly-Supervised Learning](https://mlsanity.com/paper/1802.05335/) (2018)
- [Variational Mixture-of-Experts Autoencoders for Multi-Modal Deep Generative Models](https://mlsanity.com/paper/1911.03393/) (2019)
- [Generalized Product of Experts for Automatic and Principled Fusion of Gaussian Process Predictions](https://mlsanity.com/paper/1410.7827/) (2014)
- [On the Limitations of Multimodal VAEs](https://mlsanity.com/paper/2110.04121/) (2021)
- [Generalized Product-of-Experts for Learning Multimodal Representations in Noisy Environments](https://mlsanity.com/paper/2211.03587/) (2022)

## Built on by (most cited first, 10 of 10)

- [Towards Enabling Cardiac Digital Twins of Myocardial Infarction Using Deep Computational Models for Inverse Inference](https://mlsanity.com/paper/2307.04421/) (2023)
- [Improving Normative Modeling for Multi-modal Neuroimaging Data using mixture-of-product-of-experts variational autoencoders](https://mlsanity.com/paper/2312.00992/) (2023)
- [A tutorial on multi-view autoencoders using the multi-view-AE library](https://mlsanity.com/paper/2403.07456/) (2024)
- [Multimodal Variational Autoencoder for Low-cost Cardiac Hemodynamics Instability Detection](https://mlsanity.com/paper/2403.13658/) (2024)
- [Analyzing heterogeneity in Alzheimer Disease using multimodal normative modeling on imaging-based ATN biomarkers](https://mlsanity.com/paper/2404.05748/) (2024)
- [Normative Diffusion Autoencoders: Application to Amyotrophic Lateral Sclerosis](https://mlsanity.com/paper/2407.14191/) (2024)
- [ProM3E: Probabilistic Masked MultiModal Embedding Model for Ecology](https://mlsanity.com/paper/2511.02946/) (2025)
- [Multimodal Latent Fusion of ECG Leads for Early Assessment of Pulmonary Hypertension](https://mlsanity.com/paper/2503.13470/) (2025)
- [CADD: Context aware disease deviations via restoration of brain images using normative conditional diffusion models](https://mlsanity.com/paper/2508.03594/) (2025)
- [Multimodal normative modeling in Alzheimers Disease with introspective variational autoencoders](https://mlsanity.com/paper/2602.08077/) (2026)

## Abstract (arXiv)

One of the challenges of studying common neurological disorders is disease heterogeneity including differences in causes, neuroimaging characteristics, comorbidities, or genetic variation. Normative modelling has become a popular method for studying such cohorts where the 'normal' behaviour of a physiological system is modelled and can be used at subject level to detect deviations relating to disease pathology. For many heterogeneous diseases, we expect to observe abnormalities across a range of neuroimaging and biological variables. However, thus far, normative models have largely been developed for studying a single imaging modality. We aim to develop a multi-modal normative modelling framework where abnormality is aggregated across variables of multiple modalities and is better able to detect deviations than uni-modal baselines. We propose two multi-modal VAE normative models to detect subject level deviations across T1 and DTI data. Our proposed models were better able to detect diseased individuals, capture disease severity, and correlate with patient cognition than baseline approaches. We also propose a multivariate latent deviation metric, measuring deviations from the joint latent space, which outperformed feature-based metrics.