# Improving Multimodal Joint Variational Autoencoders through Normalizing Flows and Correlation Analysis

arXiv: https://arxiv.org/abs/2305.11832 · HTML: https://mlsanity.com/paper/2305.11832/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 27 of its 48 references that are in the map.
- First posted on arXiv 19 May 2023.

## Builds on (most cited first, 27 of 27)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Generative Adversarial Networks](https://mlsanity.com/paper/2203.00667/) (2022)
- [Image-to-Image Translation with Conditional Adversarial Networks](https://mlsanity.com/paper/1611.07004/) (2016)
- [Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms](https://mlsanity.com/paper/1708.07747/) (2017)
- [Auto-Encoding Variational Bayes](https://mlsanity.com/paper/1312.6114/) (2013)
- [Deep Learning Face Attributes in the Wild](https://mlsanity.com/paper/1411.7766/) (2014)
- [Variational Inference with Normalizing Flows](https://mlsanity.com/paper/1505.05770/) (2015)
- [Contrastive Multiview Coding](https://mlsanity.com/paper/1906.05849/) (2019)
- [Masked Autoregressive Flow for Density Estimation](https://mlsanity.com/paper/1705.07057/) (2017)
- [Importance Weighted Autoencoders](https://mlsanity.com/paper/1509.00519/) (2015)
- [Toward Multimodal Image-to-Image Translation](https://mlsanity.com/paper/1711.11586/) (2017)
- [Multimodal Generative Models for Scalable Weakly-Supervised Learning](https://mlsanity.com/paper/1802.05335/) (2018)
- [Variational Mixture-of-Experts Autoencoders for Multi-Modal Deep Generative Models](https://mlsanity.com/paper/1911.03393/) (2019)
- [Medical Image Synthesis for Data Augmentation and Anonymization using Generative Adversarial Networks](https://mlsanity.com/paper/1807.10225/) (2018)
- [Joint Multimodal Learning with Deep Generative Models](https://mlsanity.com/paper/1611.01891/) (2016)
- [Generative Adversarial Networks (GANs Survey): Challenges, Solutions, and Future Directions](https://mlsanity.com/paper/2005.00065/) (2020)
- [Multimodal Generative Learning Utilizing Jensen-Shannon-Divergence](https://mlsanity.com/paper/2006.08242/) (2020)
- [A survey of multimodal deep generative models](https://mlsanity.com/paper/2207.02127/) (2022)
- [Deep Variational Canonical Correlation Analysis](https://mlsanity.com/paper/1610.03454/) (2016)
- [Generative Models of Visually Grounded Imagination](https://mlsanity.com/paper/1705.10762/) (2017)
- [On the Limitations of Multimodal VAEs](https://mlsanity.com/paper/2110.04121/) (2021)
- [Mitigating Modality Collapse in Multimodal VAEs via Impartial Optimization](https://mlsanity.com/paper/2206.04496/) (2022)
- [Data Augmentation in High Dimensional Low Sample Size Setting Using a Geometry-Based Variational Autoencoder](https://mlsanity.com/paper/2105.00026/) (2021)
- [Pythae: Unifying Generative Autoencoders in Python -- A Benchmarking Use Case](https://mlsanity.com/paper/2206.08309/) (2022)
- [Disentangling by Partitioning: A Representation Learning Framework for Multimodal Sensory Data](https://mlsanity.com/paper/1805.11264/) (2018)
- [Generative Image Translation for Data Augmentation in Colorectal Histopathology Images](https://mlsanity.com/paper/1910.05827/) (2019)
- [Structured SUMCOR Multiview Canonical Correlation Analysis for Large-Scale Data](https://mlsanity.com/paper/1804.08806/) (2018)

## Built on by (most cited first, 1 of 1)

- [CoVAE: correlated multimodal generative modeling](https://mlsanity.com/paper/2603.01965/) (2026)

## Abstract (arXiv)

We propose a new multimodal variational autoencoder that enables to generate from the joint distribution and conditionally to any number of complex modalities. The unimodal posteriors are conditioned on the Deep Canonical Correlation Analysis embeddings which preserve the shared information across modalities leading to more coherent cross-modal generations. Furthermore, we use Normalizing Flows to enrich the unimodal posteriors and achieve more diverse data generation. Finally, we propose to use a Product of Experts for inferring one modality from several others which makes the model scalable to any number of modalities. We demonstrate that our method improves likelihood estimates, diversity of the generations and in particular coherence metrics in the conditional generations on several datasets.