mlsanityData to 14 Oct 2026

Improving Multimodal Joint Variational Autoencoders through Normalizing Flows and Correlation Analysis

arXiv 2305.11832 ↗First posted 19 May 2023stat.ML cs.LGcited by 1

Not on the map. Why?

Abstract (arXiv)

We propose a new multimodal variational autoencoder that enables to generate from the joint distribution and conditionally to any number of complex modalities. The unimodal posteriors are conditioned on the Deep Canonical Correlation Analysis embeddings which preserve the shared information across modalities leading to more coherent cross-modal generations. Furthermore, we use Normalizing Flows to enrich the unimodal posteriors and achieve more diverse data generation. Finally, we propose to use a Product of Experts for inferring one modality from several others which makes the model scalable to any number of modalities. We demonstrate that our method improves likelihood estimates, diversity of the generations and in particular coherence metrics in the conditional generations on several datasets.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 27 of its 48 references that are in the map. First posted on arXiv 19 May 2023.

Builds on

27 of 48 references.

References · 27

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2022Generative Adversarial Networkscited by 5,627
  3. 2016Image-to-Image Translation with Conditional Adversarial Networkscited by 4,527
  4. 2017Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithmscited by 4,349
  5. 2013Auto-Encoding Variational Bayescited by 4,053
  6. 2014Deep Learning Face Attributes in the Wildcited by 3,693
  7. 2015Variational Inference with Normalizing Flowscited by 2,170
  8. 2019Contrastive Multiview Codingcited by 1,301
  9. 2017Masked Autoregressive Flow for Density Estimationcited by 684
  10. 2015Importance Weighted Autoencoderscited by 456
  11. 2017Toward Multimodal Image-to-Image Translationcited by 296
  12. 2018Multimodal Generative Models for Scalable Weakly-Supervised Learningcited by 200
15 more
  1. 2019Variational Mixture-of-Experts Autoencoders for Multi-Modal Deep Generative Modelscited by 178
  2. 2018Medical Image Synthesis for Data Augmentation and Anonymization using Generative Adversarial Networkscited by 147
  3. 2016Joint Multimodal Learning with Deep Generative Modelscited by 87
  4. 2020Generative Adversarial Networks (GANs Survey): Challenges, Solutions, and Future Directionscited by 65
  5. 2020Multimodal Generative Learning Utilizing Jensen-Shannon-Divergencecited by 48
  6. 2022A survey of multimodal deep generative modelscited by 47
  7. 2016Deep Variational Canonical Correlation Analysiscited by 44
  8. 2017Generative Models of Visually Grounded Imaginationcited by 44
  9. 2021On the Limitations of Multimodal VAEscited by 38
  10. 2022Mitigating Modality Collapse in Multimodal VAEs via Impartial Optimizationcited by 32
  11. 2021Data Augmentation in High Dimensional Low Sample Size Setting Using a Geometry-Based Variational Autoencodercited by 27
  12. 2022Pythae: Unifying Generative Autoencoders in Python -- A Benchmarking Use Casecited by 26
  13. 2018Disentangling by Partitioning: A Representation Learning Framework for Multimodal Sensory Datacited by 19
  14. 2019Generative Image Translation for Data Augmentation in Colorectal Histopathology Imagescited by 15
  15. 2018Structured SUMCOR Multiview Canonical Correlation Analysis for Large-Scale Datacited by 4

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026CoVAE: correlated multimodal generative modelingcited by 0