# Generalized Multimodal ELBO

arXiv: https://arxiv.org/abs/2105.02470 · HTML: https://mlsanity.com/paper/2105.02470/

## Key facts

- Cited by 87 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 34 references that are in the map.
- First posted on arXiv 6 May 2021.

## Builds on (most cited first, 23 of 23)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Auto-Encoding Variational Bayes](https://mlsanity.com/paper/1312.6114/) (2013)
- [Deep Learning Face Attributes in the Wild](https://mlsanity.com/paper/1411.7766/) (2014)
- [Multimodal Machine Learning: A Survey and Taxonomy](https://mlsanity.com/paper/1705.09406/) (2017)
- [Deep Visual-Semantic Alignments for Generating Image Descriptions](https://mlsanity.com/paper/1412.2306/) (2014)
- [Multimodal Unsupervised Image-to-Image Translation](https://mlsanity.com/paper/1804.04732/) (2018)
- [Importance Weighted Autoencoders](https://mlsanity.com/paper/1509.00519/) (2015)
- [Multimodal Generative Models for Scalable Weakly-Supervised Learning](https://mlsanity.com/paper/1802.05335/) (2018)
- [Variational Mixture-of-Experts Autoencoders for Multi-Modal Deep Generative Models](https://mlsanity.com/paper/1911.03393/) (2019)
- [Learning Factorized Multimodal Representations](https://mlsanity.com/paper/1806.06176/) (2018)
- [Found in Translation: Learning Robust Joint Representations by Cyclic Translations Between Modalities](https://mlsanity.com/paper/1812.07809/) (2018)
- [Multi-Level Variational Autoencoder: Learning Disentangled Representations from Grouped Observations](https://mlsanity.com/paper/1705.08841/) (2017)
- [Joint Multimodal Learning with Deep Generative Models](https://mlsanity.com/paper/1611.01891/) (2016)
- [On a generalization of the Jensen-Shannon divergence and the JS-symmetrization of distances relying on abstract means](https://mlsanity.com/paper/1904.04017/) (2019)
- [Multimodal Generative Learning Utilizing Jensen-Shannon-Divergence](https://mlsanity.com/paper/2006.08242/) (2020)
- [Generative Models of Visually Grounded Imagination](https://mlsanity.com/paper/1705.10762/) (2017)
- [Relating by Contrasting: A Data-efficient Framework for Multimodal Generative Models](https://mlsanity.com/paper/2007.01179/) (2020)
- [Disentangling by Partitioning: A Representation Learning Framework for Multimodal Sensory Data](https://mlsanity.com/paper/1805.11264/) (2018)
- [Multi-Domain Adversarial Learning](https://mlsanity.com/paper/1903.09239/) (2019)
- [Multimodal Generative Models for Compositional Representation Learning](https://mlsanity.com/paper/1912.05075/) (2019)
- [Multi-Source Neural Variational Inference](https://mlsanity.com/paper/1811.04451/) (2018)
- [Latent Translation: Crossing Modalities by Bridging Generative Models](https://mlsanity.com/paper/1902.08261/) (2019)

## Built on by (most cited first, 30 of 87)

- [MultiMAE: Multi-modal Multi-task Masked Autoencoders](https://mlsanity.com/paper/2204.01678/) (2022)
- [Visual Decoding and Reconstruction via EEG Embeddings with Guided Diffusion](https://mlsanity.com/paper/2403.07721/) (2024)
- [Multimodal Conditional Image Synthesis with Product-of-Experts GANs](https://mlsanity.com/paper/2112.05130/) (2021)
- [Decoding Visual Neural Representations by Multimodal Learning of Brain-Visual-Linguistic Features](https://mlsanity.com/paper/2210.06756/) (2022)
- [A survey of multimodal deep generative models](https://mlsanity.com/paper/2207.02127/) (2022)
- [On the Limitations of Multimodal VAEs](https://mlsanity.com/paper/2110.04121/) (2021)
- [BrainCLIP: Bridging Brain and Visual-Linguistic Representation Via CLIP for Generic Natural Visual Stimulus Decoding](https://mlsanity.com/paper/2302.12971/) (2023)
- [Mitigating Modality Collapse in Multimodal VAEs via Impartial Optimization](https://mlsanity.com/paper/2206.04496/) (2022)
- [A Closer Look at Multimodal Representation Collapse](https://mlsanity.com/paper/2505.22483/) (2025)
- [Representation Uncertainty in Self-Supervised Learning as Variational Inference](https://mlsanity.com/paper/2203.11437/) (2022)
- [Visual Neural Decoding via Improved Visual-EEG Semantic Consistency](https://mlsanity.com/paper/2408.06788/) (2024)
- [Learning Multimodal VAEs through Mutual Supervision](https://mlsanity.com/paper/2106.12570/) (2021)
- [Reconstructing Nonlinear Dynamical Systems from Multi-Modal Time Series](https://mlsanity.com/paper/2111.02922/) (2021)
- [Integrating Multimodal Data for Joint Generative Modeling of Complex Dynamics](https://mlsanity.com/paper/2212.07892/) (2022)
- [Unity by Diversity: Improved Representation Learning in Multimodal VAEs](https://mlsanity.com/paper/2403.05300/) (2024)
- [Gradient-Guided Modality Decoupling for Missing-Modality Robustness](https://mlsanity.com/paper/2402.16318/) (2024)
- [Multimodal sensor fusion in the latent representation space](https://mlsanity.com/paper/2208.02183/) (2022)
- [Multi-modal Latent Diffusion](https://mlsanity.com/paper/2306.04445/) (2023)
- [Unified Cross-Modal Medical Image Synthesis with Hierarchical Mixture of Product-of-Experts](https://mlsanity.com/paper/2410.19378/) (2024)
- [Joint fMRI Decoding and Encoding with Latent Embedding Alignment](https://mlsanity.com/paper/2303.14730/) (2023)
- [Deep Incomplete Multi-view Learning via Cyclic Permutation of VAEs](https://mlsanity.com/paper/2502.11037/) (2025)
- [Learning Multimodal Latent Generative Models with Energy-Based Prior](https://mlsanity.com/paper/2409.19862/) (2024)
- [Unsupervised physics-informed disentanglement of multimodal data for high-throughput scientific discovery](https://mlsanity.com/paper/2202.03242/) (2022)
- [Multi-View Dreaming: Multi-View World Model with Contrastive Learning](https://mlsanity.com/paper/2203.11024/) (2022)
- [Cooperation in the Latent Space: The Benefits of Adding Mixture Components in Variational Autoencoders](https://mlsanity.com/paper/2209.15514/) (2022)
- [eVAE: Evolutionary Variational Autoencoder](https://mlsanity.com/paper/2301.00011/) (2023)
- [A multimodal dynamical variational autoencoder for audiovisual speech representation learning](https://mlsanity.com/paper/2305.03582/) (2023)
- [FACEMUG: A Multimodal Generative and Fusion Framework for Local Facial Editing](https://mlsanity.com/paper/2412.19009/) (2024)
- [Image Generation with Multimodal Priors using Denoising Diffusion Probabilistic Models](https://mlsanity.com/paper/2206.05039/) (2022)
- [See Through Their Minds: Learning Transferable Neural Representation from Cross-Subject fMRI](https://mlsanity.com/paper/2403.06361/) (2024)

## Abstract (arXiv)

Multiple data types naturally co-occur when describing real-world phenomena and learning from them is a long-standing goal in machine learning research. However, existing self-supervised generative models approximating an ELBO are not able to fulfill all desired requirements of multimodal models: their posterior approximation functions lead to a trade-off between the semantic coherence and the ability to learn the joint data distribution. We propose a new, generalized ELBO formulation for multimodal data that overcomes these limitations. The new objective encompasses two previous methods as special cases and combines their benefits without compromises. In extensive experiments, we demonstrate the advantage of the proposed method compared to state-of-the-art models in self-supervised, generative learning tasks.