# Multimodal Generative Learning Utilizing Jensen-Shannon-Divergence

arXiv: https://arxiv.org/abs/2006.08242 · HTML: https://mlsanity.com/paper/2006.08242/

## Key facts

- Cited by 48 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 15 Jun 2020.

## Built on by (most cited first, 30 of 48)

- [MultiMAE: Multi-modal Multi-task Masked Autoencoders](https://mlsanity.com/paper/2204.01678/) (2022)
- [Generalized Multimodal ELBO](https://mlsanity.com/paper/2105.02470/) (2021)
- [A survey of multimodal deep generative models](https://mlsanity.com/paper/2207.02127/) (2022)
- [On the Limitations of Multimodal VAEs](https://mlsanity.com/paper/2110.04121/) (2021)
- [Multimodal Variational Auto-encoder based Audio-Visual Segmentation](https://mlsanity.com/paper/2310.08303/) (2023)
- [Mitigating Modality Collapse in Multimodal VAEs via Impartial Optimization](https://mlsanity.com/paper/2206.04496/) (2022)
- [Learning Multimodal VAEs through Mutual Supervision](https://mlsanity.com/paper/2106.12570/) (2021)
- [Unity by Diversity: Improved Representation Learning in Multimodal VAEs](https://mlsanity.com/paper/2403.05300/) (2024)
- [Modeling Inter-Intra Heterogeneity for Graph Federated Learning](https://mlsanity.com/paper/2412.11402/) (2024)
- [Multi-modal Latent Diffusion](https://mlsanity.com/paper/2306.04445/) (2023)
- [Deep Incomplete Multi-view Learning via Cyclic Permutation of VAEs](https://mlsanity.com/paper/2502.11037/) (2025)
- [Learning Multimodal Latent Generative Models with Energy-Based Prior](https://mlsanity.com/paper/2409.19862/) (2024)
- [Fast Multi-Step Critiquing for VAE-based Recommender Systems](https://mlsanity.com/paper/2105.00774/) (2021)
- [A multimodal dynamical variational autoencoder for audiovisual speech representation learning](https://mlsanity.com/paper/2305.03582/) (2023)
- [Text-to-Image Cross-Modal Generation: A Systematic Review](https://mlsanity.com/paper/2401.11631/) (2024)
- [Discriminative Multimodal Learning via Conditional Priors in Generative Models](https://mlsanity.com/paper/2110.04616/) (2021)
- [Learning multi-modal generative models with permutation-invariant encoders and tighter variational objectives](https://mlsanity.com/paper/2309.00380/) (2023)
- [Inference-Time Dynamic Modality Selection for Incomplete Multimodal Classification](https://mlsanity.com/paper/2601.22853/) (2026)
- [Composable Generative Models](https://mlsanity.com/paper/2102.09249/) (2021)
- [Pneumonia Detection in Chest X-Ray Images : Handling Class Imbalance](https://mlsanity.com/paper/2301.08479/) (2023)
- [Improving Normative Modeling for Multi-modal Neuroimaging Data using mixture-of-product-of-experts variational autoencoders](https://mlsanity.com/paper/2312.00992/) (2023)
- [Multimodal ELBO with Diffusion Decoders](https://mlsanity.com/paper/2408.16883/) (2024)
- [Hellinger Multimodal Variational Autoencoders](https://mlsanity.com/paper/2601.06572/) (2026)
- [Multimodal Adversarially Learned Inference with Factorized Discriminators](https://mlsanity.com/paper/2112.10384/) (2021)
- [Benchmarking Multimodal Variational Autoencoders: CdSprites+ Dataset and Toolkit](https://mlsanity.com/paper/2209.03048/) (2022)
- [A Markov Random Field Multi-Modal Variational AutoEncoder](https://mlsanity.com/paper/2408.09576/) (2024)
- [From Concrete to Abstract: A Multimodal Generative Approach to Abstract Concept Learning](https://mlsanity.com/paper/2410.02365/) (2024)
- [Multi-Component VAE with Gaussian Markov Random Field](https://mlsanity.com/paper/2507.12165/) (2025)
- [TC-VAE: Uncovering Out-of-Distribution Data Generative Factors](https://mlsanity.com/paper/2304.04103/) (2023)
- [Improving Multimodal Joint Variational Autoencoders through Normalizing Flows and Correlation Analysis](https://mlsanity.com/paper/2305.11832/) (2023)

## Abstract (arXiv)

Learning from different data types is a long-standing goal in machine learning research, as multiple information sources co-occur when describing natural phenomena. However, existing generative models that approximate a multimodal ELBO rely on difficult or inefficient training schemes to learn a joint distribution and the dependencies between modalities. In this work, we propose a novel, efficient objective function that utilizes the Jensen-Shannon divergence for multiple distributions. It simultaneously approximates the unimodal and joint multimodal posteriors directly via a dynamic prior. In addition, we theoretically prove that the new multimodal JS-divergence (mmJSD) objective optimizes an ELBO. In extensive experiments, we demonstrate the advantage of the proposed mmJSD model compared to previous work in unsupervised, generative learning tasks.