# A Markov Random Field Multi-Modal Variational AutoEncoder

arXiv: https://arxiv.org/abs/2408.09576 · HTML: https://mlsanity.com/paper/2408.09576/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 55 references that are in the map.
- First posted on arXiv 18 Aug 2024.

## Builds on (most cited first, 18 of 18)

- [ImageBind: One Embedding Space To Bind Them All](https://mlsanity.com/paper/2305.05665/) (2023)
- [NVAE: A Deep Hierarchical Variational Autoencoder](https://mlsanity.com/paper/2007.03898/) (2020)
- [Rethinking FID: Towards a Better Evaluation Metric for Image Generation](https://mlsanity.com/paper/2401.09603/) (2023)
- [Multimodal Generative Models for Scalable Weakly-Supervised Learning](https://mlsanity.com/paper/1802.05335/) (2018)
- [Variational Mixture-of-Experts Autoencoders for Multi-Modal Deep Generative Models](https://mlsanity.com/paper/1911.03393/) (2019)
- [Composing graphical models with neural networks for structured representations and fast inference](https://mlsanity.com/paper/1603.06277/) (2016)
- [Joint Multimodal Learning with Deep Generative Models](https://mlsanity.com/paper/1611.01891/) (2016)
- [Generalized Multimodal ELBO](https://mlsanity.com/paper/2105.02470/) (2021)
- [Multimodal Generative Learning Utilizing Jensen-Shannon-Divergence](https://mlsanity.com/paper/2006.08242/) (2020)
- [Generative Models of Visually Grounded Imagination](https://mlsanity.com/paper/1705.10762/) (2017)
- [On the Limitations of Multimodal VAEs](https://mlsanity.com/paper/2110.04121/) (2021)
- [Efficient Learning of Discrete Graphical Models](https://mlsanity.com/paper/1902.00600/) (2019)
- [Statistical Guarantees for Variational Autoencoders using PAC-Bayesian Theory](https://mlsanity.com/paper/2310.04935/) (2023)
- [Multimodal Deep Learning for Finance: Integrating and Forecasting International Stock Markets](https://mlsanity.com/paper/1903.06478/) (2019)
- [AircraftVerse: A Large-Scale Multimodal Dataset of Aerial Vehicle Designs](https://mlsanity.com/paper/2306.05562/) (2023)
- [GumBolt: Extending Gumbel trick to Boltzmann priors](https://mlsanity.com/paper/1805.07349/) (2018)
- [Learning Graphical Models With Hubs](https://mlsanity.com/paper/1402.7349/) (2014)
- [Multi-modal Deep Analysis for Multimedia](https://mlsanity.com/paper/1910.04964/) (2019)

## Built on by (most cited first, 2 of 2)

- [Hellinger Multimodal Variational Autoencoders](https://mlsanity.com/paper/2601.06572/) (2026)
- [H"older++: Improving the Quality-Coherence Trade-off in Multimodal VAEs](https://mlsanity.com/paper/2606.13381/) (2026)

## Abstract (arXiv)

Recent advancements in multimodal Variational AutoEncoders (VAEs) have highlighted their potential for modeling complex data from multiple modalities. However, many existing approaches use relatively straightforward aggregating schemes that may not fully capture the complex dynamics present between different modalities. This work introduces a novel multimodal VAE that incorporates a Markov Random Field (MRF) into both the prior and posterior distributions. This integration aims to capture complex intermodal interactions more effectively. Unlike previous models, our approach is specifically designed to model and leverage the intricacies of these relationships, enabling a more faithful representation of multimodal data. Our experiments demonstrate that our model performs competitively on the standard PolyMNIST dataset and shows superior performance in managing complex intermodal dependencies in a specially designed synthetic dataset, intended to test intricate relationships.