mlsanityData to 7 Oct 2026

Generalized Multimodal ELBO

arXiv 2105.02470 ↗First posted 6 May 2021cs.LG stat.MLcited by 86

Not on the map. Why?

Abstract (arXiv)

Multiple data types naturally co-occur when describing real-world phenomena and learning from them is a long-standing goal in machine learning research. However, existing self-supervised generative models approximating an ELBO are not able to fulfill all desired requirements of multimodal models: their posterior approximation functions lead to a trade-off between the semantic coherence and the ability to learn the joint data distribution. We propose a new, generalized ELBO formulation for multimodal data that overcomes these limitations. The new objective encompasses two previous methods as special cases and combines their benefits without compromises. In extensive experiments, we demonstrate the advantage of the proposed method compared to state-of-the-art models in self-supervised, generative learning tasks.

Key facts. Cited by 86 papers in the map (data through 7 Oct 2026). Builds on 23 of its 34 references that are in the map. First posted on arXiv 6 May 2021.

Builds on

23 of 34 references.

References · 23

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2013Auto-Encoding Variational Bayescited by 3,980
  4. 2014Deep Learning Face Attributes in the Wildcited by 3,679
  5. 2017Multimodal Machine Learning: A Survey and Taxonomycited by 1,004
  6. 2014Deep Visual-Semantic Alignments for Generating Image Descriptionscited by 966
  7. 2018Multimodal Unsupervised Image-to-Image Translationcited by 795
  8. 2015Importance Weighted Autoencoderscited by 454
  9. 2018Multimodal Generative Models for Scalable Weakly-Supervised Learningcited by 200
  10. 2019Variational Mixture-of-Experts Autoencoders for Multi-Modal Deep Generative Modelscited by 177
  11. 2018Learning Factorized Multimodal Representationscited by 171
  12. 2018Found in Translation: Learning Robust Joint Representations by Cyclic Translations Between Modalitiescited by 149
11 more
  1. 2017Multi-Level Variational Autoencoder: Learning Disentangled Representations from Grouped Observationscited by 100
  2. 2016Joint Multimodal Learning with Deep Generative Modelscited by 87
  3. 2019On a generalization of the Jensen-Shannon divergence and the JS-symmetrization of distances relying on abstract meanscited by 61
  4. 2020Multimodal Generative Learning Utilizing Jensen-Shannon-Divergencecited by 48
  5. 2017Generative Models of Visually Grounded Imaginationcited by 44
  6. 2020Relating by Contrasting: A Data-efficient Framework for Multimodal Generative Modelscited by 27
  7. 2018Disentangling by Partitioning: A Representation Learning Framework for Multimodal Sensory Datacited by 19
  8. 2019Multi-Domain Adversarial Learningcited by 19
  9. 2019Multimodal Generative Models for Compositional Representation Learningcited by 13
  10. 2018Multi-Source Neural Variational Inferencecited by 9
  11. 2019Latent Translation: Crossing Modalities by Bridging Generative Modelscited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 6; 2022: 16; 2023: 11; 2024: 19; 2025: 19; 2026: 15.

86 papers.

Sort:

By area:

  1. 2022MultiMAE: Multi-modal Multi-task Masked Autoencoderscited by 287
  2. 2024Visual Decoding and Reconstruction via EEG Embeddings with Guided Diffusioncited by 79
  3. 2021Multimodal Conditional Image Synthesis with Product-of-Experts GANscited by 75
  4. 2022Decoding Visual Neural Representations by Multimodal Learning of Brain-Visual-Linguistic Featurescited by 63
  5. 2022A survey of multimodal deep generative modelscited by 47
  6. 2021On the Limitations of Multimodal VAEscited by 37
  7. 2023BrainCLIP: Bridging Brain and Visual-Linguistic Representation Via CLIP for Generic Natural Visual Stimulus Decodingcited by 36
  8. 2022Mitigating Modality Collapse in Multimodal VAEs via Impartial Optimizationcited by 32
  9. 2025A Closer Look at Multimodal Representation Collapsecited by 25
  10. 2022Representation Uncertainty in Self-Supervised Learning as Variational Inferencecited by 22
  11. 2024Visual Neural Decoding via Improved Visual-EEG Semantic Consistencycited by 21
  12. 2021Learning Multimodal VAEs through Mutual Supervisioncited by 18
38 more
  1. 2021Reconstructing Nonlinear Dynamical Systems from Multi-Modal Time Seriescited by 16
  2. 2022Integrating Multimodal Data for Joint Generative Modeling of Complex Dynamicscited by 13
  3. 2024Unity by Diversity: Improved Representation Learning in Multimodal VAEscited by 13
  4. 2024Gradient-Guided Modality Decoupling for Missing-Modality Robustnesscited by 11
  5. 2022Multimodal sensor fusion in the latent representation spacecited by 9
  6. 2023Multi-modal Latent Diffusioncited by 9
  7. 2023Joint fMRI Decoding and Encoding with Latent Embedding Alignmentcited by 8
  8. 2024Unified Cross-Modal Medical Image Synthesis with Hierarchical Mixture of Product-of-Expertscited by 8
  9. 2024Learning Multimodal Latent Generative Models with Energy-Based Priorcited by 7
  10. 2025Deep Incomplete Multi-view Learning via Cyclic Permutation of VAEscited by 7
  11. 2022Unsupervised physics-informed disentanglement of multimodal data for high-throughput scientific discoverycited by 6
  12. 2022Cooperation in the Latent Space: The Benefits of Adding Mixture Components in Variational Autoencoderscited by 6
  13. 2023eVAE: Evolutionary Variational Autoencodercited by 6
  14. 2023A multimodal dynamical variational autoencoder for audiovisual speech representation learningcited by 6
  15. 2024FACEMUG: A Multimodal Generative and Fusion Framework for Local Facial Editingcited by 6
  16. 2022Multi-View Dreaming: Multi-View World Model with Contrastive Learningcited by 5
  17. 2022Image Generation with Multimodal Priors using Denoising Diffusion Probabilistic Modelscited by 5
  18. 2024See Through Their Minds: Learning Transferable Neural Representation from Cross-Subject fMRIcited by 5
  19. 2021Discriminative Multimodal Learning via Conditional Priors in Generative Modelscited by 4
  20. 2022Generalized Product-of-Experts for Learning Multimodal Representations in Noisy Environmentscited by 4
  21. 2022Unite and Conquer: Plug & Play Multi-Modal Synthesis using Diffusion Modelscited by 4
  22. 2023Learning multi-modal generative models with permutation-invariant encoders and tighter variational objectivescited by 4
  23. 2024Through-Wall Imaging based on WiFi Channel State Informationcited by 4
  24. 2024Multimodal Variational Autoencoder: a Barycentric Viewcited by 4
  25. 2025Cohort-Based Active Modality Acquisitioncited by 4
  26. 2025Disentanglement of Variations with Multimodal Generative Modelingcited by 4
  27. 2023Improving Normative Modeling for Multi-modal Neuroimaging Data using mixture-of-product-of-experts variational autoencoderscited by 3
  28. 2024Disentangling shared and private latent factors in multimodal Variational Autoencoderscited by 3
  29. 2025Aggregation of Dependent Expert Distributions in Multimodal Variational Autoencoderscited by 3
  30. 2025Calibrated Multimodal Representation Learning with Missing Modalitiescited by 3
  31. 2026Inference-Time Dynamic Modality Selection for Incomplete Multimodal Classificationcited by 3
  32. 2021Multimodal Adversarially Learned Inference with Factorized Discriminatorscited by 2
  33. 2022Benchmarking Multimodal Variational Autoencoders: CdSprites+ Dataset and Toolkitcited by 2
  34. 2023Streamlining Multimodal Data Fusion in Wireless Communication and Sensor Networkscited by 2
  35. 2024A Markov Random Field Multi-Modal Variational AutoEncodercited by 2
  36. 2024Multimodal ELBO with Diffusion Decoderscited by 2
  37. 2025Multi-Component VAE with Gaussian Markov Random Fieldcited by 2
  38. 2025Dynamical modeling of nonlinear latent factors in multiscale neural activity with real-time inferencecited by 2

50 most cited of 86 All 86 (JSON)