mlsanityData to 7 Oct 2026

Multimodal Generative Models for Scalable Weakly-Supervised Learning

arXiv 1802.05335 ↗First posted 14 Feb 2018cs.LG stat.MLcited by 200

Not on the map. Why?

Abstract (arXiv)

Multiple modalities often co-occur when describing natural phenomena. Learning a joint representation of these modalities should yield deeper and more useful representations. Previous generative approaches to multi-modal input either do not learn a joint distribution or require additional computation to handle missing data. Here, we introduce a multimodal variational autoencoder (MVAE) that uses a product-of-experts inference network and a sub-sampled training paradigm to solve the multi-modal inference problem. Notably, our model shares parameters to efficiently learn under any combination of missing modalities. We apply the MVAE on four datasets and match state-of-the-art performance using many fewer parameters. In addition, we show that the MVAE is directly applicable to weakly-supervised learning, and is robust to incomplete supervision. We then consider two case studies, one of learning image transformations---edge detection, colorization, segmentation---as a set of modalities, followed by one of machine translation between two languages. We find appealing results across this range of tasks.

Key facts. Cited by 200 papers in the map (data through 7 Oct 2026). First posted on arXiv 14 Feb 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2019: 1; 2020: 1; 2021: 26; 2022: 35; 2023: 33; 2024: 36; 2025: 34; 2026: 34.

200 papers.

Sort:

By area:

  1. 2022MultiMAE: Multi-modal Multi-task Masked Autoencoderscited by 287
  2. 2021SMIL: Multimodal Learning with Severely Missing Modalitycited by 243
  3. 2019Variational Mixture-of-Experts Autoencoders for Multi-Modal Deep Generative Modelscited by 177
  4. 2021MultiBench: Multiscale Benchmarks for Multimodal Representation Learningcited by 143
  5. 2022CyCLIP: Cyclic Contrastive Language-Image Pretrainingcited by 127
  6. 2022Foundations and Trends in Multimodal Machine Learning: Principles, Challenges, and Open Questionscited by 116
  7. 2021Generalized Multimodal ELBOcited by 86
  8. 2023Quantifying & Modeling Multimodal Interactions: An Information Decomposition Frameworkcited by 81
  9. 2024Visual Decoding and Reconstruction via EEG Embeddings with Guided Diffusioncited by 79
  10. 2021Multimodal Conditional Image Synthesis with Product-of-Experts GANscited by 75
  11. 2023Factorized Contrastive Learning: Going Beyond Multi-view Redundancycited by 73
  12. 2024Retrieval-Augmented Generation for Natural Language Processing: A Surveycited by 70
38 more
  1. 2022Training and Inference on Any-Order Autoregressive Models the Right Waycited by 64
  2. 2022Decoding Visual Neural Representations by Multimodal Learning of Brain-Visual-Linguistic Featurescited by 63
  3. 2023Identifiability Results for Multimodal Contrastive Learningcited by 58
  4. 2021Multi-VAE: Learning Disentangled View-common and View-peculiar Visual Representations for Multi-view Clusteringcited by 51
  5. 2022Multimodal Information Bottleneck: Learning Minimal Sufficient Unimodal and Multimodal Representationscited by 48
  6. 2022A survey of multimodal deep generative modelscited by 47
  7. 2021A Variational Information Bottleneck Approach to Multi-Omics Data Integrationcited by 40
  8. 2022Geometric Multimodal Contrastive Representation Learningcited by 40
  9. 2021On the Limitations of Multimodal VAEscited by 37
  10. 2023Multimodal Variational Auto-encoder based Audio-Visual Segmentationcited by 37
  11. 2023BrainCLIP: Bridging Brain and Visual-Linguistic Representation Via CLIP for Generic Natural Visual Stimulus Decodingcited by 36
  12. 2022Mitigating Modality Collapse in Multimodal VAEs via Impartial Optimizationcited by 32
  13. 2024Recent Advances in Generative AI and Large Language Models: Current Status, Challenges, and Perspectivescited by 30
  14. 2022Cross-Modal 3D Shape Generation and Manipulationcited by 27
  15. 2022Representation Uncertainty in Self-Supervised Learning as Variational Inferencecited by 22
  16. 2024Efficient LLM Comparative Assessment: a Product of Experts Framework for Pairwise Comparisonscited by 22
  17. 2021Learning Multimodal VAEs through Mutual Supervisioncited by 18
  18. 2022Medical Image Segmentation on MRI Images with Missing Modalities: A Reviewcited by 18
  19. 2023Calibrating Multimodal Learningcited by 17
  20. 2023MoCaE: Mixture of Calibrated Experts Significantly Improves Object Detectioncited by 17
  21. 2021Reconstructing Nonlinear Dynamical Systems from Multi-Modal Time Seriescited by 16
  22. 2023Towards Anytime Classification in Early-Exit Architectures by Enforcing Conditional Monotonicitycited by 16
  23. 2025A Survey on Active Feature Acquisition Strategiescited by 16
  24. 2021Gaussian Mixture Variational Autoencoder with Contrastive Learning for Multi-Label Classificationcited by 15
  25. 2024Leveraging Foundation Models for Multi-modal Federated Learning with Incomplete Modalitycited by 14
  26. 2021Multi-Modal Mutual Information (MuMMI) Training for Robust Self-Supervised Deep Reinforcement Learningcited by 13
  27. 2022Deep Multi-Modal Structural Equations For Causal Effect Estimation With Unstructured Proxiescited by 13
  28. 2022Integrating Multimodal Data for Joint Generative Modeling of Complex Dynamicscited by 13
  29. 2023Exploring and Exploiting Uncertainty for Incomplete Multi-View Classificationcited by 13
  30. 2023Unified Brain MR-Ultrasound Synthesis using Multi-Modal Hierarchical Representationscited by 13
  31. 2024Unity by Diversity: Improved Representation Learning in Multimodal VAEscited by 13
  32. 2025Robust Multimodal Survival Prediction with the Latent Differentiation Conditional Variational AutoEncodercited by 13
  33. 2021Variational Selective Autoencoder: Learning from Partially-Observed Heterogeneous Datacited by 10
  34. 2021Lifelong Mixture of Variational Autoencoderscited by 10
  35. 2023Multi-modal Variational Autoencoders for normative modelling across multiple imaging modalitiescited by 10
  36. 2021Multimodal Variational Autoencoders for Semi-Supervised Learning: In Defense of Product-of-Expertscited by 9
  37. 2021FairyTailor: A Multimodal Generative Framework for Storytellingcited by 9
  38. 2022Multimodal sensor fusion in the latent representation spacecited by 9

50 most cited of 200 All 200 (JSON)