mlsanityData to 7 Oct 2026

A survey of multimodal deep generative models

arXiv 2207.02127 ↗First posted 5 Jul 2022cs.LG stat.MLcited by 47

Not on the map. Why?

Abstract (arXiv)

Multimodal learning is a framework for building models that make predictions based on different types of modalities. Important challenges in multimodal learning are the inference of shared representations from arbitrary modalities and cross-modal generation via these representations; however, achieving this requires taking the heterogeneous nature of multimodal data into account. In recent years, deep generative models, i.e., generative models in which distributions are parameterized by deep neural networks, have attracted much attention, especially variational autoencoders, which are suitable for accomplishing the above challenges because they can consider heterogeneity and infer good representations of data. Therefore, various multimodal generative models based on variational autoencoders, called multimodal deep generative models, have been proposed in recent years. In this paper, we provide a categorized survey of studies on multimodal deep generative models.

Key facts. Cited by 47 papers in the map (data through 7 Oct 2026). Builds on 63 of its 139 references that are in the map. First posted on arXiv 5 Jul 2022.

Builds on

63 of 139 references.

References · 63

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018Representation Learning with Contrastive Predictive Codingcited by 7,662
  3. 2022Generative Adversarial Networkscited by 5,603
  4. 2016Image-to-Image Translation with Conditional Adversarial Networkscited by 4,510
  5. 2017Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithmscited by 4,283
  6. 2014Deep Learning Face Attributes in the Wildcited by 3,679
  7. 2016Categorical Reparameterization with Gumbel-Softmaxcited by 2,708
  8. 2015Variational Inference with Normalizing Flowscited by 2,161
  9. 2014Conditional Generative Adversarial Netscited by 2,152
  10. 2016WaveNet: A Generative Model for Raw Audiocited by 1,710
  11. 2019Dream to Control: Learning Behaviors by Latent Imaginationcited by 1,598
  12. 2018Glow: Generative Flow with Invertible 1x1 Convolutionscited by 1,539
38 more
  1. 2019Contrastive Multiview Codingcited by 1,297
  2. 2018Learning Latent Dynamics for Planning from Pixelscited by 1,221
  3. 2016The Concrete Distribution: A Continuous Relaxation of Discrete Random Variablescited by 1,135
  4. 2017Multimodal Machine Learning: A Survey and Taxonomycited by 1,004
  5. 2016Pixel Recurrent Neural Networkscited by 814
  6. 2018Recurrent World Models Facilitate Policy Evolutioncited by 761
  7. 2017Unpaired Image-to-Image Translation using Cycle-Consistent Adversarial Networkscited by 707
  8. 2017Unsupervised Image-to-Image Translation Networkscited by 602
  9. 2016StackGAN: Text to Photo-realistic Image Synthesis with Stacked Generative Adversarial Networkscited by 575
  10. 2015Importance Weighted Autoencoderscited by 454
  11. 2018Pyro: Deep Universal Probabilistic Programmingcited by 379
  12. 2017Wasserstein Auto-Encoderscited by 371
  13. 2018Diverse Image-to-Image Translation via Disentangled Representationscited by 296
  14. 2020Self-Supervised MultiModal Versatile Networkscited by 276
  15. 2017Zero-Shot Learning -- The Good, the Bad and the Uglycited by 254
  16. 2017Learning to Estimate 3D Hand Pose from Single RGB Imagescited by 234
  17. 2019Recurrent Independent Mechanismscited by 219
  18. 2017A Multimodal Anomaly Detector for Robot-Assisted Feeding Using an LSTM-based Variational Autoencodercited by 209
  19. 2018Generalized Zero- and Few-Shot Learning via Aligned Variational Autoencoderscited by 206
  20. 2018Multimodal Generative Models for Scalable Weakly-Supervised Learningcited by 200
  21. 2019Variational Mixture-of-Experts Autoencoders for Multi-Modal Deep Generative Modelscited by 177
  22. 2018Learning Factorized Multimodal Representationscited by 171
  23. 2020A Review of Generalized Zero-Shot Learning Methodscited by 159
  24. 2013A Survey on Multi-view Learningcited by 152
  25. 2018CodeSLAM - Learning a Compact, Optimisable Representation for Dense Visual SLAMcited by 151
  26. 2018Recent Advances in Autoencoder-Based Representation Learningcited by 130
  27. 2016A Survey of Multi-View Representation Learningcited by 107
  28. 2017Multi-Level Variational Autoencoder: Learning Disentangled Representations from Grouped Observationscited by 100
  29. 2019Continual Learning for Robotics: Definition, Framework, Learning Strategies, Opportunities and Challengescited by 92
  30. 2019Multimodal Intelligence: Representation Learning, Information Fusion, and Applicationscited by 89
  31. 2016Joint Multimodal Learning with Deep Generative Modelscited by 87
  32. 2021Generalized Multimodal ELBOcited by 86
  33. 2021Coordination Among Neural Modules Through a Shared Global Workspacecited by 83
  34. 2016A Comprehensive Survey on Cross-modal Retrievalcited by 62
  35. 2020Multimodal Generative Learning Utilizing Jensen-Shannon-Divergencecited by 48
  36. 2020Multimodal Deep Generative Models for Trajectory Prediction: A Conditional Variational Autoencoder Approachcited by 47
  37. 2016Deep Variational Canonical Correlation Analysiscited by 44
  38. 2017Generative Models of Visually Grounded Imaginationcited by 44

50 most cited of 63 All 63 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 3; 2023: 11; 2024: 19; 2025: 11; 2026: 3.

47 papers.

Sort:

By area:

  1. 2023A Comprehensive Survey of AI-Generated Content (AIGC): A History of Generative AI from GAN to ChatGPTcited by 221
  2. 2022Versatile Diffusion: Text, Images and Variations All in One Diffusion Modelcited by 199
  3. 2023Human Motion Generation: A Surveycited by 95
  4. 2023A Comprehensive Survey on Applications of Transformers for Deep Learning Taskscited by 70
  5. 2024What to align in multimodal contrastive learning?cited by 41
  6. 2024Recent Advances in Generative AI and Large Language Models: Current Status, Challenges, and Perspectivescited by 30
  7. 2024Unity by Diversity: Improved Representation Learning in Multimodal VAEscited by 13
  8. 2024On the Limitations and Prospects of Machine Unlearning for Generative AIcited by 12
  9. 2024A Survey on Integrated Sensing, Communication, and Computationcited by 12
  10. 2023Interpretable and intervenable ultrasonography-based machine learning models for pediatric appendicitiscited by 11
  11. 2023Lewis's Signaling Game as beta-VAE For Natural Word Lengths and Segmentscited by 11
  12. 2023An Overview on Generative AI at Scale with Edge-Cloud Computingcited by 9
35 more
  1. 2024Applications of Generative AI (GAI) for Mobile and Wireless Networking: A Surveycited by 9
  2. 2024Audio-Agent: Leveraging LLMs For Audio Generation, Editing and Compositioncited by 8
  3. 2024Physics-Informed Latent Diffusion for Multimodal Brain MRI Synthesiscited by 7
  4. 2023A multimodal dynamical variational autoencoder for audiovisual speech representation learningcited by 6
  5. 2024Constructive Approach to Bidirectional Influence between Qualia Structure and Language Emergencecited by 5
  6. 2024FolAI: Synchronized Foley Sound Generation with Semantic and Temporal Alignmentcited by 5
  7. 2025Efficient Utility-Preserving Machine Unlearning with Implicit Gradient Surgerycited by 5
  8. 2023A Survey on Image-text Multimodal Modelscited by 4
  9. 2025Zeus: Zero-shot LLM Instruction for Union Segmentation in Multimodal Medical Imagingcited by 4
  10. 2025SwarmDiff: Swarm Robotic Trajectory Planning in Cluttered Environments via Diffusion Transformercited by 3
  11. 2025Multimodal Representation Learning and Fusioncited by 3
  12. 2026MechVQA: Benchmarking and Enhancing Multimodal LLMs on Comprehensive Mechanical Drawing Understandingcited by 3
  13. 2022Benchmarking Multimodal Variational Autoencoders: CdSprites+ Dataset and Toolkitcited by 2
  14. 2023A Survey on Interpretable Cross-modal Reasoningcited by 2
  15. 2024From Concrete to Abstract: A Multimodal Generative Approach to Abstract Concept Learningcited by 2
  16. 2025Fusing Cross-Domain Knowledge from Multimodal Data to Solve Problems in the Physical Worldcited by 2
  17. 2023Sense, Imagine, Act: Multimodal Perception Improves Model-Based Reinforcement Learning for Head-to-Head Autonomous Racingcited by 1
  18. 2023Improving Multimodal Joint Variational Autoencoders through Normalizing Flows and Correlation Analysiscited by 1
  19. 2024A tutorial on multi-view autoencoders using the multi-view-AE librarycited by 1
  20. 2024Russian-Language Multimodal Dataset for Automatic Summarization of Scientific Paperscited by 1
  21. 2024Learning Multimodal Latent Space with EBM Prior and MCMC Inferencecited by 1
  22. 2024Analyzing Multimodal Integration in the Variational Autoencoder from an Information-Theoretic Perspectivecited by 1
  23. 2025Hierarchical Self-Attention: Generalizing Neural Attention Mechanics to Multi-Scale Problemscited by 1
  24. 2026Instruction-Driven 3D Facial Expression Generation and Transitioncited by 1
  25. 2022A Generalized EigenGame with Extensions to Multiview Representation Learningcited by 0
  26. 2024Generative Kaleidoscopic Networkscited by 0
  27. 2024Bridging Language, Vision and Action: Multimodal VAEs in Robotic Manipulation Taskscited by 0
  28. 2024Bayesian Structural Model Updating with Multimodal Variational Autoencodercited by 0
  29. 2024Enhancing Unimodal Latent Representations in Multimodal VAEs through Iterative Amortized Inferencecited by 0
  30. 2025Co-Creative Learning via Metropolis-Hastings Interaction between Humans and AIcited by 0
  31. 2025Deep Generative Methods and Tire Architecture Designcited by 0
  32. 2025Video Understanding by Design: How Datasets Shape Video Modelscited by 0
  33. 2025Testing chatbots on the creation of encoders for audio conditioned image generationcited by 0
  34. 2025Reconstruction-Driven Multimodal Representation Learning for Automated Media Understandingcited by 0
  35. 2026CogGen: Cognitive-Load-Inspired Fully Unsupervised Deep Generative Modeling for Compressively Sampled MRI Reconstructioncited by 0