mlsanityData to 7 Oct 2026

Learning Multimodal Latent Generative Models with Energy-Based Prior

arXiv 2409.19862 ↗First posted 30 Sep 2024cs.LG cs.CVcited by 7

Not on the map. Why?

Abstract (arXiv)

Multimodal generative models have recently gained significant attention for their ability to learn representations across various modalities, enhancing joint and cross-generation coherence. However, most existing works use standard Gaussian or Laplacian distributions as priors, which may struggle to capture the diverse information inherent in multiple data types due to their unimodal and less informative nature. Energy-based models (EBMs), known for their expressiveness and flexibility across various tasks, have yet to be thoroughly explored in the context of multimodal generative models. In this paper, we propose a novel framework that integrates the multimodal latent generative model with the EBM. Both models can be trained jointly through a variational scheme. This approach results in a more expressive and informative prior, better-capturing of information across multiple modalities. Our experiments validate the proposed model, demonstrating its superior generation coherence.

Key facts. Cited by 7 papers in the map (data through 7 Oct 2026). Builds on 32 of its 40 references that are in the map. First posted on arXiv 30 Sep 2024.

Builds on

32 of 40 references.

References · 32

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  3. 2020Score-Based Generative Modeling through Stochastic Differential Equationscited by 8,964
  4. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  5. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,595
  6. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,612
  7. 2017Multimodal Machine Learning: A Survey and Taxonomycited by 1,004
  8. 2021Vector Quantized Diffusion Model for Text-to-Image Synthesiscited by 773
  9. 2020NVAE: A Deep Hierarchical Variational Autoencodercited by 635
  10. 2016Ladder Variational Autoencoderscited by 315
  11. 2023Photorealistic Video Generation with Diffusion Modelscited by 304
  12. 2021Cross-Modal Contrastive Learning for Text-to-Image Generationcited by 259
20 more
  1. 2018Multimodal Generative Models for Scalable Weakly-Supervised Learningcited by 200
  2. 2019Controllable Text-to-Image Generationcited by 187
  3. 2019Variational Mixture-of-Experts Autoencoders for Multi-Modal Deep Generative Modelscited by 177
  4. 2019Learning Non-Convergent Non-Persistent Short-Run MCMC Toward Energy-Based Modelcited by 147
  5. 2020Improved Contrastive Divergence Training of Energy Based Modelscited by 127
  6. 2020Learning Energy-Based Models by Diffusion Recovery Likelihoodcited by 122
  7. 2019On the Anatomy of MCMC-Based Maximum Likelihood Learning of Energy-Based Modelscited by 105
  8. 2020Learning Latent Space Energy-Based Prior Modelcited by 104
  9. 2017Long Text Generation via Adversarial Training with Leaked Informationcited by 100
  10. 2021Generalized Multimodal ELBOcited by 86
  11. 2023Understanding and Constructing Latent Modality Structures in Multi-modal Representation Learningcited by 53
  12. 2017Adversarial Ranking for Language Generationcited by 50
  13. 2020Multimodal Generative Learning Utilizing Jensen-Shannon-Divergencecited by 48
  14. 2021VX2TEXT: End-to-End Learning of Video-Based Text Generation From Multimodal Inputscited by 42
  15. 2020Joint Training of Variational Auto-Encoder and Latent Energy-Based Modelcited by 30
  16. 2022Adaptive Multi-stage Density Ratio Estimation for Learning Latent Space Energy-based Modelcited by 18
  17. 2023Learning Joint Latent Space EBM Prior Model for Multi-layer Generatorcited by 8
  18. 2023Learning Hierarchical Features with Joint Latent Space Energy-Based Priorcited by 6
  19. 2023Learning Energy-based Model via Dual-MCMC Teachingcited by 6
  20. 2024Learning Multimodal Latent Space with EBM Prior and MCMC Inferencecited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 3; 2026: 3.

7 papers.

  1. 2024Multimodal Variational Autoencoder: a Barycentric Viewcited by 4
  2. 2025Learning Latent Energy-Based Models via Interacting Particle Langevin Dynamicscited by 3
  3. 2026Hellinger Multimodal Variational Autoencoderscited by 2
  4. 2025Kolmogorov-Arnold Energy Models: Fast, Interpretable Generative Modelingcited by 1
  5. 2025ShaLa: Multimodal Shared Latent Space Modellingcited by 1
  6. 2026MUNI: Multimodal Unified Latent Diffusion for Coherent Any-to-Any Generationcited by 1
  7. 2026H"older++: Improving the Quality-Coherence Trade-off in Multimodal VAEscited by 0