# Variational Hetero-Encoder Randomized GANs for Joint Image-Text Modeling

arXiv: https://arxiv.org/abs/1905.08622 · HTML: https://mlsanity.com/paper/1905.08622/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 18 May 2019.

## Built on by (most cited first, 4 of 4)

- [Diffusion-GAN: Training GANs with Diffusion](https://mlsanity.com/paper/2206.02262/) (2022)
- [Sawtooth Factorial Topic Embeddings Guided Gamma Belief Network](https://mlsanity.com/paper/2107.02757/) (2021)
- [Guided Score identity Distillation for Data-Free One-Step Text-to-Image Generation](https://mlsanity.com/paper/2406.01561/) (2024)
- [HP-GAN: Harnessing pretrained networks for GAN improvement with FakeTwins and discriminator consistency](https://mlsanity.com/paper/2602.03039/) (2026)

## Abstract (arXiv)

For bidirectional joint image-text modeling, we develop variational hetero-encoder (VHE) randomized generative adversarial network (GAN), a versatile deep generative model that integrates a probabilistic text decoder, probabilistic image encoder, and GAN into a coherent end-to-end multi-modality learning framework. VHE randomized GAN (VHE-GAN) encodes an image to decode its associated text, and feeds the variational posterior as the source of randomness into the GAN image generator. We plug three off-the-shelf modules, including a deep topic model, a ladder-structured image encoder, and StackGAN++, into VHE-GAN, which already achieves competitive performance. This further motivates the development of VHE-raster-scan-GAN that generates photo-realistic images in not only a multi-scale low-to-high-resolution manner, but also a hierarchical-semantic coarse-to-fine fashion. By capturing and relating hierarchical semantic and visual concepts with end-to-end training, VHE-raster-scan-GAN achieves state-of-the-art performance in a wide variety of image-text multi-modality learning and generation tasks.