mlsanityData to 7 Oct 2026

Multimodal Conditional Image Synthesis with Product-of-Experts GANs

arXiv 2112.05130 ↗First posted 9 Dec 2021cs.CVcited by 75

Not on the map. Why?

Cited by 27 in Diffusion models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Existing conditional image synthesis frameworks generate images based on user inputs in a single modality, such as text, segmentation, sketch, or style reference. They are often unable to leverage multimodal user inputs when available, which reduces their practicality. To address this limitation, we propose the Product-of-Experts Generative Adversarial Networks (PoE-GAN) framework, which can synthesize images conditioned on multiple input modalities or any subset of them, even the empty set. PoE-GAN consists of a product-of-experts generator and a multimodal multiscale projection discriminator. Through our carefully designed training scheme, PoE-GAN learns to synthesize images with high quality and diversity. Besides advancing the state of the art in multimodal conditional image synthesis, PoE-GAN also outperforms the best existing unimodal conditional image synthesis approaches when tested in the unimodal setting. The project website is available at https://deepimagination.github.io/PoE-GAN .

Key facts. Cited by 75 papers in the map (data through 7 Oct 2026). Builds on 70 of its 83 references that are in the map. First posted on arXiv 9 Dec 2021.

Builds on

70 of 83 references.

References · 70

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  5. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  6. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  7. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  8. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,029
  9. 2019Momentum Contrast for Unsupervised Visual Representation Learningcited by 7,482
  10. 2018A Style-Based Generator Architecture for Generative Adversarial Networkscited by 5,986
  11. 2016Image-to-Image Translation with Conditional Adversarial Networkscited by 4,510
  12. 2021Zero-Shot Text-to-Image Generationcited by 4,276
38 more
  1. 2014Deep Learning Face Attributes in the Wildcited by 3,679
  2. 2019Analyzing and Improving the Image Quality of StyleGANcited by 3,580
  3. 2016DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFscited by 3,239
  4. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,229
  5. 2017Progressive Growing of GANs for Improved Quality, Stability, and Variationcited by 2,967
  6. 2018Large Scale GAN Training for High Fidelity Natural Image Synthesiscited by 2,609
  7. 2015Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networkscited by 2,048
  8. 2017Arbitrary Style Transfer in Real-time with Adaptive Instance Normalizationcited by 1,998
  9. 2018Spectral Normalization for Generative Adversarial Networkscited by 1,618
  10. 2019Semantic Image Synthesis with Spatially-Adaptive Normalizationcited by 1,295
  11. 2020Training Generative Adversarial Networks with Limited Datacited by 1,184
  12. 2021Alias-Free Generative Adversarial Networkscited by 1,181
  13. 2017High-Resolution Image Synthesis and Semantic Manipulation with Conditional GANscited by 1,176
  14. 2016Deep Variational Information Bottleneckcited by 915
  15. 2016COCO-Stuff: Thing and Stuff Classes in Contextcited by 858
  16. 2019CNN-generated images are surprisingly easy to spot... for nowcited by 842
  17. 2018Multimodal Unsupervised Image-to-Image Translationcited by 795
  18. 2016Generative Adversarial Text to Image Synthesiscited by 735
  19. 2020Encoding in Style: a StyleGAN Encoder for Image-to-Image Translationcited by 727
  20. 2018Which Training Methods for GANs do actually Converge?cited by 672
  21. 2017AttnGAN: Fine-Grained Text to Image Generation with Attentional Generative Adversarial Networkscited by 664
  22. 2020Contrastive Learning for Unpaired Image-to-Image Translationcited by 637
  23. 2020Contrastive Learning of Medical Visual Representations from Paired Images and Textcited by 636
  24. 2020NVAE: A Deep Hierarchical Variational Autoencodercited by 635
  25. 2019MaskGAN: Towards Diverse and Interactive Facial Image Manipulationcited by 624
  26. 2015Holistically-Nested Edge Detectioncited by 590
  27. 2016StackGAN: Text to Photo-realistic Image Synthesis with Stacked Generative Adversarial Networkscited by 575
  28. 2016Conditional Image Synthesis With Auxiliary Classifier GANscited by 562
  29. 2019Making Convolutional Networks Shift-Invariant Againcited by 390
  30. 2016Ladder Variational Autoencoderscited by 315
  31. 2019DM-GAN: Dynamic Memory Generative Adversarial Networks for Text-to-Image Synthesiscited by 308
  32. 2017Toward Multimodal Image-to-Image Translationcited by 295
  33. 2020What makes fake images detectable? Understanding properties that generalizecited by 272
  34. 2020Very Deep VAEs Generalize Autoregressive Models and Can Outperform Them on Imagescited by 261
  35. 2021Cross-Modal Contrastive Learning for Text-to-Image Generationcited by 259
  36. 2018cGANs with Projection Discriminatorcited by 253
  37. 2017Improved Texture Networks: Maximizing Quality and Diversity in Feed-forward Stylization and Texture Synthesiscited by 215
  38. 2018Multimodal Generative Models for Scalable Weakly-Supervised Learningcited by 200

50 most cited of 70 All 70 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 1; 2022: 25; 2023: 28; 2024: 8; 2025: 8; 2026: 5.

75 papers.

Sort:

By area:

  1. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,466
  2. 2023T2I-Adapter: Learning Adapters to Dig out More Controllable Ability for Text-to-Image Diffusion Modelscited by 1,339
  3. 2022Multi-Concept Customization of Text-to-Image Diffusioncited by 1,099
  4. 2022eDiff-I: Text-to-Image Diffusion Models with an Ensemble of Expert Denoiserscited by 921
  5. 2022GET3D: A Generative Model of High Quality 3D Textured Shapes Learned from Imagescited by 446
  6. 2023OmniObject3D: Large-Vocabulary 3D Object Dataset for Realistic Perception, Reconstruction and Generationcited by 324
  7. 2023Ablating Concepts in Text-to-Image Diffusion Modelscited by 312
  8. 2023Training-Free Layout Control with Cross-Attention Guidancecited by 312
  9. 2023Composer: Creative and Controllable Image Synthesis with Composable Conditionscited by 303
  10. 2022The Role of ImageNet Classes in Fr'echet Inception Distancecited by 211
  11. 2022Sketch-Guided Text-to-Image Diffusion Modelscited by 180
  12. 2022ReCo: Region-Controlled Text-to-Image Generationcited by 177
38 more
  1. 2023Dense Text-to-Image Generation with Attention Modulationcited by 162
  2. 2022Pretraining is All You Need for Image-to-Image Translationcited by 160
  3. 20234M: Massively Multimodal Masked Modelingcited by 118
  4. 2023DiffCollage: Parallel Generation of Large Content with Diffusion Modelscited by 100
  5. 2025Cosmos-Transfer1: Conditional World Generation with Adaptive Multimodal Controlcited by 95
  6. 2022InfiniteNature-Zero: Learning Perpetual View Generation of Natural Scenes from Single Imagescited by 82
  7. 2023Orthogonal Adaptation for Modular Customization of Diffusion Modelscited by 70
  8. 2023Freestyle Layout-to-Image Synthesiscited by 67
  9. 2021Multimodal Image Synthesis and Editing: The Generative AI Eracited by 52
  10. 2022SceneComposer: Any-Level Semantic Image Synthesiscited by 43
  11. 2023Modulating Pretrained Diffusion Models for Multimodal Image Synthesiscited by 23
  12. 2024Efficient LLM Comparative Assessment: a Product of Experts Framework for Pairwise Comparisonscited by 22
  13. 2024LoRACLR: Contrastive Adaptation for Customization of Diffusion Modelscited by 20
  14. 2023Steered Diffusion: A Generalized Framework for Plug-and-Play Conditional Image Synthesiscited by 18
  15. 2023Towards Anytime Classification in Early-Exit Architectures by Enforcing Conditional Monotonicitycited by 16
  16. 2022Name Your Style: An Arbitrary Artist-aware Image Style Transfercited by 14
  17. 2022CoGS: Controllable Generation and Search from Sketch and Stylecited by 14
  18. 2025Erasing More Than Intended? How Concept Erasure Degrades the Generation of Non-Target Conceptscited by 12
  19. 2023Always Clear Days: Degradation Type and Severity Aware All-In-One Adverse Weather Removalcited by 11
  20. 2023Multi-modal Latent Diffusioncited by 9
  21. 2023Controllable Visual-Tactile Synthesiscited by 8
  22. 2023Vision + Language Applications: A Surveycited by 8
  23. 2022End-to-End Visual Editing with a Generatively Pre-Trained Artistcited by 7
  24. 2022Content-Based Search for Deep Generative Modelscited by 7
  25. 20233D Generative Model Latent Disentanglement via Local Eigenprojectioncited by 6
  26. 2023Class-Guided Image-to-Image Diffusion: Cell Painting from Brightfield Images with Class Labelscited by 6
  27. 2023Unlocking Pre-trained Image Backbones for Semantic Image Synthesiscited by 6
  28. 2024FACEMUG: A Multimodal Generative and Fusion Framework for Local Facial Editingcited by 6
  29. 2025Product of Experts for Visual Generationcited by 6
  30. 2022Image Generation with Multimodal Priors using Denoising Diffusion Probabilistic Modelscited by 5
  31. 2022Vitruvio: 3D Building Meshes via Single Perspective Sketchescited by 5
  32. 2023PARASOL: Parametric Style Control for Diffusion Image Synthesiscited by 5
  33. 2024Text-to-Image Cross-Modal Generation: A Systematic Reviewcited by 5
  34. 2022Synthesizing Photorealistic Virtual Humans Through Cross-modal Disentanglementcited by 4
  35. 2022Generalized Product-of-Experts for Learning Multimodal Representations in Noisy Environmentscited by 4
  36. 2022Unite and Conquer: Plug & Play Multi-Modal Synthesis using Diffusion Modelscited by 4
  37. 2022Disentangling Random and Cyclic Effects in Time-Lapse Sequencescited by 3
  38. 2022Intelligent Painter: Picture Composition With Resampling Diffusion Modelcited by 3

50 most cited of 75 All 75 (JSON)