mlsanityData to 7 Oct 2026

Deep Image Synthesis from Intuitive User Input: A Review and Perspectives

arXiv 2107.04240 ↗First posted 9 Jul 2021cs.CV cs.GR cs.LGcited by 8

Not on the map. Why?

Abstract (arXiv)

In many applications of computer graphics, art and design, it is desirable for a user to provide intuitive non-image input, such as text, sketch, stroke, graph or layout, and have a computer system automatically generate photo-realistic images that adhere to the input content. While classic works that allow such automatic image content generation have followed a framework of image retrieval and composition, recent advances in deep generative models such as generative adversarial networks (GANs), variational autoencoders (VAEs), and flow-based methods have enabled more powerful and versatile image generation tasks. This paper reviews recent works for image synthesis given intuitive user input, covering advances in input versatility, image generation methodology, benchmark datasets, and evaluation metrics. This motivates new perspectives on input representation and interactivity, cross pollination between major image generation paradigms, and evaluation and comparison of generation methods.

Key facts. Cited by 8 papers in the map (data through 7 Oct 2026). Builds on 109 of its 155 references that are in the map. First posted on arXiv 9 Jul 2021.

Builds on

109 of 155 references.

References · 109

  1. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  2. 2016Semi-Supervised Classification with Graph Convolutional Networkscited by 11,282
  3. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,029
  4. 2017Graph Attention Networkscited by 8,854
  5. 2018A Style-Based Generator Architecture for Generative Adversarial Networkscited by 5,986
  6. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,451
  7. 2016Image-to-Image Translation with Conditional Adversarial Networkscited by 4,510
  8. 2016The Cityscapes Dataset for Semantic Urban Scene Understandingcited by 4,427
  9. 2014Deep Learning Face Attributes in the Wildcited by 3,679
  10. 2016DeepLab: Semantic Image Segmentation with Deep Convolutional Nets, Atrous Convolution, and Fully Connected CRFscited by 3,239
  11. 2018Encoder-Decoder with Atrous Separable Convolution for Semantic Image Segmentationcited by 3,210
  12. 2017Progressive Growing of GANs for Improved Quality, Stability, and Variationcited by 2,967
38 more
  1. 2016Improved Techniques for Training GANscited by 2,938
  2. 2015Fast R-CNNcited by 2,930
  3. 2016Perceptual Losses for Real-Time Style Transfer and Super-Resolutioncited by 2,678
  4. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  5. 2017Improved Training of Wasserstein GANscited by 2,223
  6. 2015Variational Inference with Normalizing Flowscited by 2,161
  7. 2014Conditional Generative Adversarial Netscited by 2,152
  8. 2015Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networkscited by 2,048
  9. 2016Density estimation using Real NVPcited by 2,029
  10. 2017Arbitrary Style Transfer in Real-time with Adaptive Instance Normalizationcited by 1,998
  11. 2018Spectral Normalization for Generative Adversarial Networkscited by 1,618
  12. 2018Glow: Generative Flow with Invertible 1x1 Convolutionscited by 1,539
  13. 2019Semantic Image Synthesis with Spatially-Adaptive Normalizationcited by 1,295
  14. 2017High-Resolution Image Synthesis and Semantic Manipulation with Conditional GANscited by 1,176
  15. 2016Semantic Understanding of Scenes through the ADE20K Datasetcited by 1,063
  16. 2017StarGAN: Unified Generative Adversarial Networks for Multi-Domain Image-to-Image Translationcited by 1,036
  17. 2016Least Squares Generative Adversarial Networkscited by 1,000
  18. 2016Conditional Image Generation with PixelCNN Decoderscited by 859
  19. 2016COCO-Stuff: Thing and Stuff Classes in Contextcited by 858
  20. 2016Pixel Recurrent Neural Networkscited by 814
  21. 2018Self-Attention Generative Adversarial Networkscited by 809
  22. 2018Multimodal Unsupervised Image-to-Image Translationcited by 795
  23. 2016Generative Adversarial Text to Image Synthesiscited by 735
  24. 2018Which Training Methods for GANs do actually Converge?cited by 672
  25. 2017AttnGAN: Fine-Grained Text to Image Generation with Attentional Generative Adversarial Networkscited by 664
  26. 2020NVAE: A Deep Hierarchical Variational Autoencodercited by 635
  27. 2019MaskGAN: Towards Diverse and Interactive Facial Image Manipulationcited by 624
  28. 2015Holistically-Nested Edge Detectioncited by 590
  29. 2016StackGAN: Text to Photo-realistic Image Synthesis with Stacked Generative Adversarial Networkscited by 575
  30. 2016Conditional Image Synthesis With Auxiliary Classifier GANscited by 562
  31. 2016Semantic Scene Completion from a Single Depth Imagecited by 468
  32. 2017PixelCNN++: Improving the PixelCNN with Discretized Logistic Mixture Likelihood and Other Modificationscited by 415
  33. 2015Autoencoding beyond pixels using a learned similarity metriccited by 407
  34. 2018Image Generation from Scene Graphscited by 324
  35. 2019DM-GAN: Dynamic Memory Generative Adversarial Networks for Text-to-Image Synthesiscited by 308
  36. 2021Cross-Modal Contrastive Learning for Text-to-Image Generationcited by 259
  37. 2018cGANs with Projection Discriminatorcited by 253
  38. 2019SEAN: Image Synthesis with Semantic Region-Adaptive Normalizationcited by 253

50 most cited of 109 All 109 (JSON)

Built on by

Cited by year

Citing papers per year: 2021: 1; 2022: 4; 2023: 1; 2024: 2.

8 papers.

  1. 2023Conditional Image-to-Video Generation with Latent Flow Diffusion Modelscited by 180
  2. 2021Multimodal Image Synthesis and Editing: The Generative AI Eracited by 52
  3. 2022SceneComposer: Any-Level Semantic Image Synthesiscited by 43
  4. 2022Draw Your Art Dream: Diverse Digital Art Synthesis with Multimodal Guided Diffusioncited by 30
  5. 2022User-Controllable Latent Transformer for StyleGAN Image Layout Editingcited by 28
  6. 2022CoGS: Controllable Generation and Search from Sketch and Stylecited by 14
  7. 2024Exploring the Potentials and Challenges of Deep Generative Models in Product Design Conceptioncited by 3
  8. 2024A transition towards virtual representations of visual scenescited by 0