mlsanityData to 7 Oct 2026

OptGAN: Optimizing and Interpreting the Latent Space of the Conditional Text-to-Image GANs

arXiv 2202.12929 ↗First posted 25 Feb 2022cs.CV cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Text-to-image generation intends to automatically produce a photo-realistic image, conditioned on a textual description. It can be potentially employed in the field of art creation, data augmentation, photo-editing, etc. Although many efforts have been dedicated to this task, it remains particularly challenging to generate believable, natural scenes. To facilitate the real-world applications of text-to-image synthesis, we focus on studying the following three issues: 1) How to ensure that generated samples are believable, realistic or natural? 2) How to exploit the latent space of the generator to edit a synthesized image? 3) How to improve the explainability of a text-to-image generation framework? In this work, we constructed two novel data sets (i.e., the Good & Bad bird and face data sets) consisting of successful as well as unsuccessful generated samples, according to strict criteria. To effectively and efficiently acquire high-quality images by increasing the probability of generating Good latent codes, we use a dedicated Good/Bad classifier for generated images. It is based on a pre-trained front end and fine-tuned on the basis of the proposed Good & Bad data set. After that, we present a novel algorithm which identifies semantically-understandable directions in the latent space of a conditional text-to-image GAN architecture by performing independent component analysis on the pre-trained weight values of the generator. Furthermore, we develop a background-flattening loss (BFL), to improve the background appearance in the edited image. Subsequently, we introduce linear interpolation analysis between pairs of keywords. This is extended into a similar triangular `linguistic' interpolation in order to take a deep look into what a text-to-image synthesis model has learned within the linguistic embeddings. Our data set is available at https://zenodo.org/record/6283798#.YhkN_ujMI2w.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 43 of its 75 references that are in the map. First posted on arXiv 25 Feb 2022.

Builds on

43 of 75 references.

References · 43

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  4. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  5. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  6. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  7. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  8. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,029
  9. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
  10. 2018A Style-Based Generator Architecture for Generative Adversarial Networkscited by 5,986
  11. 2017Axiomatic Attribution for Deep Networkscited by 3,350
  12. 2016Perceptual Losses for Real-Time Style Transfer and Super-Resolutioncited by 2,678
31 more
  1. 2018Large Scale GAN Training for High Fidelity Natural Image Synthesiscited by 2,609
  2. 2014Conditional Generative Adversarial Netscited by 2,152
  3. 2015Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networkscited by 2,048
  4. 2016Paying More Attention to Attention: Improving the Performance of Convolutional Neural Networks via Attention Transfercited by 1,059
  5. 2021StyleCLIP: Text-Driven Manipulation of StyleGAN Imagerycited by 990
  6. 2018Self-Attention Generative Adversarial Networkscited by 809
  7. 2019Interpreting the Latent Space of GANs for Semantic Face Editingcited by 691
  8. 2017AttnGAN: Fine-Grained Text to Image Generation with Attentional Generative Adversarial Networkscited by 664
  9. 2015Cyclical Learning Rates for Training Neural Networkscited by 627
  10. 2020GANSpace: Discovering Interpretable GAN Controlscited by 612
  11. 2016StackGAN: Text to Photo-realistic Image Synthesis with Stacked Generative Adversarial Networkscited by 575
  12. 2020Closed-Form Factorization of Latent Semantics in GANscited by 407
  13. 2019DM-GAN: Dynamic Memory Generative Adversarial Networks for Text-to-Image Synthesiscited by 308
  14. 2020Unsupervised Discovery of Interpretable Directions in the GAN Latent Spacecited by 288
  15. 2021Cross-Modal Contrastive Learning for Text-to-Image Generationcited by 259
  16. 2019Understanding Deep Networks via Extremal Perturbations and Smooth Maskscited by 257
  17. 2019On the "steerability" of generative adversarial networkscited by 247
  18. 2017StackGAN++: Realistic Image Synthesis with Stacked Generative Adversarial Networkscited by 229
  19. 2015Deep Generative Image Models using a Laplacian Pyramid of Adversarial Networkscited by 214
  20. 2019GANalyze: Toward Visual Definitions of Cognitive Image Propertiescited by 193
  21. 2013Maxout Networkscited by 186
  22. 2019MirrorGAN: Learning Text-to-image Generation by Redescriptioncited by 176
  23. 2019Semantic Hierarchy Emerges in Deep Generative Representations for Scene Synthesiscited by 106
  24. 2019Semantic Object Accuracy for Generative Text-to-Image Synthesiscited by 89
  25. 2021Cycle-Consistent Inverse GAN for Text-to-Image Synthesiscited by 27
  26. 2020Hijack-GAN: Unintended-Use of Pretrained, Black-Box GANscited by 23
  27. 2020DTGAN: Dual Attention Generative Adversarial Networks for Text-to-Image Generationcited by 14
  28. 2017Faster ICA under orthogonal constraintcited by 9
  29. 2021DiverGAN: An Efficient and Effective Single-Stage Framework for Diverse Text-to-Image Generationcited by 7
  30. 2021Dynamic hardware system for cascade SVM classification of melanomacited by 2
  31. 2021Generative Adversarial Network (GAN) and Enhanced Root Mean Square Error (ERMSE): Deep Learning for Stock Price Movement Predictioncited by 1

Built on by

Not cited yet.