mlsanityData to 7 Oct 2026

DualVAE: Controlling Colours of Generated and Real Images

arXiv 2305.18769 ↗First posted 30 May 2023cs.CV cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Colour controlled image generation and manipulation are of interest to artists and graphic designers. Vector Quantised Variational AutoEncoders (VQ-VAEs) with autoregressive (AR) prior are able to produce high quality images, but lack an explicit representation mechanism to control colour attributes. We introduce DualVAE, a hybrid representation model that provides such control by learning disentangled representations for colour and geometry. The geometry is represented by an image intensity mapping that identifies structural features. The disentangled representation is obtained by two novel mechanisms: (i) a dual branch architecture that separates image colour attributes from geometric attributes, and (ii) a new ELBO that trains the combined colour and geometry representations. DualVAE can control the colour of generated images, and recolour existing images by transferring the colour latent representation obtained from an exemplar image. We demonstrate that DualVAE generates images with FID nearly two times better than VQ-GAN on a diverse collection of datasets, including animated faces, logos and artistic landscapes.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 42 of its 56 references that are in the map. First posted on arXiv 30 May 2023.

Builds on

42 of 56 references.

References · 42

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  3. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,029
  4. 2022Generative Adversarial Networkscited by 5,603
  5. 2016Layer Normalizationcited by 5,579
  6. 2016Image-to-Image Translation with Conditional Adversarial Networkscited by 4,510
  7. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  8. 2013Auto-Encoding Variational Bayescited by 3,980
  9. 2019Analyzing and Improving the Image Quality of StyleGANcited by 3,580
  10. 2018Group Normalizationcited by 1,514
  11. 2019Generating Diverse High-Fidelity Images with VQ-VAE-2cited by 1,376
  12. 2021Palette: Image-to-Image Diffusion Modelscited by 1,296
30 more
  1. 2020Training Generative Adversarial Networks with Limited Datacited by 1,184
  2. 2022MaskGIT: Masked Generative Image Transformercited by 1,082
  3. 2016Realtime Multi-Person 2D Pose Estimation using Part Affinity Fieldscited by 1,061
  4. 2015Holistically-Nested Edge Detectioncited by 590
  5. 2018Assessing Generative Models via Precision and Recallcited by 401
  6. 2015Large-scale Classification of Fine-Art Paintings: Learning The Right Metric on The Right Featurecited by 181
  7. 2017Deep Photo Style Transfercited by 168
  8. 2018A Variational U-Net for Conditional Appearance and Shape Generationcited by 124
  9. 2021ImageBART: Bidirectional Context with Multinomial Diffusion for Autoregressive Image Synthesiscited by 122
  10. 2021Colorization Transformercited by 89
  11. 2016Disentangling factors of variation in deep representations using adversarial trainingcited by 87
  12. 2018Variational Autoencoders Pursue PCA Directions (by Accident)cited by 84
  13. 2020Instance-aware Image Colorizationcited by 76
  14. 2018Unsupervised Discovery of Object Landmarks as Structural Representationscited by 73
  15. 2020SurVAE Flows: Surjections to Bridge the Gap between VAEs and Flowscited by 71
  16. 2021Aligning Latent and Image Spaces to Connect the Unconnectablecited by 70
  17. 2020HistoGAN: Controlling Colors of GAN-Generated and Real Images via Color Histogramscited by 64
  18. 2019ChromaGAN: Adversarial Picture Colorization with Semantic Class Distributioncited by 60
  19. 2018Disentangling Factors of Variation with Cycle-Consistent Variational Auto-Encoderscited by 49
  20. 2016Learning Diverse Image Colorizationcited by 43
  21. 2017Unsupervised Diverse Colorization via Generative Adversarial Networkscited by 30
  22. 2019Make a Face: Towards Arbitrary High Fidelity Face Manipulationcited by 29
  23. 2017Logo Synthesis and Manipulation with Clustered Generative Adversarial Networkscited by 20
  24. 2019Sensor-Independent Illumination Estimation for DNN Modelscited by 18
  25. 2020Multilinear Latent Conditioning for Generating Unseen Attribute Combinationscited by 14
  26. 2019Unsupervised Robust Disentangling of Latent Characteristics for Image Synthesiscited by 12
  27. 2019Disentangling Content and Style via Unsupervised Geometry Distillationcited by 9
  28. 2020DeepHist: Differentiable Joint and Color Histogram Layers for Image-to-Image Translationcited by 9
  29. 2022UniColor: A Unified Framework for Multi-Modal Colorization with Transformercited by 7
  30. 2019Rethinking Generative Mode Coverage: A Pointwise Guaranteed Approachcited by 3

Built on by

Not cited yet.