# Describe What to Change: A Text-guided Unsupervised Image-to-Image Translation Approach

arXiv: https://arxiv.org/abs/2008.04200 · HTML: https://mlsanity.com/paper/2008.04200/

## Key facts

- Cited by 29 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 10 Aug 2020.

## Built on by (most cited first, 29 of 29)

- [StyleCLIP: Text-Driven Manipulation of StyleGAN Imagery](https://mlsanity.com/paper/2103.17249/) (2021)
- [Uncovering the Disentanglement Capability in Text-to-Image Diffusion Models](https://mlsanity.com/paper/2212.08698/) (2022)
- [Talk-to-Edit: Fine-Grained Facial Editing via Dialog](https://mlsanity.com/paper/2109.04425/) (2021)
- [EnvEdit: Environment Editing for Vision-and-Language Navigation](https://mlsanity.com/paper/2203.15685/) (2022)
- [Adversarial Text-to-Image Synthesis: A Review](https://mlsanity.com/paper/2101.09983/) (2021)
- [Improving Generation and Evaluation of Visual Stories via Semantic Consistency](https://mlsanity.com/paper/2105.10026/) (2021)
- [Smoothing the Disentangled Latent Style Space for Unsupervised Image-to-Image Translation](https://mlsanity.com/paper/2106.09016/) (2021)
- [Predict, Prevent, and Evaluate: Disentangled Text-Driven Image Manipulation Empowered by Pre-Trained Vision-Language Model](https://mlsanity.com/paper/2111.13333/) (2021)
- [Draw Your Art Dream: Diverse Digital Art Synthesis with Multimodal Guided Diffusion](https://mlsanity.com/paper/2209.13360/) (2022)
- [DeltaEdit: Exploring Text-free Training for Text-Driven Image Manipulation](https://mlsanity.com/paper/2303.06285/) (2023)
- [Towards Open-World Text-Guided Face Image Generation and Manipulation](https://mlsanity.com/paper/2104.08910/) (2021)
- [Language-guided Human Motion Synthesis with Atomic Actions](https://mlsanity.com/paper/2308.09611/) (2023)
- [DE-Net: Dynamic Text-guided Image Editing Adversarial Networks](https://mlsanity.com/paper/2206.01160/) (2022)
- [DR-GAN: Distribution Regularization for Text-to-Image Generation](https://mlsanity.com/paper/2204.07945/) (2022)
- [ISF-GAN: An Implicit Style Function for High-Resolution Image-to-Image Translation](https://mlsanity.com/paper/2109.12492/) (2021)
- [CoralStyleCLIP: Co-optimized Region and Layer Selection for Image Editing](https://mlsanity.com/paper/2303.05031/) (2023)
- [Fake News Detection and Manipulation Reasoning via Large Vision-Language Models](https://mlsanity.com/paper/2407.02042/) (2024)
- [Semi-supervised reference-based sketch extraction using a contrastive learning framework](https://mlsanity.com/paper/2407.14026/) (2024)
- [A Survey of Multimodal Composite Editing and Retrieval](https://mlsanity.com/paper/2409.05405/) (2024)
- [Pixel VQ-VAEs for Improved Pixel Art Representation](https://mlsanity.com/paper/2203.12130/) (2022)
- [Aerial Diffusion: Text Guided Ground-to-Aerial View Translation from a Single Image using Diffusion Models](https://mlsanity.com/paper/2303.11444/) (2023)
- [ALR-GAN: Adaptive Layout Refinement for Text-to-Image Synthesis](https://mlsanity.com/paper/2304.06297/) (2023)
- [360PanT: Training-Free Text-Driven 360-Degree Panorama-to-Panorama Translation](https://mlsanity.com/paper/2409.08397/) (2024)
- [A comprehensive survey on semantic facial attribute editing using generative adversarial networks](https://mlsanity.com/paper/2205.10587/) (2022)
- [DeltaSpace: A Semantic-aligned Feature Space for Flexible Text-guided Image Editing](https://mlsanity.com/paper/2310.08785/) (2023)
- [AMSS-Net: Audio Manipulation on User-Specified Sources with Textual Queries](https://mlsanity.com/paper/2104.13553/) (2021)
- [Revealing Directions for Text-guided 3D Face Editing](https://mlsanity.com/paper/2410.04965/) (2024)
- [Leveraging Local Domains for Image-to-Image Translation](https://mlsanity.com/paper/2109.04468/) (2021)
- [Style Generation: Image Synthesis based on Coarsely Matched Texts](https://mlsanity.com/paper/2309.04608/) (2023)

## Abstract (arXiv)

Manipulating visual attributes of images through human-written text is a very challenging task. On the one hand, models have to learn the manipulation without the ground truth of the desired output. On the other hand, models have to deal with the inherent ambiguity of natural language. Previous research usually requires either the user to describe all the characteristics of the desired image or to use richly-annotated image captioning datasets. In this work, we propose a novel unsupervised approach, based on image-to-image translation, that alters the attributes of a given image through a command-like sentence such as "change the hair color to black". Contrarily to state-of-the-art approaches, our model does not require a human-annotated dataset nor a textual description of all the attributes of the desired image, but only those that have to be modified. Our proposed model disentangles the image content from the visual attributes, and it learns to modify the latter using the textual description, before generating a new image from the content and the modified attribute representation. Because text might be inherently ambiguous (blond hair may refer to different shadows of blond, e.g. golden, icy, sandy), our method generates multiple stochastic versions of the same translation. Experiments show that the proposed model achieves promising performances on two large-scale public datasets: CelebA and CUB. We believe our approach will pave the way to new avenues of research combining textual and speech commands with visual attributes.