Target-Free Text-guided Image Manipulation
Not on the map. Why?
Abstract (arXiv)
We tackle the problem of target-free text-guided image manipulation, which requires one to modify the input reference image based on the given text instruction, while no ground truth target image is observed during training. To address this challenging task, we propose a Cyclic-Manipulation GAN (cManiGAN) in this paper, which is able to realize where and how to edit the image regions of interest. Specifically, the image editor in cManiGAN learns to identify and complete the input image, while cross-modal interpreter and reasoner are deployed to verify the semantic correctness of the output image based on the input instruction. While the former utilizes factual/counterfactual description learning for authenticating the image semantics, the latter predicts the "undo" instruction and provides pixel-level supervision for the training of cManiGAN. With such operational cycle-consistency, our cManiGAN can be trained in the above weakly supervised setting. We conduct extensive experiments on the datasets of CLEVR and COCO, and the effectiveness and generalizability of our proposed method can be successfully verified. Project page: https://sites.google.com/view/wancyuanfan/projects/cmanigan.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Builds on 26 of its 32 references that are in the map. First posted on arXiv 26 Nov 2022.
Builds on
26 of 32 references.
References · 26
- 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
- 2014Microsoft COCO: Common Objects in Contextcited by 17,608
- 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,364
- 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
- 2022Generative Adversarial Networkscited by 5,627
- 2016Perceptual Losses for Real-Time Style Transfer and Super-Resolutioncited by 2,694
- 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,222
- 2016CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoningcited by 1,457
- 2017StarGAN: Unified Generative Adversarial Networks for Multi-Domain Image-to-Image Translationcited by 1,039
- 2021StyleCLIP: Text-Driven Manipulation of StyleGAN Imagerycited by 991
- 2016Generative Adversarial Text to Image Synthesiscited by 735
- 2016Conditional Image Synthesis With Auxiliary Classifier GANscited by 565
14 more
- 2018Image Generation from Scene Graphscited by 324
- 2018Composing Text and Image for Image Retrieval - An Empirical Odysseycited by 236
- 2019Controllable Text-to-Image Generationcited by 187
- 2019ManiGAN: Text-Guided Image Manipulationcited by 156
- 2020Semantic Image Manipulation Using Scene Graphscited by 82
- 2020StyleGAN2 Distillation for Feed-forward Image Manipulationcited by 78
- 2018Tell, Draw, and Repeat: Generating and Modifying Images Based on Continual Linguistic Instructioncited by 61
- 2019Learning Canonical Representations for Scene Graph to Image Generationcited by 60
- 2020Lightweight Generative Adversarial Networks for Text-Guided Image Manipulationcited by 33
- 2018Adversarial Scene Editing: Automatic Object Removal from Weak Supervisioncited by 27
- 2021Cycle-Consistent Inverse GAN for Text-to-Image Synthesiscited by 27
- 2020Text as Neural Operator: Image Manipulation by Text Instructioncited by 26
- 2020Open-Edit: Open-Domain Image Manipulation with Open-Vocabulary Instructionscited by 25
- 2021Towards Open-World Text-Guided Face Image Generation and Manipulationcited by 25
Built on by
Citing papers per year: 2023: 1.
1 paper.