# Target-Free Text-guided Image Manipulation

arXiv: https://arxiv.org/abs/2211.14544 · HTML: https://mlsanity.com/paper/2211.14544/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 32 references that are in the map.
- First posted on arXiv 26 Nov 2022.

## Builds on (most cited first, 26 of 26)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [Generative Adversarial Networks](https://mlsanity.com/paper/2203.00667/) (2022)
- [Perceptual Losses for Real-Time Style Transfer and Super-Resolution](https://mlsanity.com/paper/1603.08155/) (2016)
- [ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks](https://mlsanity.com/paper/1908.02265/) (2019)
- [CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoning](https://mlsanity.com/paper/1612.06890/) (2016)
- [StarGAN: Unified Generative Adversarial Networks for Multi-Domain Image-to-Image Translation](https://mlsanity.com/paper/1711.09020/) (2017)
- [StyleCLIP: Text-Driven Manipulation of StyleGAN Imagery](https://mlsanity.com/paper/2103.17249/) (2021)
- [Generative Adversarial Text to Image Synthesis](https://mlsanity.com/paper/1605.05396/) (2016)
- [Conditional Image Synthesis With Auxiliary Classifier GANs](https://mlsanity.com/paper/1610.09585/) (2016)
- [Image Generation from Scene Graphs](https://mlsanity.com/paper/1804.01622/) (2018)
- [Composing Text and Image for Image Retrieval - An Empirical Odyssey](https://mlsanity.com/paper/1812.07119/) (2018)
- [Controllable Text-to-Image Generation](https://mlsanity.com/paper/1909.07083/) (2019)
- [ManiGAN: Text-Guided Image Manipulation](https://mlsanity.com/paper/1912.06203/) (2019)
- [Semantic Image Manipulation Using Scene Graphs](https://mlsanity.com/paper/2004.03677/) (2020)
- [StyleGAN2 Distillation for Feed-forward Image Manipulation](https://mlsanity.com/paper/2003.03581/) (2020)
- [Tell, Draw, and Repeat: Generating and Modifying Images Based on Continual Linguistic Instruction](https://mlsanity.com/paper/1811.09845/) (2018)
- [Learning Canonical Representations for Scene Graph to Image Generation](https://mlsanity.com/paper/1912.07414/) (2019)
- [Lightweight Generative Adversarial Networks for Text-Guided Image Manipulation](https://mlsanity.com/paper/2010.12136/) (2020)
- [Adversarial Scene Editing: Automatic Object Removal from Weak Supervision](https://mlsanity.com/paper/1806.01911/) (2018)
- [Cycle-Consistent Inverse GAN for Text-to-Image Synthesis](https://mlsanity.com/paper/2108.01361/) (2021)
- [Text as Neural Operator: Image Manipulation by Text Instruction](https://mlsanity.com/paper/2008.04556/) (2020)
- [Open-Edit: Open-Domain Image Manipulation with Open-Vocabulary Instructions](https://mlsanity.com/paper/2008.01576/) (2020)
- [Towards Open-World Text-Guided Face Image Generation and Manipulation](https://mlsanity.com/paper/2104.08910/) (2021)

## Built on by (most cited first, 1 of 1)

- [Inst-Inpaint: Instructing to Remove Objects with Diffusion Models](https://mlsanity.com/paper/2304.03246/) (2023)

## Abstract (arXiv)

We tackle the problem of target-free text-guided image manipulation, which requires one to modify the input reference image based on the given text instruction, while no ground truth target image is observed during training. To address this challenging task, we propose a Cyclic-Manipulation GAN (cManiGAN) in this paper, which is able to realize where and how to edit the image regions of interest. Specifically, the image editor in cManiGAN learns to identify and complete the input image, while cross-modal interpreter and reasoner are deployed to verify the semantic correctness of the output image based on the input instruction. While the former utilizes factual/counterfactual description learning for authenticating the image semantics, the latter predicts the "undo" instruction and provides pixel-level supervision for the training of cManiGAN. With such operational cycle-consistency, our cManiGAN can be trained in the above weakly supervised setting. We conduct extensive experiments on the datasets of CLEVR and COCO, and the effectiveness and generalizability of our proposed method can be successfully verified. Project page: https://sites.google.com/view/wancyuanfan/projects/cmanigan.