# Text as Neural Operator: Image Manipulation by Text Instruction

arXiv: https://arxiv.org/abs/2008.04556 · HTML: https://mlsanity.com/paper/2008.04556/

## Key facts

- Cited by 26 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 11 Aug 2020.

## Built on by (most cited first, 26 of 26)

- [Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Models](https://mlsanity.com/paper/2303.04671/) (2023)
- [MagicBrush: A Manually Annotated Dataset for Instruction-Guided Image Editing](https://mlsanity.com/paper/2306.10012/) (2023)
- [Talk-to-Edit: Fine-Grained Facial Editing via Dialog](https://mlsanity.com/paper/2109.04425/) (2021)
- [Inst-Inpaint: Instructing to Remove Objects with Diffusion Models](https://mlsanity.com/paper/2304.03246/) (2023)
- [ZONE: Zero-Shot Instruction-Guided Local Editing](https://mlsanity.com/paper/2312.16794/) (2023)
- [Paint by Inpaint: Learning to Add Image Objects by Removing Them First](https://mlsanity.com/paper/2404.18212/) (2024)
- [EditWorld: Simulating World Dynamics for Instruction-Following Image Editing](https://mlsanity.com/paper/2405.14785/) (2024)
- [Text-Driven Image Editing via Learnable Regions](https://mlsanity.com/paper/2311.16432/) (2023)
- [Towards Language-guided Visual Recognition via Dynamic Convolutions](https://mlsanity.com/paper/2110.08797/) (2021)
- [Eliminating Contextual Prior Bias for Semantic Image Editing via Dual-Cycle Diffusion](https://mlsanity.com/paper/2302.02394/) (2023)
- [ManiTrans: Entity-Level Text-Guided Image Manipulation via Token-wise Semantic Alignment and Generation](https://mlsanity.com/paper/2204.04428/) (2022)
- [PPMN: Pixel-Phrase Matching Network for One-Stage Panoptic Narrative Grounding](https://mlsanity.com/paper/2208.05647/) (2022)
- [Instruction Clarification Requests in Multimodal Collaborative Dialogue Games: Tasks, and an Analysis of the CoDraw Dataset](https://mlsanity.com/paper/2302.14406/) (2023)
- [Hadamard product in deep learning: Introduction, Advances and Challenges](https://mlsanity.com/paper/2504.13112/) (2025)
- [Unified Questioner Transformer for Descriptive Question Generation in Goal-Oriented Visual Dialogue](https://mlsanity.com/paper/2106.15550/) (2021)
- [Repositioning the Subject within Image](https://mlsanity.com/paper/2401.16861/) (2024)
- [LatteGAN: Visually Guided Language Attention for Multi-Turn Text-Conditioned Image Manipulation](https://mlsanity.com/paper/2112.13985/) (2021)
- [Instilling Multi-round Thinking to Text-guided Image Generation](https://mlsanity.com/paper/2401.08472/) (2024)
- [A Survey of Multimodal Composite Editing and Retrieval](https://mlsanity.com/paper/2409.05405/) (2024)
- [CCGG: A Deep Autoregressive Model for Class-Conditional Graph Generation](https://mlsanity.com/paper/2110.03800/) (2021)
- [A Diffusion-based Method for Multi-turn Compositional Image Generation](https://mlsanity.com/paper/2304.02192/) (2023)
- [GraPE: A Generate-Plan-Edit Framework for Compositional T2I Synthesis](https://mlsanity.com/paper/2412.06089/) (2024)
- [Entity-Level Text-Guided Image Manipulation](https://mlsanity.com/paper/2302.11383/) (2023)
- [Image Manipulation via Multi-Hop Instructions -- A New Dataset and Weakly-Supervised Neuro-Symbolic Approach](https://mlsanity.com/paper/2305.14410/) (2023)
- [Target-Free Text-guided Image Manipulation](https://mlsanity.com/paper/2211.14544/) (2022)
- [Diffusion idea exploration for art generation](https://mlsanity.com/paper/2307.04978/) (2023)

## Abstract (arXiv)

In recent years, text-guided image manipulation has gained increasing attention in the multimedia and computer vision community. The input to conditional image generation has evolved from image-only to multimodality. In this paper, we study a setting that allows users to edit an image with multiple objects using complex text instructions to add, remove, or change the objects. The inputs of the task are multimodal including (1) a reference image and (2) an instruction in natural language that describes desired modifications to the image. We propose a GAN-based method to tackle this problem. The key idea is to treat text as neural operators to locally modify the image feature. We show that the proposed model performs favorably against recent strong baselines on three public datasets. Specifically, it generates images of greater fidelity and semantic relevance, and when used as a image query, leads to better retrieval performance.