mlsanityData to 7 Oct 2026

Text as Neural Operator: Image Manipulation by Text Instruction

arXiv 2008.04556 ↗First posted 11 Aug 2020cs.CVcited by 26

Not on the map. Why?

Abstract (arXiv)

In recent years, text-guided image manipulation has gained increasing attention in the multimedia and computer vision community. The input to conditional image generation has evolved from image-only to multimodality. In this paper, we study a setting that allows users to edit an image with multiple objects using complex text instructions to add, remove, or change the objects. The inputs of the task are multimodal including (1) a reference image and (2) an instruction in natural language that describes desired modifications to the image. We propose a GAN-based method to tackle this problem. The key idea is to treat text as neural operators to locally modify the image feature. We show that the proposed model performs favorably against recent strong baselines on three public datasets. Specifically, it generates images of greater fidelity and semantic relevance, and when used as a image query, leads to better retrieval performance.

Key facts. Cited by 26 papers in the map (data through 7 Oct 2026). First posted on arXiv 11 Aug 2020.

Builds on

No reference list yet.

Built on by

Cited by year

Citing papers per year: 2021: 5; 2022: 3; 2023: 11; 2024: 6; 2025: 1.

26 papers.

Sort:

By area:

  1. 2023Visual ChatGPT: Talking, Drawing and Editing with Visual Foundation Modelscited by 659
  2. 2023MagicBrush: A Manually Annotated Dataset for Instruction-Guided Image Editingcited by 615
  3. 2021Talk-to-Edit: Fine-Grained Facial Editing via Dialogcited by 102
  4. 2023Inst-Inpaint: Instructing to Remove Objects with Diffusion Modelscited by 66
  5. 2023ZONE: Zero-Shot Instruction-Guided Local Editingcited by 61
  6. 2024Paint by Inpaint: Learning to Add Image Objects by Removing Them Firstcited by 39
  7. 2024EditWorld: Simulating World Dynamics for Instruction-Following Image Editingcited by 37
  8. 2023Text-Driven Image Editing via Learnable Regionscited by 31
  9. 2021Towards Language-guided Visual Recognition via Dynamic Convolutionscited by 17
  10. 2023Eliminating Contextual Prior Bias for Semantic Image Editing via Dual-Cycle Diffusioncited by 14
  11. 2022ManiTrans: Entity-Level Text-Guided Image Manipulation via Token-wise Semantic Alignment and Generationcited by 13
  12. 2022PPMN: Pixel-Phrase Matching Network for One-Stage Panoptic Narrative Groundingcited by 12
14 more
  1. 2023Instruction Clarification Requests in Multimodal Collaborative Dialogue Games: Tasks, and an Analysis of the CoDraw Datasetcited by 11
  2. 2025Hadamard product in deep learning: Introduction, Advances and Challengescited by 9
  3. 2021Unified Questioner Transformer for Descriptive Question Generation in Goal-Oriented Visual Dialoguecited by 8
  4. 2024Repositioning the Subject within Imagecited by 6
  5. 2021LatteGAN: Visually Guided Language Attention for Multi-Turn Text-Conditioned Image Manipulationcited by 5
  6. 2024Instilling Multi-round Thinking to Text-guided Image Generationcited by 5
  7. 2024A Survey of Multimodal Composite Editing and Retrievalcited by 5
  8. 2021CCGG: A Deep Autoregressive Model for Class-Conditional Graph Generationcited by 4
  9. 2023A Diffusion-based Method for Multi-turn Compositional Image Generationcited by 4
  10. 2024GraPE: A Generate-Plan-Edit Framework for Compositional T2I Synthesiscited by 4
  11. 2023Entity-Level Text-Guided Image Manipulationcited by 3
  12. 2023Image Manipulation via Multi-Hop Instructions -- A New Dataset and Weakly-Supervised Neuro-Symbolic Approachcited by 2
  13. 2022Target-Free Text-guided Image Manipulationcited by 1
  14. 2023Diffusion idea exploration for art generationcited by 1