# Remember What You have drawn: Semantic Image Manipulation with Memory

arXiv: https://arxiv.org/abs/2107.12579 · HTML: https://mlsanity.com/paper/2107.12579/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 22 references that are in the map.
- First posted on arXiv 27 Jul 2021.

## Builds on (most cited first, 18 of 18)

- [A Style-Based Generator Architecture for Generative Adversarial Networks](https://mlsanity.com/paper/1812.04948/) (2018)
- [Image-to-Image Translation with Conditional Adversarial Networks](https://mlsanity.com/paper/1611.07004/) (2016)
- [Conditional Generative Adversarial Nets](https://mlsanity.com/paper/1411.1784/) (2014)
- [InfoGAN: Interpretable Representation Learning by Information Maximizing Generative Adversarial Nets](https://mlsanity.com/paper/1606.03657/) (2016)
- [Generative Adversarial Text to Image Synthesis](https://mlsanity.com/paper/1605.05396/) (2016)
- [AttnGAN: Fine-Grained Text to Image Generation with Attentional Generative Adversarial Networks](https://mlsanity.com/paper/1711.10485/) (2017)
- [StackGAN: Text to Photo-realistic Image Synthesis with Stacked Generative Adversarial Networks](https://mlsanity.com/paper/1612.03242/) (2016)
- [DM-GAN: Dynamic Memory Generative Adversarial Networks for Text-to-Image Synthesis](https://mlsanity.com/paper/1904.01310/) (2019)
- [ManiGAN: Text-Guided Image Manipulation](https://mlsanity.com/paper/1912.06203/) (2019)
- [Learning What and Where to Draw](https://mlsanity.com/paper/1610.02454/) (2016)
- [Text-Adaptive Generative Adversarial Networks: Manipulating Images with Natural Language](https://mlsanity.com/paper/1810.11919/) (2018)
- [Semantic Image Synthesis via Adversarial Learning](https://mlsanity.com/paper/1707.06873/) (2017)
- [Semi-parametric Image Synthesis](https://mlsanity.com/paper/1804.10992/) (2018)
- [Be Your Own Prada: Fashion Synthesis with Structural Coherence](https://mlsanity.com/paper/1710.07346/) (2017)
- [RetrieveGAN: Image Synthesis via Differentiable Patch Retrieval](https://mlsanity.com/paper/2007.08513/) (2020)
- [Coloring With Limited Data: Few-Shot Colorization via Memory-Augmented Networks](https://mlsanity.com/paper/1906.11888/) (2019)
- [Texture Memory-Augmented Deep Patch-Based Image Inpainting](https://mlsanity.com/paper/2009.13240/) (2020)
- [Text2FaceGAN: Face Generation from Fine Grained Textual Descriptions](https://mlsanity.com/paper/1911.11378/) (2019)

## Built on by (most cited first, 3 of 3)

- [Learning Meta-class Memory for Few-Shot Semantic Segmentation](https://mlsanity.com/paper/2108.02958/) (2021)
- [Cross-Image Region Mining with Region Prototypical Network for Weakly Supervised Segmentation](https://mlsanity.com/paper/2108.07413/) (2021)
- [SA-LUT: Spatial Adaptive 4D Look-Up Table for Photorealistic Style Transfer](https://mlsanity.com/paper/2506.13465/) (2025)

## Abstract (arXiv)

Image manipulation with natural language, which aims to manipulate images with the guidance of language descriptions, has been a challenging problem in the fields of computer vision and natural language processing (NLP). Currently, a number of efforts have been made for this task, but their performances are still distant away from generating realistic and text-conformed manipulated images. Therefore, in this paper, we propose a memory-based Image Manipulation Network (MIM-Net), where a set of memories learned from images is introduced to synthesize the texture information with the guidance of the textual description. We propose a two-stage network with an additional reconstruction stage to learn the latent memories efficiently. To avoid the unnecessary background changes, we propose a Target Localization Unit (TLU) to focus on the manipulation of the region mentioned by the text. Moreover, to learn a robust memory, we further propose a novel randomized memory training loss. Experiments on the four popular datasets show the better performance of our method compared to the existing ones.