# Tell, Draw, and Repeat: Generating and Modifying Images Based on Continual Linguistic Instruction

arXiv: https://arxiv.org/abs/1811.09845 · HTML: https://mlsanity.com/paper/1811.09845/

## Key facts

- Cited by 61 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 24 Nov 2018.

## Built on by (most cited first, 30 of 61)

- [Scaling Autoregressive Models for Content-Rich Text-to-Image Generation](https://mlsanity.com/paper/2206.10789/) (2022)
- [Vector Quantized Diffusion Model for Text-to-Image Synthesis](https://mlsanity.com/paper/2111.14822/) (2021)
- [MagicBrush: A Manually Annotated Dataset for Instruction-Guided Image Editing](https://mlsanity.com/paper/2306.10012/) (2023)
- [Training-Free Structured Diffusion Guidance for Compositional Text-to-Image Synthesis](https://mlsanity.com/paper/2212.05032/) (2022)
- [LayoutGPT: Compositional Visual Planning and Generation with Large Language Models](https://mlsanity.com/paper/2305.15393/) (2023)
- [ImgEdit: A Unified Image Editing Dataset and Benchmark](https://mlsanity.com/paper/2505.20275/) (2025)
- [Guiding Instruction-based Image Editing via Multimodal Large Language Models](https://mlsanity.com/paper/2309.17102/) (2023)
- [EnvEdit: Environment Editing for Vision-and-Language Navigation](https://mlsanity.com/paper/2203.15685/) (2022)
- [Inst-Inpaint: Instructing to Remove Objects with Diffusion Models](https://mlsanity.com/paper/2304.03246/) (2023)
- [Improving Text-to-Image Synthesis Using Contrastive Learning](https://mlsanity.com/paper/2107.02423/) (2021)
- [Improved Vector Quantized Diffusion Models](https://mlsanity.com/paper/2205.16007/) (2022)
- [ZONE: Zero-Shot Instruction-Guided Local Editing](https://mlsanity.com/paper/2312.16794/) (2023)
- [Adversarial Text-to-Image Synthesis: A Review](https://mlsanity.com/paper/2101.09983/) (2021)
- [Improving Generation and Evaluation of Visual Stories via Semantic Consistency](https://mlsanity.com/paper/2105.10026/) (2021)
- [Paint by Inpaint: Learning to Add Image Objects by Removing Them First](https://mlsanity.com/paper/2404.18212/) (2024)
- [EditWorld: Simulating World Dynamics for Instruction-Following Image Editing](https://mlsanity.com/paper/2405.14785/) (2024)
- [Language-Driven Image Style Transfer](https://mlsanity.com/paper/2106.00178/) (2021)
- [Sound-Guided Semantic Image Manipulation](https://mlsanity.com/paper/2112.00007/) (2021)
- [Tell Me What Happened: Unifying Text-guided Video Completion via Multimodal Masked Video Generation](https://mlsanity.com/paper/2211.12824/) (2022)
- [DialogGen: Multi-modal Interactive Dialogue System for Multi-turn Text-to-Image Generation](https://mlsanity.com/paper/2403.08857/) (2024)
- [Learning by Planning: Language-Guided Global Image Editing](https://mlsanity.com/paper/2106.13156/) (2021)
- [TheaterGen: Character Management with LLM for Consistent Multi-turn Image Generation](https://mlsanity.com/paper/2404.18919/) (2024)
- [Towards Language-guided Interactive 3D Generation: LLMs as Layout Interpreter with Generative Feedback](https://mlsanity.com/paper/2305.15808/) (2023)
- [ManiTrans: Entity-Level Text-Guided Image Manipulation via Token-wise Semantic Alignment and Generation](https://mlsanity.com/paper/2204.04428/) (2022)
- [CHATEDIT: Towards Multi-turn Interactive Facial Image Editing via Dialogue](https://mlsanity.com/paper/2303.11108/) (2023)
- [Learning to Follow Object-Centric Image Editing Instructions Faithfully](https://mlsanity.com/paper/2310.19145/) (2023)
- [A survey on knowledge-enhanced multimodal learning](https://mlsanity.com/paper/2211.12328/) (2022)
- [Instruction Clarification Requests in Multimodal Collaborative Dialogue Games: Tasks, and an Analysis of the CoDraw Dataset](https://mlsanity.com/paper/2302.14406/) (2023)
- [Object-Centric Learning with Slot Mixture Module](https://mlsanity.com/paper/2311.04640/) (2023)
- [Are We Done with Object-Centric Learning?](https://mlsanity.com/paper/2504.07092/) (2025)

## Abstract (arXiv)

Conditional text-to-image generation is an active area of research, with many possible applications. Existing research has primarily focused on generating a single image from available conditioning information in one step. One practical extension beyond one-step generation is a system that generates an image iteratively, conditioned on ongoing linguistic input or feedback. This is significantly more challenging than one-step generation tasks, as such a system must understand the contents of its generated images with respect to the feedback history, the current feedback, as well as the interactions among concepts present in the feedback history. In this work, we present a recurrent image generation model which takes into account both the generated output up to the current step as well as all past instructions for generation. We show that our model is able to generate the background, add new objects, and apply simple transformations to existing objects. We believe our approach is an important step toward interactive generation. Code and data is available at: https://www.microsoft.com/en-us/research/project/generative-neural-visual-artist-geneva/ .