mlsanityData to 14 Oct 2026

Tell, Draw, and Repeat: Generating and Modifying Images Based on Continual Linguistic Instruction

arXiv 1811.09845 ↗First posted 24 Nov 2018cs.CVcited by 61

Not on the map. Why?

Abstract (arXiv)

Conditional text-to-image generation is an active area of research, with many possible applications. Existing research has primarily focused on generating a single image from available conditioning information in one step. One practical extension beyond one-step generation is a system that generates an image iteratively, conditioned on ongoing linguistic input or feedback. This is significantly more challenging than one-step generation tasks, as such a system must understand the contents of its generated images with respect to the feedback history, the current feedback, as well as the interactions among concepts present in the feedback history. In this work, we present a recurrent image generation model which takes into account both the generated output up to the current step as well as all past instructions for generation. We show that our model is able to generate the background, add new objects, and apply simple transformations to existing objects. We believe our approach is an important step toward interactive generation. Code and data is available at: https://www.microsoft.com/en-us/research/project/generative-neural-visual-artist-geneva/ .

Key facts. Cited by 61 papers in the map (data through 14 Oct 2026). First posted on arXiv 24 Nov 2018.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 11; 2022: 13; 2023: 21; 2024: 13; 2025: 2; 2026: 1.

61 papers.

Sort:

By area:

  1. 2022Scaling Autoregressive Models for Content-Rich Text-to-Image Generationcited by 1,295
  2. 2021Vector Quantized Diffusion Model for Text-to-Image Synthesiscited by 774
  3. 2023MagicBrush: A Manually Annotated Dataset for Instruction-Guided Image Editingcited by 624
  4. 2022Training-Free Structured Diffusion Guidance for Compositional Text-to-Image Synthesiscited by 388
  5. 2023LayoutGPT: Compositional Visual Planning and Generation with Large Language Modelscited by 347
  6. 2025ImgEdit: A Unified Image Editing Dataset and Benchmarkcited by 231
  7. 2023Guiding Instruction-based Image Editing via Multimodal Large Language Modelscited by 185
  8. 2022EnvEdit: Environment Editing for Vision-and-Language Navigationcited by 83
  9. 2023Inst-Inpaint: Instructing to Remove Objects with Diffusion Modelscited by 66
  10. 2021Improving Text-to-Image Synthesis Using Contrastive Learningcited by 62
  11. 2022Improved Vector Quantized Diffusion Modelscited by 62
  12. 2023ZONE: Zero-Shot Instruction-Guided Local Editingcited by 62
38 more
  1. 2021Adversarial Text-to-Image Synthesis: A Reviewcited by 52
  2. 2021Improving Generation and Evaluation of Visual Stories via Semantic Consistencycited by 43
  3. 2024Paint by Inpaint: Learning to Add Image Objects by Removing Them Firstcited by 39
  4. 2024EditWorld: Simulating World Dynamics for Instruction-Following Image Editingcited by 38
  5. 2021Language-Driven Image Style Transfercited by 37
  6. 2021Sound-Guided Semantic Image Manipulationcited by 35
  7. 2022Tell Me What Happened: Unifying Text-guided Video Completion via Multimodal Masked Video Generationcited by 32
  8. 2024DialogGen: Multi-modal Interactive Dialogue System for Multi-turn Text-to-Image Generationcited by 26
  9. 2021Learning by Planning: Language-Guided Global Image Editingcited by 25
  10. 2024TheaterGen: Character Management with LLM for Consistent Multi-turn Image Generationcited by 19
  11. 2023Towards Language-guided Interactive 3D Generation: LLMs as Layout Interpreter with Generative Feedbackcited by 17
  12. 2022ManiTrans: Entity-Level Text-Guided Image Manipulation via Token-wise Semantic Alignment and Generationcited by 13
  13. 2023CHATEDIT: Towards Multi-turn Interactive Facial Image Editing via Dialoguecited by 12
  14. 2023Learning to Follow Object-Centric Image Editing Instructions Faithfullycited by 12
  15. 2022A survey on knowledge-enhanced multimodal learningcited by 11
  16. 2023Instruction Clarification Requests in Multimodal Collaborative Dialogue Games: Tasks, and an Analysis of the CoDraw Datasetcited by 11
  17. 2023Object-Centric Learning with Slot Mixture Modulecited by 11
  18. 2025Are We Done with Object-Centric Learning?cited by 11
  19. 2021M3L: Language-based Video Editing via Multi-Modal Multi-Level Transformerscited by 8
  20. 2021Unified Questioner Transformer for Descriptive Question Generation in Goal-Oriented Visual Dialoguecited by 8
  21. 2023Vision + Language Applications: A Surveycited by 8
  22. 2022Robust Sound-Guided Image Manipulationcited by 7
  23. 2024InstructRL4Pix: Training Diffusion for Image Editing by Reinforcement Learningcited by 7
  24. 2023Fine-grained Cross-modal Fusion based Refinement for Text-to-Image Synthesiscited by 6
  25. 2024Repositioning the Subject within Imagecited by 6
  26. 2021LatteGAN: Visually Guided Language Attention for Multi-Turn Text-Conditioned Image Manipulationcited by 5
  27. 2024Instilling Multi-round Thinking to Text-guided Image Generationcited by 5
  28. 2024Text-to-Image Cross-Modal Generation: A Systematic Reviewcited by 5
  29. 2024A Survey of Multimodal Composite Editing and Retrievalcited by 5
  30. 2023A Diffusion-based Method for Multi-turn Compositional Image Generationcited by 4
  31. 2023Text-guided 3D Human Generation from 2D Collectionscited by 4
  32. 2024GraPE: A Generate-Plan-Edit Framework for Compositional T2I Synthesiscited by 4
  33. 2022CAISE: Conversational Agent for Image Search and Editingcited by 3
  34. 2022IR-GAN: Image Manipulation with Linguistic Instruction by Increment Reasoningcited by 3
  35. 2023Entity-Level Text-Guided Image Manipulationcited by 3
  36. 2023Video Generation Beyond a Single Clipcited by 3
  37. 2024BlendScape: Enabling End-User Customization of Video-Conferencing Environments through Generative AIcited by 3
  38. 2021SpaceEdit: Learning a Unified Editing Space for Open-Domain Image Editingcited by 2

50 most cited of 61 All 61 (JSON)