mlsanityData to 7 Oct 2026

Paint by Word

arXiv 2103.10951 ↗First posted 19 Mar 2021cs.CV cs.AI cs.GRcited by 107

Not on the map. Why?

Cited by 38 in Diffusion models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

We investigate the problem of zero-shot semantic image painting. Instead of painting modifications into an image using only concrete colors or a finite set of semantic concepts, we ask how to create semantic paint based on open full-text descriptions: our goal is to be able to point to a location in a synthesized image and apply an arbitrary new concept such as "rustic" or "opulent" or "happy dog." To do this, our method combines a state-of-the art generative model of realistic images with a state-of-the-art text-image semantic similarity network. We find that, to make large changes, it is important to use non-gradient methods to explore latent space, and it is important to relax the computations of the GAN to target changes to a specific region. We conduct user studies to compare our methods to several baselines.

Key facts. Cited by 107 papers in the map (data through 7 Oct 2026). Builds on 44 of its 60 references that are in the map. First posted on arXiv 19 Mar 2021.

Builds on

44 of 60 references.

References · 44

  1. 2017Attention Is All You Needcited by 59,932
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,029
  6. 2013Intriguing properties of neural networkscited by 5,114
  7. 2016Image-to-Image Translation with Conditional Adversarial Networkscited by 4,510
  8. 2021Zero-Shot Text-to-Image Generationcited by 4,276
  9. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  10. 2019Analyzing and Improving the Image Quality of StyleGANcited by 3,580
  11. 2018Large Scale GAN Training for High Fidelity Natural Image Synthesiscited by 2,609
  12. 2015Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networkscited by 2,048
32 more
  1. 2019Generating Diverse High-Fidelity Images with VQ-VAE-2cited by 1,376
  2. 2015LSUN: Construction of a Large-scale Image Dataset using Deep Learning with Humans in the Loopcited by 1,350
  3. 2019Semantic Image Synthesis with Spatially-Adaptive Normalizationcited by 1,295
  4. 2020Training Generative Adversarial Networks with Limited Datacited by 1,184
  5. 2017High-Resolution Image Synthesis and Semantic Manipulation with Conditional GANscited by 1,176
  6. 2016Generative Adversarial Text to Image Synthesiscited by 735
  7. 2020Encoding in Style: a StyleGAN Encoder for Image-to-Image Translationcited by 727
  8. 2019Interpreting the Latent Space of GANs for Semantic Face Editingcited by 691
  9. 2019Image2StyleGAN: How to Embed Images Into the StyleGAN Latent Space?cited by 675
  10. 2017AttnGAN: Fine-Grained Text to Image Generation with Attentional Generative Adversarial Networkscited by 664
  11. 2020GANSpace: Discovering Interpretable GAN Controlscited by 612
  12. 2016StackGAN: Text to Photo-realistic Image Synthesis with Stacked Generative Adversarial Networkscited by 575
  13. 2020In-Domain GAN Inversion for Real Image Editingcited by 420
  14. 2016The CMA Evolution Strategy: A Tutorialcited by 364
  15. 2019Image2StyleGAN++: How to Edit the Embedded Images?cited by 356
  16. 2016Generative Visual Manipulation on the Natural Image Manifoldcited by 348
  17. 2020StyleSpace Analysis: Disentangled Controls for StyleGAN Image Generationcited by 331
  18. 2018Image Generation from Scene Graphscited by 324
  19. 2019DM-GAN: Dynamic Memory Generative Adversarial Networks for Text-to-Image Synthesiscited by 308
  20. 2017StackGAN++: Realistic Image Synthesis with Stacked Generative Adversarial Networkscited by 229
  21. 2019GANalyze: Toward Visual Definitions of Cognitive Image Propertiescited by 193
  22. 2015DRAW: A Recurrent Neural Network For Image Generationcited by 190
  23. 2020Semantic Photo Manipulation with a Generative Image Priorcited by 178
  24. 2019MirrorGAN: Learning Text-to-image Generation by Redescriptioncited by 176
  25. 2015Generating Images from Captions with Attentioncited by 135
  26. 2019Object-driven Text-to-Image Synthesis via Adversarial Trainingcited by 117
  27. 2020Improved StyleGAN Embedding: Where are the Good Latents?cited by 88
  28. 2020Transforming and Projecting Images into Class-conditional Generative Networkscited by 76
  29. 2019Image Counterfactual Sensitivity Analysis for Detecting Unintended Biascited by 74
  30. 2020Collaborative Learning for Faster StyleGAN Embeddingcited by 65
  31. 2018Photographic Text-to-Image Synthesis with a Hierarchically-nested Adversarial Networkcited by 64
  32. 2019Semantics Disentangling for Text-to-Image Generationcited by 61

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 18; 2022: 32; 2023: 28; 2024: 20; 2025: 7; 2026: 2.

107 papers.

Sort:

By area:

  1. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Modelscited by 3,595
  2. 2022DreamBooth: Fine Tuning Text-to-Image Diffusion Models for Subject-Driven Generationcited by 3,286
  3. 2022Prompt-to-Prompt Image Editing with Cross Attention Controlcited by 2,409
  4. 2021Alias-Free Generative Adversarial Networkscited by 1,181
  5. 2021StyleCLIP: Text-Driven Manipulation of StyleGAN Imagerycited by 990
  6. 2021Blended Diffusion for Text-driven Editing of Natural Imagescited by 924
  7. 2022eDiff-I: Text-to-Image Diffusion Models with an Ensemble of Expert Denoiserscited by 921
  8. 2022Plug-and-Play Diffusion Features for Text-Driven Image-to-Image Translationcited by 897
  9. 2023Attend-and-Excite: Attention-Based Semantic Guidance for Text-to-Image Diffusion Modelscited by 696
  10. 2022DiffEdit: Diffusion-based semantic image editing with mask guidancecited by 643
  11. 2022Compositional Visual Generation with Composable Diffusion Modelscited by 619
  12. 2023MagicBrush: A Manually Annotated Dataset for Instruction-Guided Image Editingcited by 615
38 more
  1. 2021ClipCap: CLIP Prefix for Image Captioningcited by 557
  2. 2023Zero-shot Image-to-Image Translationcited by 503
  3. 2022Paint by Example: Exemplar-based Image Editing with Diffusion Modelscited by 489
  4. 2022Blended Latent Diffusioncited by 430
  5. 2023FastComposer: Tuning-Free Multi-Subject Image Generation with Localized Attentioncited by 313
  6. 2022Text2LIVE: Text-Driven Layered Image and Video Editingcited by 312
  7. 2022SmartBrush: Text and Shape Guided Object Inpainting with Diffusion Modelcited by 276
  8. 2021N"UWA: Visual Synthesis Pre-training for Neural visUal World creAtioncited by 253
  9. 2023Break-A-Scene: Extracting Multiple Concepts from a Single Imagecited by 230
  10. 2022Imagen Editor and EditBench: Advancing and Evaluating Text-Guided Image Inpaintingcited by 229
  11. 2022SpaText: Spatio-Textual Representation for Controllable Image Generationcited by 217
  12. 2023Key-Locked Rank One Editing for Text-to-Image Personalizationcited by 194
  13. 2024InstanceDiffusion: Instance-level Control for Image Generationcited by 175
  14. 2021StyleGAN-NADA: CLIP-Guided Domain Adaptation of Image Generatorscited by 164
  15. 2021EditGAN: High-Precision Semantic Image Editingcited by 156
  16. 2024Stable Flow: Vital Layers for Training-Free Image Editingcited by 111
  17. 2021CLOOB: Modern Hopfield Networks with InfoLOOB Outperform CLIPcited by 88
  18. 2022Touch and Go: Learning from Human-Collected Vision and Touchcited by 84
  19. 2022EnvEdit: Environment Editing for Vision-and-Language Navigationcited by 83
  20. 2023ProSpect: Prompt Spectrum for Attribute-Aware Personalization of Diffusion Modelscited by 75
  21. 2023Orthogonal Adaptation for Modular Customization of Diffusion Modelscited by 70
  22. 2022ClipFace: Text-guided Editing of Textured 3D Morphable Modelscited by 59
  23. 2022Disentangling visual and written concepts in CLIPcited by 57
  24. 2021Multimodal Image Synthesis and Editing: The Generative AI Eracited by 52
  25. 2024CoCoCo: Improving Text-Guided Video Inpainting for Better Consistency, Controllability and Compatibilitycited by 49
  26. 2023DiffUTE: Universal Text Editing Diffusion Modelcited by 44
  27. 2022UniTune: Text-Driven Image Editing by Fine Tuning a Diffusion Model on a Single Imagecited by 42
  28. 2023Brush Your Text: Synthesize Any Scene Text on Images via Diffusion Modelcited by 41
  29. 2021StyleMC: Multi-Channel Based Fast Text-Guided Image Generation and Manipulationcited by 40
  30. 2024Magic Fixup: Streamlining Photo Editing by Watching Dynamic Videoscited by 38
  31. 2022Lighting (In)consistency of Paint by Textcited by 35
  32. 2022State-of-the-Art in the Architecture, Methods and Applications of StyleGANcited by 34
  33. 2022Perspective (In)consistency of Paint by Textcited by 34
  34. 2023GlyphDraw: Seamlessly Rendering Text with Intricate Spatial Structures in Text-to-Image Generationcited by 34
  35. 2023ReFACT: Updating Text-to-Image Models by Editing the Text Encodercited by 32
  36. 2023Text-Driven Image Editing via Learnable Regionscited by 31
  37. 2021An Image is Worth More Than a Thousand Words: Towards Disentanglement in the Wildcited by 30
  38. 2021Towards Open-World Text-Guided Face Image Generation and Manipulationcited by 25

50 most cited of 107 All 107 (JSON)