mlsanityData to 14 Oct 2026

PromptPaint: Steering Text-to-Image Generation Through Paint Medium-like Interactions

arXiv 2308.05184 ↗First posted 9 Aug 2023cs.HC cs.AIcited by 34

On the map: Diffusion models → Image generation, text-to-image and editing

Abstract (arXiv)

While diffusion-based text-to-image (T2I) models provide a simple and powerful way to generate images, guiding this generation remains a challenge. For concepts that are difficult to describe through language, users may struggle to create prompts. Moreover, many of these models are built as end-to-end systems, lacking support for iterative shaping of the image. In response, we introduce PromptPaint, which combines T2I generation with interactions that model how we use colored paints. PromptPaint allows users to go beyond language to mix prompts that express challenging concepts. Just as we iteratively tune colors through layered placements of paint on a physical canvas, PromptPaint similarly allows users to apply different prompts to different canvas areas and times of the generative process. Through a set of studies, we characterize different approaches for mixing prompts, design trade-offs, and socio-technical challenges for generative models. With PromptPaint we provide insight into future steerable generative tools.

Key facts. Cited by 34 papers in the map (data through 14 Oct 2026). Placed in Image generation, text-to-image and editing, one of 14 directions in Diffusion models. Builds on 55 of its 94 references that are in the map. First posted on arXiv 9 Aug 2023.

Builds on

55 of 94 references.

Closest ancestors · 14

Same direction, newest first

  1. 2023Adding Conditional Control to Text-to-Image Diffusion Models10 Feb
  2. 2023GLIGEN: Open-Set Grounded Text-to-Image Generation17 Jan
  3. 2022Diffusion Art or Digital Forgery? Investigating Data Replication in Diffusion Models7 Dec
  4. 2022InstructPix2Pix: Learning to Follow Image Editing Instructions17 Nov
  5. 2022MagicMix: Semantic Mixing with Diffusion Models28 Oct
  6. 2022DiffEdit: Diffusion-based semantic image editing with mask guidance20 Oct
  7. 2022UniTune: Text-Driven Image Editing by Fine Tuning a Diffusion Model on a Single Image17 Oct
  8. 2022Imagic: Text-Based Real Image Editing with Diffusion Models17 Oct
  9. 2022Compositional Visual Generation with Composable Diffusion Models3 Jun
  10. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding23 May
  11. 2022Hierarchical Text-Conditional Image Generation with CLIP Latents13 Apr
  12. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models20 Dec
2 more
  1. 2021Blended Diffusion for Text-driven Editing of Natural Images29 Nov
  2. 2021Palette: Image-to-Image Diffusion Models10 Nov

Other references · 41

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2020Denoising Diffusion Probabilistic Modelscited by 21,076
  4. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  5. 2020Denoising Diffusion Implicit Modelscited by 9,781
  6. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  7. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,255
  8. 2018A Style-Based Generator Architecture for Generative Adversarial Networkscited by 6,013
  9. 2022Classifier-Free Diffusion Guidancecited by 5,847
  10. 2022DreamFusion: Text-to-3D using 2D Diffusioncited by 3,025
  11. 2022Prompt-to-Prompt Image Editing with Cross Attention Controlcited by 2,431
  12. 2015Unsupervised Representation Learning with Deep Convolutional Generative Adversarial Networkscited by 2,054
29 more
  1. 2021SDEdit: Guided Image Synthesis and Editing with Stochastic Differential Equationscited by 1,937
  2. 2022Make-A-Video: Text-to-Video Generation without Text-Video Datacited by 1,742
  3. 2017StarGAN: Unified Generative Adversarial Networks for Multi-Domain Image-to-Image Translationcited by 1,039
  4. 2021Resolution-robust Large Mask Inpainting with Fourier Convolutionscited by 738
  5. 2021Zero-Shot Text-Guided Object Generation with Dream Fieldscited by 513
  6. 2023Zero-shot Image-to-Image Translationcited by 505
  7. 2022Phenaki: Variable Length Video Generation From Open Domain Textual Descriptioncited by 484
  8. 2015A Neural Algorithm of Artistic Stylecited by 483
  9. 2023MusicLM: Generating Music From Textcited by 448
  10. 2020GeDi: Generative Discriminator Guided Sequence Generationcited by 378
  11. 2016A Learned Representation For Artistic Stylecited by 341
  12. 2023Composer: Creative and Controllable Image Synthesis with Composable Conditionscited by 304
  13. 2022VQGAN-CLIP: Open Domain Image Generation and Editing with Natural Language Guidancecited by 285
  14. 2022Cold Diffusion: Inverting Arbitrary Image Transforms Without Noisecited by 276
  15. 2021AI Chains: Transparent and Controllable Human-AI Interaction by Chaining Large Language Model Promptscited by 268
  16. 2021Design Guidelines for Prompt Engineering Text-to-Image Generative Modelscited by 218
  17. 2020Swapping Autoencoder for Deep Image Manipulationcited by 215
  18. 2021Wav2CLIP: Learning Robust Audio Representations From CLIPcited by 213
  19. 2021HyperStyle: StyleGAN Inversion with HyperNetworks for Real Image Editingcited by 208
  20. 2021CLIPstyler: Image Style Transfer with a Single Text Conditioncited by 203
  21. 2018Arbitrary Style Transfer with Style-Attentional Networkscited by 198
  22. 2018Avatar-Net: Multi-scale Zero-shot Style Transfer by Feature Decorationcited by 115
  23. 2022Interactive and Visual Prompt Engineering for Ad-hoc Task Adaptation with Large Language Modelscited by 70
  24. 2023RePrompt: Automatic Prompt Editing to Refine AI-Generative Art Towards Precise Expressionscited by 43
  25. 2022Opal: Multimodal Image Generation for News Illustrationcited by 36
  26. 2022User-Controllable Latent Transformer for StyleGAN Image Layout Editingcited by 28
  27. 2020Sequential Gallery for Interactive Visual Design Optimizationcited by 25
  28. 2021Toward a Visual Concept Vocabulary for GAN Latent Spacecited by 15
  29. 2022GANSlider: How Users Control Generative Models for Images using Multiple Sliders with and without Feedforward Informationcited by 13

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 1; 2024: 13; 2025: 12; 2026: 8.

34 papers.

Sort:

By area:

  1. 2024Homogenization Effects of Large Language Models on Human Creative Ideationcited by 127
  2. 2024PromptCharm: Text-to-Image Generation through Multi-modal Prompting and Refinementcited by 62
  3. 2024Patchview: LLM-Powered Worldbuilding with Generative Dust and Magnet Visualizationcited by 23
  4. 2023Interactive AI Alignment: Specification, Process, and Evaluation Alignmentcited by 22
  5. 2024Diffusion-Based Visual Art Creation: A Survey and New Perspectivescited by 17
  6. 2024TypeDance: Creating Semantic Typographic Logos from Image through Personalized Generationcited by 10
  7. 2024VRCopilot: Authoring 3D Layouts with Generative AI Models in VRcited by 10
  8. 2024Human-Guided Image Generation for Expanding Small-Scale Training Image Datasetscited by 10
  9. 2025AI-Instruments: Embodying Prompts as Instruments to Abstract & Reflect Graphical Interface Commands as General-Purpose Toolscited by 8
  10. 2024ModalChorus: Visual Probing and Alignment of Multi-modal Embeddings via Modal Fusion Mapcited by 7
  11. 2024Exploring Empty Spaces: Human-in-the-Loop Data Augmentationcited by 6
  12. 2025SketchFlex: Facilitating Spatial-Semantic Coherence in Text-to-Image Generation with Region-Based Sketchescited by 6
22 more
  1. 2025DesignWeaver: Dimensional Scaffolding for Text-to-Image Product Designcited by 6
  2. 2025Intent Tagging: Exploring Micro-Prompting Interactions for Supporting Granular Human-GenAI Co-Creation Workflowscited by 6
  3. 2025GenTune: Toward Traceable Prompts to Improve Controllability of Image Refinement in Environment Designcited by 5
  4. 2024Survey of User Interface Design and Interaction Techniques in Generative AI Applicationscited by 4
  5. 2024BlendScape: Enabling End-User Customization of Video-Conferencing Environments through Generative AIcited by 3
  6. 2024Seizing the Means of Production: Exploring the Landscape of Crafting, Adapting and Navigating Generative AI Models in the Visual Artscited by 2
  7. 2024CUPID: Contextual Understanding of Prompt-conditioned Image Distributionscited by 2
  8. 2025ACAI for SBOs: AI Co-creation for Advertising and Inspiration for Small Business Ownerscited by 2
  9. 2025Scaffolding Recursive Divergence and Convergence in Story Ideationcited by 2
  10. 2025StoryEnsemble: Enabling Dynamic Exploration & Iteration in the Design Process with AI and Forward-Backward Propagationcited by 2
  11. 2026Say It My Way: Exploring Control in Conversational Visual Question Answering with Blind Userscited by 2
  12. 2026Texterial: A Text-as-Material Interaction Paradigm for LLM-Mediated Writingcited by 2
  13. 2025Steerable Chatbots: Exploring Personalization Control Interfaces via LLM Activation Steeringcited by 1
  14. 2025AnimAgents: Coordinating Multi-Stage Animation Pre-Production with Human-Multi-Agent Collaborationcited by 1
  15. 2025ORIBA: Exploring LLM-Driven Role-Play Chatbot as a Creativity Support Tool for Original Character Artistscited by 1
  16. 2025FairT2I: Mitigating Social Bias in Text-to-Image Generation via Large Language Model-Assisted Detection and Attribute Rebalancingcited by 0
  17. 2026Adaptive Prompt Elicitation for Text-to-Image Generationcited by 0
  18. 2026Multi-Object Advertisement Creative Generationcited by 0
  19. 2026Human-AI Coordination Zones: A Framework for Designing Human-in-the-Loop Experiences with Agentic AIcited by 0
  20. 2026Compos3D: Interactive Part-Based Composition for Creative Control in Generative 3D Modelscited by 0
  21. 2026SemanticSlider3D: Training-Free Continuous Semantic Editing for 3D Objectscited by 0
  22. 2026InstructMesh: Selective Refinement of Generative 3D Models for Fabricationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Manipulating Embeddings of Stable Diffusion Prompts
  2. 2022Best Prompts for Text-to-Image Models and How to Find Them
  3. 2024No Longer Trending on Artstation: Prompt Analysis of Generative AI Art
  4. 2024Draw Like an Artist: Complex Scene Generation with Diffusion Model via Composition, Painting, and Retouching
  5. 2025Blending Concepts with Text-to-Image Diffusion Models
  6. 2023DreamDistribution: Learning Prompt Distribution for Diverse In-distribution Generation
  7. 2026From Prompting to Composing: A Spatial Canvas Interface for Poster Generation
  8. 2024I Dream My Painting: Connecting MLLMs and Diffusion Models via Prompt Generation for Text-Guided Multi-Mask Inpainting