mlsanityData to 7 Oct 2026

Investigating Prompt Engineering in Diffusion Models

arXiv 2211.15462 ↗First posted 21 Nov 2022cs.CV cs.AI cs.CLcited by 58

On the map: Diffusion models → Image generation, text-to-image and editing

Abstract (arXiv)

With the spread of the use of Text2Img diffusion models such as DALL-E 2, Imagen, Mid Journey and Stable Diffusion, one challenge that artists face is selecting the right prompts to achieve the desired artistic output. We present techniques for measuring the effect that specific words and phrases in prompts have, and (in the Appendix) present guidance on the selection of prompts to produce desired effects.

Key facts. Cited by 58 papers in the map (data through 7 Oct 2026). Placed in Image generation, text-to-image and editing, one of 14 directions in Diffusion models. Builds on 11 of its 13 references that are in the map. First posted on arXiv 21 Nov 2022.

Builds on

11 of 13 references.

Closest ancestors · 3

Same direction, newest first

  1. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding23 May
  2. 2022Hierarchical Text-Conditional Image Generation with CLIP Latents13 Apr
  3. 2021GLIDE: Towards Photorealistic Image Generation and Editing with Text-Guided Diffusion Models20 Dec

Other references · 8

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  3. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  4. 2018The Unreasonable Effectiveness of Deep Features as a Perceptual Metriccited by 11,029
  5. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  6. 2022LAION-5B: An open large-scale dataset for training next generation image-text modelscited by 4,131
  7. 2016Perceptual Losses for Real-Time Style Transfer and Super-Resolutioncited by 2,678
  8. 2020A Loss Function for Generative Neural Networks Based on Watson's Perceptual Modelcited by 39

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 21; 2024: 22; 2025: 12; 2026: 3.

58 papers.

Sort:

By area:

  1. 2023IP-Adapter: Text Compatible Image Prompt Adapter for Text-to-Image Diffusion Modelscited by 1,439
  2. 2023Attend-and-Excite: Attention-Based Semantic Guidance for Text-to-Image Diffusion Modelscited by 696
  3. 2023TF-ICON: Diffusion-Based Training-Free Cross-Domain Image Compositioncited by 182
  4. 2023DiffuMask: Synthesizing Images with Pixel-level Annotations for Semantic Segmentation Using Diffusion Modelscited by 162
  5. 2023In-Context Impersonation Reveals Large Language Models' Strengths and Biasescited by 154
  6. 2024VidProM: A Million-scale Real Prompt-Gallery Dataset for Text-to-Video Diffusion Modelscited by 124
  7. 2023A-STAR: Test-time Attention Segregation and Retention for Text-to-image Synthesiscited by 78
  8. 2023Free-Bloom: Zero-Shot Text-to-Video Generator with LLM Director and LDM Animatorcited by 74
  9. 2023I2V-Adapter: A General Image-to-Video Adapter for Diffusion Modelscited by 70
  10. 2023Promptify: Text-to-Image Generation through Interactive Prompt Exploration with Large Language Modelscited by 67
  11. 2024MJ-Bench: Is Your Multimodal Reward Model Really a Good Judge for Text-to-Image Generation?cited by 64
  12. 2023Prompt-Free Diffusion: Taking "Text" out of Text-to-Image Diffusion Modelscited by 62
38 more
  1. 2023CONFORM: Contrast is All You Need For High-Fidelity Text-to-Image Diffusion Modelscited by 54
  2. 2023Enhancing Diffusion Models with Text-Encoder Reinforcement Learningcited by 30
  3. 2024On Discrete Prompt Optimization for Diffusion Modelscited by 28
  4. 2024TextCraftor: Your Text Encoder Can be Image Quality Controllercited by 27
  5. 2023Zero-shot Generation of Coherent Storybook from Plain Text Story using Diffusion Modelscited by 26
  6. 2023Visual Instruction Inversion: Image Editing via Visual Promptingcited by 26
  7. 2025RelationAdapter: Learning and Transferring Visual Relation with Diffusion Transformerscited by 26
  8. 2024PrimeComposer: Faster Progressively Combined Diffusion for Image Composition with Attention Steeringcited by 20
  9. 2024IPAdapter-Instruct: Resolving Ambiguity in Image-based Conditioning using Instruct Promptscited by 18
  10. 2024Towards Reliable Advertising Image Generation Using Human Feedbackcited by 15
  11. 2023A Systematic Survey of Prompt Engineering on Vision-Language Foundation Modelscited by 12
  12. 2023DiffSynth: Latent In-Iteration Deflickering for Realistic Video Synthesiscited by 11
  13. 2024Prompt Refinement with Image Pivot for Text-to-Image Generationcited by 11
  14. 2024Enhancing MMDiT-Based Text-to-Image Models for Similar Subject Generationcited by 11
  15. 2024ReNeg: Learning Negative Embedding with Reward Guidancecited by 9
  16. 2023Manipulating Embeddings of Stable Diffusion Promptscited by 8
  17. 2023Cut-and-Paste: Subject-Driven Video Editing with Attention Controlcited by 8
  18. 2024EBDM: Exemplar-guided Image Translation with Brownian-bridge Diffusion Modelscited by 8
  19. 2025Free-Lunch Color-Texture Disentanglement for Stylized Image Generationcited by 8
  20. 2024CoCoNO: Attention Contrast-and-Complete for Initial Noise Optimization in Text-to-Image Synthesiscited by 7
  21. 2023The Multimodal And Modular Ai Chef: Complex Recipe Generation From Imagerycited by 5
  22. 2024Capability-aware Prompt Reformulation Learning for Text-to-Image Generationcited by 5
  23. 2024FRAP: Faithful and Realistic Text-to-Image Generation with Adaptive Prompt Weightingcited by 5
  24. 2024SCAPE: Searching Conceptual Architecture Prompts using Evolutioncited by 4
  25. 2024CTRLorALTer: Conditional LoRAdapter for Efficient 0-Shot Control & Altering of T2I Modelscited by 4
  26. 2026SCMAPR: Self-Correcting Multi-Agent Prompt Refinement for Complex-Scenario Text-to-Video Generationcited by 4
  27. 2024Misrepresented Technological Solutions in Imagined Futures: The Origins and Dangers of AI Hype in the Research Communitycited by 3
  28. 2024RAVEL: Rare Concept Generation and Editing via Graph-driven Relational Guidancecited by 3
  29. 2025Poetry in Pixels: Prompt Tuning for Poem Image Generation via Diffusion Modelscited by 3
  30. 2025GenColor: Generative Color-Concept Association in Visual Designcited by 3
  31. 2024CoProNN: Concept-based Prototypical Nearest Neighbors for Explaining Vision Modelscited by 2
  32. 2024Financial Models in Generative Art: Black-Scholes-Inspired Concept Blending in Text-to-Image Diffusioncited by 2
  33. 2025FreSca: Scaling in Frequency Space Enhances Diffusion Modelscited by 2
  34. 2025ConceptMix++: Leveling the Playing Field in Text-to-Image Benchmarking via Iterative Prompt Optimizationcited by 2
  35. 2026RefAny3D: 3D Asset-Referenced Diffusion Models for Image Generationcited by 2
  36. 2024Stylecodes: Encoding Stylistic Information For Image Generationcited by 1
  37. 2024Reward Incremental Learning in Text-to-Image Generationcited by 1
  38. 2025Energy-Guided Optimization for Personalized Image Editing with Pretrained Text-to-Image Diffusion Modelscited by 1

50 most cited of 58 All 58 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Reward-Agnostic Prompt Optimization for Text-to-Image Diffusion Models
  2. 2023Prompting Hard or Hardly Prompting: Prompt Inversion for Text-to-Image Diffusion Models
  3. 2024Words Worth a Thousand Pictures: Measuring and Understanding Perceptual Variability in Text-to-Image Generation
  4. 2023Reverse Stable Diffusion: What prompt was used to generate this image?
  5. 2022DiffusionDB: A Large-scale Prompt Gallery Dataset for Text-to-Image Generative Models
  6. 2023Diffusion Explainer: Visual Explanation for Text-to-image Stable Diffusion
  7. 2024No Longer Trending on Artstation: Prompt Analysis of Generative AI Art
  8. 2024Seek for Incantations: Towards Accurate Text-to-Image Diffusion Synthesis through Prompt Engineering