mlsanityData to 7 Oct 2026

On Discrete Prompt Optimization for Diffusion Models

arXiv 2407.01606 ↗First posted 27 Jun 2024cs.LG cs.AI cs.CLcited by 28

On the map: Diffusion models → Image generation, text-to-image and editing

Abstract (arXiv)

This paper introduces the first gradient-based framework for prompt optimization in text-to-image diffusion models. We formulate prompt engineering as a discrete optimization problem over the language space. Two major challenges arise in efficiently finding a solution to this problem: (1) Enormous Domain Space: Setting the domain to the entire language space poses significant difficulty to the optimization process. (2) Text Gradient: Efficiently computing the text gradient is challenging, as it requires backpropagating through the inference steps of the diffusion model and a non-differentiable embedding lookup table. Beyond the problem formulation, our main technical contributions lie in solving the above challenges. First, we design a family of dynamically generated compact subspaces comprised of only the most relevant words to user input, substantially restricting the domain space. Second, we introduce "Shortcut Text Gradient" -- an effective replacement for the text gradient that can be obtained with constant memory and runtime. Empirical evaluation on prompts collected from diverse sources (DiffusionDB, ChatGPT, COCO) suggests that our method can discover prompts that substantially improve (prompt enhancement) or destroy (adversarial attack) the faithfulness of images generated by the text-to-image diffusion model.

Key facts. Cited by 28 papers in the map (data through 7 Oct 2026). Placed in Image generation, text-to-image and editing, one of 14 directions in Diffusion models. Builds on 42 of its 51 references that are in the map. First posted on arXiv 27 Jun 2024.

Builds on

42 of 51 references.

Closest ancestors · 7

Same direction, newest first

  1. 2023LLM-grounded Diffusion: Enhancing Prompt Understanding of Text-to-Image Diffusion Models with Large Language Models23 May
  2. 2022Training-Free Structured Diffusion Guidance for Compositional Text-to-Image Synthesis9 Dec
  3. 2022Investigating Prompt Engineering in Diffusion Models21 Nov
  4. 2022DiffusionDB: A Large-scale Prompt Gallery Dataset for Text-to-Image Generative Models26 Oct
  5. 2022Compositional Visual Generation with Composable Diffusion Models3 Jun
  6. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding23 May
  7. 2022Hierarchical Text-Conditional Image Generation with CLIP Latents13 Apr

Other references · 35

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  3. 2015U-Net: Convolutional Networks for Biomedical Image Segmentationcited by 19,604
  4. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  5. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  6. 2022Training language models to follow instructions with human feedbackcited by 15,712
  7. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  8. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  9. 2020Denoising Diffusion Implicit Modelscited by 9,655
  10. 2021Diffusion Models Beat GANs on Image Synthesiscited by 8,185
  11. 2022Classifier-Free Diffusion Guidancecited by 5,717
  12. 2017Deep reinforcement learning from human preferencescited by 3,846
23 more
  1. 2022An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversioncited by 2,231
  2. 2022Imagen Video: High Definition Video Generation with Diffusion Modelscited by 1,765
  3. 2022Scaling Autoregressive Models for Content-Rich Text-to-Image Generationcited by 1,287
  4. 2022Large Language Models Are Human-Level Prompt Engineerscited by 1,020
  5. 2023ImageReward: Learning and Evaluating Human Preferences for Text-to-Image Generationcited by 1,010
  6. 2023Large Language Models as Optimizerscited by 830
  7. 2023Muse: Text-To-Image Generation via Masked Generative Transformerscited by 640
  8. 2023Automatic Prompt Optimization with "Gradient Descent" and Beam Searchcited by 628
  9. 2018Black-box Adversarial Attacks with Limited Queries and Informationcited by 499
  10. 2018FBNet: Hardware-Aware Efficient ConvNet Design via Differentiable Neural Architecture Searchcited by 480
  11. 2022Diffusion Models for Adversarial Purificationcited by 469
  12. 2018Generating Natural Language Adversarial Examplescited by 422
  13. 2023Hard Prompts Made Easy: Gradient-Based Discrete Optimization for Prompt Tuning and Discoverycited by 329
  14. 2016Ladder Variational Autoencoderscited by 315
  15. 2022VQGAN-CLIP: Open Domain Image Generation and Editing with Natural Language Guidancecited by 285
  16. 2019Searching for A Robust Neural Architecture in Four GPU Hourscited by 258
  17. 2021Gradient-based Adversarial Attacks against Text Transformerscited by 233
  18. 2021Design Guidelines for Prompt Engineering Text-to-Image Generative Modelscited by 216
  19. 2022Optimizing Prompts for Text-to-Image Generationcited by 204
  20. 2023EvoPrompt: Connecting LLMs with Evolutionary Algorithms Yields Powerful Prompt Optimizerscited by 189
  21. 2022Learning Fast Samplers for Diffusion Models by Differentiating Through Sample Qualitycited by 181
  22. 2018Query-Efficient Hard-label Black-box Attack:An Optimization-based Approachcited by 130
  23. 2023InstructZero: Efficient Instruction Optimization for Black-Box Large Language Modelscited by 68

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 5; 2025: 15; 2026: 8.

28 papers.

Sort:

By area:

  1. 2024Prompt-A-Video: Prompt Your Video Diffusion Model via Preference-Aligned LLMcited by 22
  2. 2025Maestro: Self-Improving Text-to-Image Generation via Agent Orchestrationcited by 18
  3. 2025Learning to Sample Effective and Diverse Prompts for Text-to-Image Generationcited by 16
  4. 2024ReNeg: Learning Negative Embedding with Reward Guidancecited by 9
  5. 2024Safeguarding Text-to-Image Generation via Inference-Time Prompt-Noise Optimizationcited by 7
  6. 2025SketchFlex: Facilitating Spatial-Semantic Coherence in Text-to-Image Generation with Region-Based Sketchescited by 6
  7. 2025A Survey of Automatic Prompt Optimization with Instruction-focused Heuristic-based Search Algorithmcited by 6
  8. 2025SUB: Benchmarking CBM Generalization via Synthetic Attribute Substitutionscited by 6
  9. 2025Iterative Prompt Refinement for Safer Text-to-Image Generationcited by 6
  10. 2025GenPilot: A Multi-Agent System for Test-Time Prompt Optimization in Image Generationcited by 6
  11. 2024ComfyGI: Automatic Improvement of Image Generation Workflowscited by 5
  12. 2025Reward-Agnostic Prompt Optimization for Text-to-Image Diffusion Modelscited by 5
16 more
  1. 2026Embedding-perturbed Exploration Preference Optimization for Flow Modelscited by 5
  2. 2025A First-order Generative Bilevel Optimization Framework for Diffusion Modelscited by 4
  3. 2025Guiding What Not to Generate: Automated Negative Prompting for Text-Image Alignmentcited by 4
  4. 2025NegFaceDiff: The Power of Negative Context in Identity-Conditioned Diffusion for Synthetic Face Generationcited by 3
  5. 2026IDperturb: Enhancing Variation in Synthetic Face Generation via Angular Perturbationcited by 2
  6. 2026Preference Score Distillation: Leveraging 2D Rewards to Align Text-to-3D Generation with Human Preferencecited by 2
  7. 2024ArtAug: Enhancing Text-to-Image Generation through Synthesis-Understanding Interactioncited by 1
  8. 2025Which Way from B to A: The role of embedding geometry in image interpolation for Stable Diffusioncited by 1
  9. 2026SIDiffAgent: Self-Improving Diffusion Agentcited by 1
  10. 2026Attention to Neural Plagiarism: Diffusion Models Can Plagiarize Your Copyrighted Images!cited by 1
  11. 2026AdaptDiff: Adaptive Guidance in Diffusion Models for Diverse and Identity-Consistent Face Synthesis (Student Abstract)cited by 1
  12. 2026DCMorph: Face Morphing via Dual-Stream Cross-Attention Diffusioncited by 1
  13. 2026Consistency Regularised Gradient Flows for Inverse Problemscited by 1
  14. 2025IPGO: Indirect Prompt Gradient Optimization for Parameter-Efficient Prompt-level Fine-Tuning on Text-to-Image Modelscited by 0
  15. 2025Interpretable Diffusion Models with B-cos Networkscited by 0
  16. 2025Exposing Hidden Biases in Text-to-Image Models via Automated Prompt Searchcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023Prompting Hard or Hardly Prompting: Prompt Inversion for Text-to-Image Diffusion Models
  2. 2025Reusing Computation in Text-to-Image Diffusion for Efficient Generation of Image Sets
  3. 2023Prompt-Free Diffusion: Taking "Text" out of Text-to-Image Diffusion Models
  4. 2026ELROND: Exploring and decomposing intrinsic capabilities of diffusion models
  5. 2024Prompt Diffusion Robustifies Any-Modality Prompt Learning
  6. 2026Rethinking Global Text Conditioning in Diffusion Transformers
  7. 2024Exploring the Role of Large Language Models in Prompt Encoding for Diffusion Models
  8. 2026Evolutionary Token-Level Prompt Optimization for Diffusion Models