mlsanityData to 7 Oct 2026

Prompt Recovery for Image Generation Models: A Comparative Study of Discrete Optimizers

arXiv 2408.06502 ↗First posted 12 Aug 2024cs.CV cs.LGcited by 5

Not on the map. Why?

Abstract (arXiv)

Recovering natural language prompts for image generation models, solely based on the generated images is a difficult discrete optimization problem. In this work, we present the first head-to-head comparison of recent discrete optimization techniques for the problem of prompt inversion. We evaluate Greedy Coordinate Gradients (GCG), PEZ , Random Search, AutoDAN and BLIP2's image captioner across various evaluation metrics related to the quality of inverted prompts and the quality of the images generated by the inverted prompts. We find that focusing on the CLIP similarity between the inverted prompts and the ground truth image acts as a poor proxy for the similarity between ground truth image and the image generated by the inverted prompts. While the discrete optimizers effectively minimize their objectives, simply using responses from a well-trained captioner often leads to generated images that more closely resemble those produced by the original prompts.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Builds on 26 of its 33 references that are in the map. First posted on arXiv 12 Aug 2024.

Builds on

26 of 33 references.

References · 26

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  4. 2020Denoising Diffusion Implicit Modelscited by 9,655
  5. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  6. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 2,914
  7. 2021CLIPScore: A Reference-free Evaluation Metric for Image Captioningcited by 2,388
  8. 2023Jailbreaking Black Box Large Language Models in Twenty Queriescited by 1,326
  9. 2018Demystifying MMD GANscited by 1,268
  10. 2022Null-text Inversion for Editing Real Images using Guided Diffusion Modelscited by 1,168
  11. 2023Adversarial Diffusion Distillationcited by 741
  12. 2023Automatic Prompt Optimization with "Gradient Descent" and Beam Searchcited by 628
14 more
  1. 2022DiffusionDB: A Large-scale Prompt Gallery Dataset for Text-to-Image Generative Modelscited by 376
  2. 2023Hard Prompts Made Easy: Gradient-Based Discrete Optimization for Prompt Tuning and Discoverycited by 329
  3. 2016Model-Agnostic Interpretability of Machine Learningcited by 219
  4. 2023StyleDrop: Text-to-Image Generation in Any Stylecited by 197
  5. 2020Counterfactual Explanations and Algorithmic Recourses for Machine Learning: A Reviewcited by 174
  6. 2024Rethinking LLM Memorization through the Lens of Adversarial Compressioncited by 91
  7. 2021Prompt Waywardness: The Curious Case of Discretized Interpretation of Continuous Promptscited by 64
  8. 2023Prompting Hard or Hardly Prompting: Prompt Inversion for Text-to-Image Diffusion Modelscited by 40
  9. 2024Extracting Prompts by Inverting LLM Outputscited by 40
  10. 2024Alpaca against Vicuna: Using LLMs to Uncover Memorization of LLMscited by 33
  11. 2024Automated Black-box Prompt Engineering for Personalized Text-to-Image Generationcited by 12
  12. 2024Prompt Optimizer of Text-to-Image Diffusion Models for Abstract Concept Understandingcited by 9
  13. 2024DrawL: Understanding the Effects of Non-Mainstream Dialects in Prompted Image Generationcited by 2
  14. 2024FUSE-ing Language Models: Zero-Shot Adapter Discovery for Prompt Optimization Across Tokenizerscited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 3; 2026: 2.

5 papers.

  1. 2025Reverse Prompt: Cracking the Recipe Inside Text-to-Image Generationcited by 3
  2. 2025The Resurgence of GCG Adversarial Attacks on Large Language Modelscited by 2
  3. 2025Prompt Pirates Need a Map: Stealing Seeds helps Stealing Promptscited by 1
  4. 2026FoeGlass: Simple In-Context Learning Is Enough for Red Teaming Audio Deepfake Detectorscited by 0
  5. 2026PROVE: Training-Free Prompt Recovery using Verifiable Evidencecited by 0