# Prompt Recovery for Image Generation Models: A Comparative Study of Discrete Optimizers

arXiv: https://arxiv.org/abs/2408.06502 · HTML: https://mlsanity.com/paper/2408.06502/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 33 references that are in the map.
- First posted on arXiv 12 Aug 2024.

## Builds on (most cited first, 26 of 26)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Chain-of-Thought Prompting Elicits Reasoning in Large Language Models](https://mlsanity.com/paper/2201.11903/) (2022)
- [Denoising Diffusion Implicit Models](https://mlsanity.com/paper/2010.02502/) (2020)
- [BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models](https://mlsanity.com/paper/2301.12597/) (2023)
- [Universal and Transferable Adversarial Attacks on Aligned Language Models](https://mlsanity.com/paper/2307.15043/) (2023)
- [CLIPScore: A Reference-free Evaluation Metric for Image Captioning](https://mlsanity.com/paper/2104.08718/) (2021)
- [Jailbreaking Black Box Large Language Models in Twenty Queries](https://mlsanity.com/paper/2310.08419/) (2023)
- [Demystifying MMD GANs](https://mlsanity.com/paper/1801.01401/) (2018)
- [Null-text Inversion for Editing Real Images using Guided Diffusion Models](https://mlsanity.com/paper/2211.09794/) (2022)
- [Adversarial Diffusion Distillation](https://mlsanity.com/paper/2311.17042/) (2023)
- [Automatic Prompt Optimization with "Gradient Descent" and Beam Search](https://mlsanity.com/paper/2305.03495/) (2023)
- [DiffusionDB: A Large-scale Prompt Gallery Dataset for Text-to-Image Generative Models](https://mlsanity.com/paper/2210.14896/) (2022)
- [Hard Prompts Made Easy: Gradient-Based Discrete Optimization for Prompt Tuning and Discovery](https://mlsanity.com/paper/2302.03668/) (2023)
- [Model-Agnostic Interpretability of Machine Learning](https://mlsanity.com/paper/1606.05386/) (2016)
- [StyleDrop: Text-to-Image Generation in Any Style](https://mlsanity.com/paper/2306.00983/) (2023)
- [Counterfactual Explanations and Algorithmic Recourses for Machine Learning: A Review](https://mlsanity.com/paper/2010.10596/) (2020)
- [Rethinking LLM Memorization through the Lens of Adversarial Compression](https://mlsanity.com/paper/2404.15146/) (2024)
- [Prompt Waywardness: The Curious Case of Discretized Interpretation of Continuous Prompts](https://mlsanity.com/paper/2112.08348/) (2021)
- [Extracting Prompts by Inverting LLM Outputs](https://mlsanity.com/paper/2405.15012/) (2024)
- [Prompting Hard or Hardly Prompting: Prompt Inversion for Text-to-Image Diffusion Models](https://mlsanity.com/paper/2312.12416/) (2023)
- [Alpaca against Vicuna: Using LLMs to Uncover Memorization of LLMs](https://mlsanity.com/paper/2403.04801/) (2024)
- [Automated Black-box Prompt Engineering for Personalized Text-to-Image Generation](https://mlsanity.com/paper/2403.19103/) (2024)
- [Prompt Optimizer of Text-to-Image Diffusion Models for Abstract Concept Understanding](https://mlsanity.com/paper/2404.11589/) (2024)
- [DrawL: Understanding the Effects of Non-Mainstream Dialects in Prompted Image Generation](https://mlsanity.com/paper/2405.05382/) (2024)
- [FUSE-ing Language Models: Zero-Shot Adapter Discovery for Prompt Optimization Across Tokenizers](https://mlsanity.com/paper/2408.04816/) (2024)

## Built on by (most cited first, 5 of 5)

- [Reverse Prompt: Cracking the Recipe Inside Text-to-Image Generation](https://mlsanity.com/paper/2503.19937/) (2025)
- [The Resurgence of GCG Adversarial Attacks on Large Language Models](https://mlsanity.com/paper/2509.00391/) (2025)
- [Prompt Pirates Need a Map: Stealing Seeds helps Stealing Prompts](https://mlsanity.com/paper/2509.09488/) (2025)
- [FoeGlass: Simple In-Context Learning Is Enough for Red Teaming Audio Deepfake Detectors](https://mlsanity.com/paper/2606.05101/) (2026)
- [PROVE: Training-Free Prompt Recovery using Verifiable Evidence](https://mlsanity.com/paper/2608.13671/) (2026)

## Abstract (arXiv)

Recovering natural language prompts for image generation models, solely based on the generated images is a difficult discrete optimization problem. In this work, we present the first head-to-head comparison of recent discrete optimization techniques for the problem of prompt inversion. We evaluate Greedy Coordinate Gradients (GCG), PEZ , Random Search, AutoDAN and BLIP2's image captioner across various evaluation metrics related to the quality of inverted prompts and the quality of the images generated by the inverted prompts. We find that focusing on the CLIP similarity between the inverted prompts and the ground truth image acts as a poor proxy for the similarity between ground truth image and the image generated by the inverted prompts. While the discrete optimizers effectively minimize their objectives, simply using responses from a well-trained captioner often leads to generated images that more closely resemble those produced by the original prompts.