# Prompt Stealing Attacks Against Text-to-Image Generation Models

arXiv: https://arxiv.org/abs/2302.09923 · HTML: https://mlsanity.com/paper/2302.09923/

## Key facts

- Cited by 51 papers in the map (data through 14 Oct 2026).
- Builds on 42 of its 71 references that are in the map.
- First posted on arXiv 20 Feb 2023.

## Builds on (most cited first, 30 of 42)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [High-Resolution Image Synthesis with Latent Diffusion Models](https://mlsanity.com/paper/2112.10752/) (2021)
- [Hierarchical Text-Conditional Image Generation with CLIP Latents](https://mlsanity.com/paper/2204.06125/) (2022)
- [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://mlsanity.com/paper/2205.11487/) (2022)
- [BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation](https://mlsanity.com/paper/2201.12086/) (2022)
- [Towards Evaluating the Robustness of Neural Networks](https://mlsanity.com/paper/1608.04644/) (2016)
- [Large Scale GAN Training for High Fidelity Natural Image Synthesis](https://mlsanity.com/paper/1809.11096/) (2018)
- [InstructPix2Pix: Learning to Follow Image Editing Instructions](https://mlsanity.com/paper/2211.09800/) (2022)
- [Membership Inference Attacks against Machine Learning Models](https://mlsanity.com/paper/1610.05820/) (2016)
- [Adversarial examples in the physical world](https://mlsanity.com/paper/1607.02533/) (2016)
- [High-Resolution Image Synthesis and Semantic Manipulation with Conditional GANs](https://mlsanity.com/paper/1711.11585/) (2017)
- [BadNets: Identifying Vulnerabilities in the Machine Learning Model Supply Chain](https://mlsanity.com/paper/1708.06733/) (2017)
- [Practical Black-Box Attacks against Machine Learning](https://mlsanity.com/paper/1602.02697/) (2016)
- [Membership Inference Attacks From First Principles](https://mlsanity.com/paper/2112.03570/) (2021)
- [Ensemble Adversarial Training: Attacks and Defenses](https://mlsanity.com/paper/1705.07204/) (2017)
- [The Limitations of Deep Learning in Adversarial Settings](https://mlsanity.com/paper/1511.07528/) (2015)
- [Extracting Training Data from Diffusion Models](https://mlsanity.com/paper/2301.13188/) (2023)
- [ClipCap: CLIP Prefix for Image Captioning](https://mlsanity.com/paper/2111.09734/) (2021)
- [GIT: A Generative Image-to-text Transformer for Vision and Language](https://mlsanity.com/paper/2205.14100/) (2022)
- [ML-Leaks: Model and Data Independent Membership Inference Attacks and Defenses on Machine Learning Models](https://mlsanity.com/paper/1806.01246/) (2018)
- [Label-Only Membership Inference Attacks](https://mlsanity.com/paper/2007.14321/) (2020)
- [Hidden Trigger Backdoor Attacks](https://mlsanity.com/paper/1910.00033/) (2019)
- [Reflection Backdoor: A Natural Backdoor Attack on Deep Neural Networks](https://mlsanity.com/paper/2007.02343/) (2020)
- [HopSkipJumpAttack: A Query-Efficient Decision-Based Attack](https://mlsanity.com/paper/1904.02144/) (2019)
- [Input-Aware Dynamic Backdoor Attack](https://mlsanity.com/paper/2010.08138/) (2020)
- [VQGAN-CLIP: Open Domain Image Generation and Editing with Natural Language Guidance](https://mlsanity.com/paper/2204.08583/) (2022)
- [Design Guidelines for Prompt Engineering Text-to-Image Generative Models](https://mlsanity.com/paper/2109.06977/) (2021)
- [BadNL: Backdoor Attacks against NLP Models with Semantic-preserving Improvements](https://mlsanity.com/paper/2006.01043/) (2020)
- [Optimizing Prompts for Text-to-Image Generation](https://mlsanity.com/paper/2212.09611/) (2022)

## Built on by (most cited first, 30 of 51)

- ["Do Anything Now": Characterizing and Evaluating In-The-Wild Jailbreak Prompts on Large Language Models](https://mlsanity.com/paper/2308.03825/) (2023)
- [Large Language Model Agent: A Survey on Methodology, Applications and Challenges](https://mlsanity.com/paper/2503.21460/) (2025)
- [Unsafe Diffusion: On the Generation of Unsafe Images and Hateful Memes From Text-To-Image Models](https://mlsanity.com/paper/2305.13873/) (2023)
- [Unleashing the potential of prompt engineering for large language models](https://mlsanity.com/paper/2310.14735/) (2023)
- [Prompt Stealing Attacks Against Large Language Models](https://mlsanity.com/paper/2402.12959/) (2024)
- [Watermarking Diffusion Model](https://mlsanity.com/paper/2305.12502/) (2023)
- [The Attack and Defense Landscape of Agentic AI: A Comprehensive Survey](https://mlsanity.com/paper/2603.11088/) (2026)
- [Large Model Based Agents: State-of-the-Art, Cooperation Paradigms, Security and Privacy, and Future Trends](https://mlsanity.com/paper/2409.14457/) (2024)
- [A Survey on Model Extraction Attacks and Defenses for Large Language Models](https://mlsanity.com/paper/2506.22521/) (2025)
- [Rethinking Data Protection in the (Generative) Artificial Intelligence Era](https://mlsanity.com/paper/2507.03034/) (2025)
- [Rag and Roll: An End-to-End Evaluation of Indirect Prompt Manipulations in LLM-based Application Frameworks](https://mlsanity.com/paper/2408.05025/) (2024)
- [A Systematic Survey of Prompt Engineering on Vision-Language Foundation Models](https://mlsanity.com/paper/2307.12980/) (2023)
- [ControlNET: A Firewall for RAG-based LLM System](https://mlsanity.com/paper/2504.09593/) (2025)
- [Chain-of-Jailbreak Attack for Image Generation Models via Editing Step by Step](https://mlsanity.com/paper/2410.03869/) (2024)
- [SOS! Soft Prompt Attack Against Open-Source Large Language Models](https://mlsanity.com/paper/2407.03160/) (2024)
- [Iteratively Prompting Multimodal LLMs to Reproduce Natural and AI-Generated Images](https://mlsanity.com/paper/2404.13784/) (2024)
- [A Survey on Responsible Generative AI: What to Generate and What Not](https://mlsanity.com/paper/2404.05783/) (2024)
- [SafeGuider: Robust and Practical Content Safety Control for Text-to-Image Models](https://mlsanity.com/paper/2510.05173/) (2025)
- [Capability-aware Prompt Reformulation Learning for Text-to-Image Generation](https://mlsanity.com/paper/2403.19716/) (2024)
- [Vulnerability of Text-to-Image Models to Prompt Template Stealing: A Differential Evolution Approach](https://mlsanity.com/paper/2502.14285/) (2025)
- [World-To-Image: Grounding Text-to-Image Generation with Agent-Driven World Knowledge](https://mlsanity.com/paper/2510.04201/) (2025)
- [Are You Copying My Prompt? Protecting the Copyright of Vision Prompt for VPaaS via Watermark](https://mlsanity.com/paper/2405.15161/) (2024)
- [Regeneration Based Training-free Attribution of Fake Images Generated by Text-to-Image Generative Models](https://mlsanity.com/paper/2403.01489/) (2024)
- [CapRecover: A Cross-Modality Feature Inversion Attack Framework on Vision Language Models](https://mlsanity.com/paper/2507.22828/) (2025)
- [VideoGuard: Protecting Video Content from Unauthorized Editing](https://mlsanity.com/paper/2508.03480/) (2025)
- [PromptCOS: Towards Content-only System Prompt Copyright Auditing for LLMs](https://mlsanity.com/paper/2509.03117/) (2025)
- [Responsible Diffusion: A Comprehensive Survey on Safety, Ethics, and Trust in Diffusion Models](https://mlsanity.com/paper/2509.22723/) (2025)
- [SafeGen-Bench: Benchmarking Safety in Image-Conditioned Text-to-Video Generation](https://mlsanity.com/paper/2606.01481/) (2026)
- [Double Backdoored: Converting Code Large Language Model Backdoors to Traditional Malware via Adversarial Instruction Tuning Attacks](https://mlsanity.com/paper/2404.18567/) (2024)
- [Promptly Yours? A Human Subject Study on Prompt Inference in AI-Generated Art](https://mlsanity.com/paper/2410.08406/) (2024)

## Abstract (arXiv)

Text-to-Image generation models have revolutionized the artwork design process and enabled anyone to create high-quality images by entering text descriptions called prompts. Creating a high-quality prompt that consists of a subject and several modifiers can be time-consuming and costly. In consequence, a trend of trading high-quality prompts on specialized marketplaces has emerged. In this paper, we perform the first study on understanding the threat of a novel attack, namely prompt stealing attack, which aims to steal prompts from generated images by text-to-image generation models. Successful prompt stealing attacks directly violate the intellectual property of prompt engineers and jeopardize the business model of prompt marketplaces. We first perform a systematic analysis on a dataset collected by ourselves and show that a successful prompt stealing attack should consider a prompt's subject as well as its modifiers. Based on this observation, we propose a simple yet effective prompt stealing attack, PromptStealer. It consists of two modules: a subject generator trained to infer the subject and a modifier detector for identifying the modifiers within the generated image. Experimental results demonstrate that PromptStealer is superior over three baseline methods, both quantitatively and qualitatively. We also make some initial attempts to defend PromptStealer. In general, our study uncovers a new attack vector within the ecosystem established by the popular text-to-image generation models. We hope our results can contribute to understanding and mitigating this emerging threat.