Reinforcement Learning-Based Prompt Template Stealing for Text-to-Image Models
On the map: Multimodal LLMs / vision-language models → MLLM jailbreaks and adversarial robustness
also in 1 more
Abstract (arXiv)
Multimodal Large Language Models (MLLMs) have transformed text-to-image workflows, allowing designers to create novel visual concepts with unprecedented speed. This progress has given rise to a thriving prompt trading market, where curated prompts that induce trademark styles are bought and sold. Although commercially attractive, prompt trading also introduces a largely unexamined security risk: the prompts themselves can be stolen. In this paper, we expose this vulnerability and present RLStealer, a reinforcement learning based prompt inversion framework that recovers its template from only a small set of example images. RLStealer treats template stealing as a sequential decision making problem and employs multiple similarity based feedback signals as reward functions to effectively explore the prompt space. Comprehensive experiments on publicly available benchmarks demonstrate that RLStealer gets state-of-the-art performance while reducing the total attack cost to under 13% of that required by existing baselines. Our further analysis confirms that RLStealer can effectively generalize across different image styles to efficiently steal unseen prompt templates. Our study highlights an urgent security threat inherent in prompt trading and lays the groundwork for developing protective standards in the emerging MLLMs marketplace.
Key facts. No citing papers in the map yet (data through 13 Oct 2026). Placed in MLLM jailbreaks and adversarial robustness, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 16 of its 21 references that are in the map. First posted on arXiv 27 Sep 2025.
Builds on
16 of 21 references.
Closest ancestors · 1
Same direction, newest first
Other references · 15
- 2014Adam: A Method for Stochastic Optimizationcited by 37,093
- 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,588
- 2022Training language models to follow instructions with human feedbackcited by 15,856
- 2017Proximal Policy Optimization Algorithmscited by 14,592
- 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,322
- 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,916
- 2013Playing Atari with Deep Reinforcement Learningcited by 3,221
- 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,068
- 2022Diffusion Models in Vision: A Surveycited by 1,012
- 2024DeepSeek-VL: Towards Real-World Vision-Language Understandingcited by 726
- 2021Design Guidelines for Prompt Engineering Text-to-Image Generative Modelscited by 218
- 2024Evolutionary Computation in the Era of Large Language Model: Survey and Roadmapcited by 90
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2025PROMPTMINER: Black-Box Prompt Stealing against Text-to-Image Generative Models via Reinforcement Learning and Fuzz Optimization
- 2025Towards Effective Prompt Stealing Attack against Text-to-Image Diffusion Models
- 2026PromptRL: Prompt Matters in RL for Flow-Based Image Generation
- 2025PromptEnhancer: A Simple Approach to Enhance Text-to-Image Models via Chain-of-Thought Prompt Rewriting
- 2025BadReward: Clean-Label Poisoning of Reward Models in Text-to-Image RLHF
- 2026Covert Visual Prompt Injection against Commercial Multimodal Large Language Models
- 2026PromptEcho: Annotation-Free Reward from Vision-Language Models for Text-to-Image Reinforcement Learning
- 2024Red-Teaming Text-to-Image Models via In-Context Experience Replay and Semantic-Preserving Prompt Rewriting