Position: Towards Implicit Prompt For Text-To-Image Models
Not on the map. Why?
Abstract (arXiv)
Recent text-to-image (T2I) models have had great success, and many benchmarks have been proposed to evaluate their performance and safety. However, they only consider explicit prompts while neglecting implicit prompts (hint at a target without explicitly mentioning it). These prompts may get rid of safety constraints and pose potential threats to the applications of these models. This position paper highlights the current state of T2I models toward implicit prompts. We present a benchmark named ImplicitBench and conduct an investigation on the performance and impacts of implicit prompts with popular T2I models. Specifically, we design and collect more than 2,000 implicit prompts of three aspects: General Symbols, Celebrity Privacy, and Not-Safe-For-Work (NSFW) Issues, and evaluate six well-known T2I models' capabilities under these implicit prompts. Experiment results show that (1) T2I models are able to accurately create various target symbols indicated by implicit prompts; (2) Implicit prompts bring potential risks of privacy leakage for T2I models. (3) Constraints of NSFW in most of the evaluated T2I models can be bypassed with implicit prompts. We call for increased attention to the potential and risks of implicit prompts in the T2I community and further investigation into the capabilities and impacts of implicit prompts, advocating for a balanced approach that harnesses their benefits while mitigating their risks.
Key facts. Cited by 1 paper in the map (data through 7 Oct 2026). Builds on 35 of its 48 references that are in the map. First posted on arXiv 4 Mar 2024.
Builds on
35 of 48 references.
References · 35
- 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
- 2020Denoising Diffusion Probabilistic Modelscited by 20,851
- 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
- 2014Microsoft COCO: Common Objects in Contextcited by 17,498
- 2023Visual Instruction Tuningcited by 7,792
- 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
- 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
- 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
- 2023SDXL: Improving Latent Diffusion Models for High-Resolution Image Synthesiscited by 4,290
- 2021Zero-Shot Text-to-Image Generationcited by 4,276
- 2016Improved Techniques for Training GANscited by 2,938
- 2021CLIPScore: A Reference-free Evaluation Metric for Image Captioningcited by 2,388
23 more
- 2022An Image is Worth One Word: Personalizing Text-to-Image Generation using Textual Inversioncited by 2,231
- 2023ImageReward: Learning and Evaluating Human Preferences for Text-to-Image Generationcited by 1,010
- 2016Joint Face Detection and Alignment using Multi-task Cascaded Convolutional Networkscited by 895
- 2024Eyes Wide Shut? Exploring the Visual Shortcomings of Multimodal LLMscited by 719
- 2023Siren's Song in the AI Ocean: A Survey on Hallucination in Large Language Modelscited by 695
- 2017AttnGAN: Fine-Grained Text to Image Generation with Attentional Generative Adversarial Networkscited by 664
- 2017Generative Adversarial Networks: An Overviewcited by 588
- 2022Safe Latent Diffusion: Mitigating Inappropriate Degeneration in Diffusion Modelscited by 501
- 2022Make-A-Scene: Scene-Based Text-to-Image Generation with Human Priorscited by 487
- 2022CogView2: Faster and Better Text-to-Image Generation via Hierarchical Transformerscited by 305
- 2023SPHINX: The Joint Mixing of Weights, Tasks, and Visual Embeddings for Multi-modal Large Language Modelscited by 283
- 2022Red-Teaming the Stable Diffusion Safety Filtercited by 257
- 2023T2I-CompBench++: An Enhanced and Comprehensive Benchmark for Compositional Text-to-image Generationcited by 253
- 2023Stable Bias: Analyzing Societal Representations in Diffusion Modelscited by 225
- 2022Optimizing Prompts for Text-to-Image Generationcited by 204
- 2023Unsafe Diffusion: On the Generation of Unsafe Images and Hateful Memes From Text-To-Image Modelscited by 171
- 2014Deep Learning Face Representation by Joint Identification-Verificationcited by 162
- 2020DF-GAN: A Simple and Effective Baseline for Text-to-Image Synthesiscited by 161
- 2023Holistic Evaluation of Text-To-Image Modelscited by 160
- 2023HRS-Bench: Holistic, Reliable and Scalable Benchmark for Text-to-Image Modelscited by 94
- 2023The Bias Amplification Paradox in Text-to-Image Generationcited by 69
- 2022Human Evaluation of Text-to-Image Models on a Multi-Task Benchmarkcited by 41
- 2023MultiFusion: Fusing Pre-Trained Models for Multi-Lingual, Multi-Modal Image Generationcited by 14
Built on by
Citing papers per year: 2024: 1.
1 paper.