mlsanityData to 14 Oct 2026

ART: Automatic Red-teaming for Text-to-Image Models to Protect Benign Users

arXiv 2405.19360 ↗First posted 24 May 2024cs.CR cs.AIcited by 35

On the map: LLM safety and security → Vision-language and text-to-image safety

Abstract (arXiv)

Large-scale pre-trained generative models are taking the world by storm, due to their abilities in generating creative content. Meanwhile, safeguards for these generative models are developed, to protect users' rights and safety, most of which are designed for large language models. Existing methods primarily focus on jailbreak and adversarial attacks, which mainly evaluate the model's safety under malicious prompts. Recent work found that manually crafted safe prompts can unintentionally trigger unsafe generations. To further systematically evaluate the safety risks of text-to-image models, we propose a novel Automatic Red-Teaming framework, ART. Our method leverages both vision language model and large language model to establish a connection between unsafe generations and their prompts, thereby more efficiently identifying the model's vulnerabilities. With our comprehensive experiments, we reveal the toxicity of the popular open-source text-to-image models. The experiments also validate the effectiveness, adaptability, and great diversity of ART. Additionally, we introduce three large-scale red-teaming datasets for studying the safety risks associated with text-to-image models. Datasets and models can be found in https://github.com/GuanlinLee/ART.

Key facts. Cited by 35 papers in the map (data through 14 Oct 2026). Placed in Vision-language and text-to-image safety, one of 12 directions in LLM safety and security. Builds on 27 of its 38 references that are in the map. First posted on arXiv 24 May 2024.

Builds on

27 of 38 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024Adversarial Nibbler: An Open Red-Teaming Method for Identifying Diverse Harms in Text-to-Image Generation14 Feb
  2. 2024Jailbreaking Attack against Multimodal Large Language Model4 Feb
  3. 2023FLIRT: Feedback Loop In-context Red Teaming8 Aug

Other references · 24

  1. 2017Attention Is All You Needcited by 60,361
  2. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,863
  3. 2023GPT-4 Technical Reportcited by 18,396
  4. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  5. 2022Training language models to follow instructions with human feedbackcited by 15,971
  6. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  7. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  8. 2023Visual Instruction Tuningcited by 7,869
  9. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,386
  10. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 3,104
  11. 2023AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Modelscited by 785
  12. 2023Otter: A Multi-Modal Model with In-Context Instruction Tuningcited by 550
12 more
  1. 2023Erasing Concepts from Diffusion Modelscited by 539
  2. 2022Safe Latent Diffusion: Mitigating Inappropriate Degeneration in Diffusion Modelscited by 513
  3. 2024MACE: Mass Concept Erasure in Diffusion Modelscited by 307
  4. 2021Gradient-based Adversarial Attacks against Text Transformerscited by 234
  5. 2023MMA-Diffusion: MultiModal Attack on Diffusion Modelscited by 194
  6. 2023Unsafe Diffusion: On the Generation of Unsafe Images and Hateful Memes From Text-To-Image Modelscited by 175
  7. 2023MART: Improving LLM Safety with Multi-round Automatic Red-Teamingcited by 163
  8. 2022Can Machines Help Us Answering Question 16 in Datasheets, and In Turn Reflecting on Inappropriate Content?cited by 155
  9. 2023Receler: Reliable Concept Erasing of Text-to-Image Diffusion Models via Lightweight Eraserscited by 120
  10. 2024ALERT: A Comprehensive Benchmark for Assessing Large Language Models' Safety through Red Teamingcited by 98
  11. 2024Curiosity-driven Red-teaming for Large Language Modelscited by 88
  12. 2024Groot: Adversarial Testing for Generative Text-to-Image Models with Tree-based Semantic Transformationcited by 14

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 7; 2025: 17; 2026: 11.

35 papers.

Sort:

By area:

  1. 2024SAFREE: Training-Free and Adaptive Guard for Safe Text-to-Image And Video Generationcited by 133
  2. 2024Any-Resolution AI-Generated Image Detection by Spectral Learningcited by 57
  3. 2025CURE: Concept Unlearning via Orthogonal Representation Editing in Diffusion Modelscited by 27
  4. 2025Detect-and-Guide: Self-regulation of Diffusion Models for Safe Text-to-Image Generation via Guideline Token Optimizationcited by 13
  5. 2024SteerDiff: Steering towards Safe Text-to-Image Diffusion Modelscited by 10
  6. 2024Safe + Safe = Unsafe? Exploring How Safe Images Can Be Exploited to Jailbreak Large Vision-Language Modelscited by 10
  7. 2025Image Can Bring Your Memory Back: A Novel Multi-Modal Guided Attack against Image Generation Model Unlearningcited by 10
  8. 2025Towards Resilient Safety-driven Unlearning for Diffusion Models against Downstream Fine-tuningcited by 9
  9. 2024Recent advancements in LLM Red-Teaming: Techniques, Defenses, and Ethical Considerationscited by 8
  10. 2025Red-Teaming Text-to-Image Systems by Rule-based Preference Modelingcited by 8
  11. 2025SafeGuider: Robust and Practical Content Safety Control for Text-to-Image Modelscited by 6
  12. 2024Towards Effective Discrimination Testing for Generative AIcited by 5
23 more
  1. 2025OVERT: A Benchmark for Over-Refusal Evaluation on Text-to-Image Modelscited by 5
  2. 2025DREAM: Scalable Red Teaming for Text-to-Image Generative Systems via Distribution Modelingcited by 5
  3. 2025PurifyGen: A Risk-Discrimination and Semantic-Purification Model for Safe Text-to-Image Generationcited by 5
  4. 2024Insights on Disagreement Patterns in Multimodal Safety Perception across Diverse Rater Groupscited by 4
  5. 2025GenBreak: Red Teaming Text-to-Image Generators Using Large Language Modelscited by 4
  6. 2025Now You See It, Now You Don't - Instant Concept Erasure for Safe Text-to-Image and Video Generationcited by 4
  7. 2025Wukong Framework for Not Safe For Work Detection in Text-to-Image systemscited by 3
  8. 2025TEAR: Temporal-aware Automated Red-teaming for Text-to-Video Modelscited by 3
  9. 2026JANUS: A Lightweight Framework for Jailbreaking Text-to-Image Models via Distribution Optimizationcited by 3
  10. 2025A Systematic Review of Open Datasets Used in Text-to-Image (T2I) Gen AI Model Safetycited by 2
  11. 2025Mind the Trojan Horse: Image Prompt Adapter Enabling Scalable and Deceptive Jailbreakingcited by 2
  12. 2026MacPrompt: Maraconic-guided Jailbreak against Text-to-Image Modelscited by 2
  13. 2025PolyJuice Makes It Real: Black-Box, Universal Red Teaming for Synthetic Image Detectorscited by 1
  14. 2025When Search Goes Wrong: Red-Teaming Web-Augmented Large Language Modelscited by 1
  15. 2026STARE: Step-wise Temporal Alignment and Red-teaming Engine for Multi-modal Toxicity Attackcited by 0
  16. 2026No Safe Dose: How Training Data Drives Unsafe Image Generationcited by 0
  17. 2026FoeGlass: Simple In-Context Learning Is Enough for Red Teaming Audio Deepfake Detectorscited by 0
  18. 2026RedEdit: Agentic Red-Teaming of Image Safety Classifiers via MCTS-Guided Photo-Editingcited by 0
  19. 2026Position: Preventing AI-Generated CSAM Necessitates New Approaches to AI Safetycited by 0
  20. 2026ARENA: Automated Red-Teaming for Large Audio Language Modelscited by 0
  21. 2026DiSCO: Defending text-to-image generation through distribution-guided contrastive prompt optimizationcited by 0
  22. 2026RISE: Red-teaming via Iterative Strategy Evolution for Modern Text-to-Image Modelscited by 0
  23. 2026Keep It CALM: Analyzing the Limits of Global Unsafety in Text-to-Image Generationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Low-Effort Jailbreak Attacks Against Text-to-Image Safety Filters
  2. 2026Open Models, Open Risks: Measuring Unsafe Generation in Text-to-Image Models In the Wild
  3. 2024Red-Teaming Text-to-Image Models via In-Context Experience Replay and Semantic-Preserving Prompt Rewriting
  4. 2026When Safe Concepts Become Unsafe: Multi-Concept Compositional Vulnerabilities in Text-to-Image Models
  5. 2026Automatic Red Teaming for Implicit Vulnerabilities of Text-to-Image Models
  6. 2025From Seed to Harvest: Augmenting Human Creativity with AI for Red-teaming Text-to-Image Models
  7. 2025Jailbreaking Safeguarded Text-to-Image Models via Large Language Models
  8. 2023Prompting4Debugging: Red-Teaming Text-to-Image Diffusion Models by Finding Problematic Prompts