mlsanityData to 7 Oct 2026

FLIRT: Feedback Loop In-context Red Teaming

arXiv 2308.04265 ↗First posted 8 Aug 2023cs.AIcited by 81

On the map: LLM safety and security → Vision-language and text-to-image safety

Abstract (arXiv)

Warning: this paper contains content that may be inappropriate or offensive. As generative models become available for public use in various applications, testing and analyzing vulnerabilities of these models has become a priority. In this work, we propose an automatic red teaming framework that evaluates a given black-box model and exposes its vulnerabilities against unsafe and inappropriate content generation. Our framework uses in-context learning in a feedback loop to red team models and trigger them into unsafe content generation. In particular, taking text-to-image models as target models, we explore different feedback mechanisms to automatically learn effective and diverse adversarial prompts. Our experiments demonstrate that even with enhanced safety features, Stable Diffusion (SD) models are vulnerable to our adversarial prompts, raising concerns on their robustness in practical uses. Furthermore, we demonstrate that the proposed framework is effective for red teaming text-to-text models.

Key facts. Cited by 81 papers in the map (data through 7 Oct 2026). Placed in Vision-language and text-to-image safety, one of 12 directions in LLM safety and security. Builds on 32 of its 47 references that are in the map. First posted on arXiv 8 Aug 2023.

Builds on

32 of 47 references.

Closest ancestors · 1

Same direction, newest first

  1. 2022Red-Teaming the Stable Diffusion Safety Filter3 Oct

Other references · 31

  1. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  2. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  3. 2022Hierarchical Text-Conditional Image Generation with CLIP Latentscited by 6,539
  4. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  5. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
  6. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  7. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
  8. 2022Red Teaming Language Models with Language Modelscited by 983
  9. 2020Aligning AI With Shared Human Valuescited by 706
  10. 2022Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learnedcited by 702
  11. 2022Safe Latent Diffusion: Mitigating Inappropriate Degeneration in Diffusion Modelscited by 501
  12. 2022ToxiGen: A Large-Scale Machine-Generated Dataset for Adversarial and Implicit Hate Speech Detectioncited by 419
19 more
  1. 2023Multi-step Jailbreaking Privacy Attacks on ChatGPTcited by 320
  2. 2022Can Machines Help Us Answering Question 16 in Datasheets, and In Turn Reflecting on Inappropriate Content?cited by 150
  3. 2021PixMix: Dreamlike Pictures Comprehensively Improve Safety Measurescited by 131
  4. 2023Explore, Establish, Exploit: Red Teaming Language Models from Scratchcited by 111
  5. 2021Neural Path Hunter: Reducing Hallucination in Dialogue Systems via Path Groundingcited by 106
  6. 2018Adversarial Examples that Fool both Computer Vision and Time-Limited Humanscited by 73
  7. 2022Safe Reinforcement Learning by Imagining the Near Futurecited by 67
  8. 2022SafeBench: A Benchmarking Platform for Safety Evaluation of Autonomous Vehiclescited by 65
  9. 2021Handcrafted Backdoors in Deep Neural Networkscited by 57
  10. 2021What Would Jiminy Cricket Do? Towards Agents That Behave Morallycited by 57
  11. 2021Fast Minimum-norm Adversarial Attacks through Adaptive Norm Constraintscited by 50
  12. 2021How Should Pre-Trained Language Models Be Fine-Tuned Towards Adversarial Robustness?cited by 49
  13. 2021Learning Barrier Certificates: Towards Safe Reinforcement Learning with Zero Training-time Violationscited by 36
  14. 2023Query-Efficient Black-Box Red Teaming via Bayesian Optimizationcited by 31
  15. 2021Indicators of Attack Failure: Debugging and Improving Optimization of Adversarial Examplescited by 23
  16. 2022Robust Conversational Agents against Imperceptible Toxicity Triggerscited by 23
  17. 2020Safe Reinforcement Learning with Natural Language Constraintscited by 22
  18. 2022Friendly Noise against Adversarial Noise: A Powerful Defense against Data Poisoning Attackscited by 20
  19. 2022FETA: Towards Specializing Foundation Models for Expert Task Applicationscited by 13

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 7; 2024: 37; 2025: 29; 2026: 8.

81 papers.

Sort:

By area:

  1. 2023Baseline Defenses for Adversarial Attacks Against Aligned Language Modelscited by 608
  2. 2024R-Judge: Benchmarking Safety Risk Awareness for LLM Agentscited by 268
  3. 2023Ring-A-Bell! How Reliable are Concept Removal Methods for Diffusion Models?cited by 239
  4. 2024When Can LLMs Actually Correct Their Own Mistakes? A Critical Survey of Self-Correction of LLMscited by 230
  5. 2023Can LLM-Generated Misinformation Be Detected?cited by 173
  6. 2023MART: Improving LLM Safety with Multi-round Automatic Red-Teamingcited by 161
  7. 2024SAFREE: Training-Free and Adaptive Guard for Safe Text-to-Image And Video Generationcited by 125
  8. 2024Attacks, Defenses and Evaluations for LLM Conversation Safety: A Surveycited by 124
  9. 2024Making Reasoning Matter: Measuring and Improving Faithfulness of Chain-of-Thought Reasoningcited by 108
  10. 2024Curiosity-driven Red-teaming for Large Language Modelscited by 83
  11. 2024Red-Teaming for Generative AI: Silver Bullet or Security Theater?cited by 81
  12. 2024CodeAttack: Revealing Safety Generalization Challenges of Large Language Models via Code Completioncited by 70
38 more
  1. 2024UnsafeBench: Benchmarking Image Safety Classifiers on Real-World and AI-Generated Imagescited by 52
  2. 2024Confidence Matters: Revisiting Intrinsic Self-Correction Capabilities of Large Language Modelscited by 47
  3. 2023Can Language Models be Instructed to Protect Personal Information?cited by 46
  4. 2024Operationalizing a Threat Model for Red-Teaming Large Language Models (LLMs)cited by 44
  5. 2024Immune: Improving Safety Against Jailbreaks in Multi-modal LLMs via Inference-Time Alignmentcited by 42
  6. 2025The Amazon Nova Family of Models: Technical Report and Model Cardcited by 41
  7. 2024Adversarial Attacks and Defenses on Text-to-Image Diffusion Models: A Surveycited by 34
  8. 2024InfAlign: Inference-aware language model alignmentcited by 34
  9. 2024Gradient-Based Language Model Red Teamingcited by 33
  10. 2024ART: Automatic Red-teaming for Text-to-Image Models to Protect Benign Userscited by 32
  11. 2024Separable Multi-Concept Erasure from Diffusion Modelscited by 25
  12. 2024Diverse and Effective Red Teaming with Auto-generated Rewards and Multi-step Reinforcement Learningcited by 23
  13. 2025Cats Confuse Reasoning LLM: Query Agnostic Adversarial Triggers for Reasoning Modelscited by 23
  14. 2024Hallucination Detection in Foundation Models for Decision-Making: A Flexible Definition and Review of the State of the Artcited by 17
  15. 2025QueryAttack: Jailbreaking Aligned Large Language Models Using Structured Non-natural Query Languagecited by 16
  16. 2024MoGU: A Framework for Enhancing Safety of Open-Sourced LLMs While Preserving Their Usabilitycited by 14
  17. 2025Reason2Attack: Jailbreaking Text-to-Image Models via LLM Reasoningcited by 14
  18. 2024Uncertainty is Fragile: Manipulating Uncertainty in Large Language Modelscited by 13
  19. 2024Data Advisor: Dynamic Data Curation for Safety Alignment of Large Language Modelscited by 13
  20. 2025Unveiling the Basin-Like Loss Landscape in Large Language Modelscited by 13
  21. 2024Automated Progressive Red Teamingcited by 12
  22. 2025Multi-lingual Multi-turn Automated Red Teaming for LLMscited by 12
  23. 2024Ruby Teaming: Improving Quality Diversity Search with Memory for Automated Red Teamingcited by 10
  24. 2025Token-Level Constraint Boundary Search for Jailbreaking Text-to-Image Modelscited by 9
  25. 2024Large Language Models are Vulnerable to Bait-and-Switch Attacks for Generating Harmful Contentcited by 8
  26. 2024Can LLMs Deeply Detect Complex Malicious Queries? A Framework for Jailbreaking via Obfuscating Intentcited by 8
  27. 2025Red-Teaming Text-to-Image Systems by Rule-based Preference Modelingcited by 8
  28. 2024SMILES-Prompting: A Novel Approach to LLM Jailbreak Attacks in Chemical Synthesiscited by 7
  29. 2024AdvAnchor: Enhancing Diffusion Model Unlearning with Adversarial Anchorscited by 7
  30. 2025Adversarial Preference Learning for Robust LLM Alignmentcited by 7
  31. 2025RedDebate: Safer Responses Through Multi-Agent Red Teaming Debatescited by 7
  32. 2024Tiny Refinements Elicit Resilience: Toward Efficient Prefix-Model Against LLM Red-Teamingcited by 6
  33. 2025Red Teaming Contemporary AI Models: Insights from Spanish and Basque Perspectivescited by 6
  34. 2023JAB: Joint Adversarial Prompting and Belief Augmentationcited by 5
  35. 2024Towards Effective Discrimination Testing for Generative AIcited by 5
  36. 2025DREAM: Scalable Red Teaming for Text-to-Image Generative Systems via Distribution Modelingcited by 5
  37. 2025Universally Unfiltered and Unseen:Input-Agnostic Multimodal Jailbreaks against Text-to-Image Model Safeguardscited by 5
  38. 2026TriPlay-RL: Tri-Role Self-Play Reinforcement Learning for LLM Safety Alignmentcited by 5

50 most cited of 81 All 81 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Automatic Red Teaming for Implicit Vulnerabilities of Text-to-Image Models
  2. 2025RedTopic: Toward Topic-Diverse Red Teaming of Large Language Models
  3. 2023Prompting4Debugging: Red-Teaming Text-to-Image Diffusion Models by Finding Problematic Prompts
  4. 2024Against The Achilles' Heel: A Survey on Red Teaming for Generative Models
  5. 2024DiffZOO: A Purely Query-Based Black-Box Attack for Red-teaming Text-to-Image Generative Model via Zeroth Order Optimization
  6. 2026OpenRT: An Open-Source Red Teaming Framework for Multimodal LLMs
  7. 2024Automated Red Teaming with GOAT: the Generative Offensive Agent Tester
  8. 2024Ferret: Faster and Effective Automated Red Teaming with Reward-Based Scoring Technique