mlsanityData to 7 Oct 2026

LLM-Assisted Red Teaming of Diffusion Models through "Failures Are Fated, But Can Be Faded"

arXiv 2410.16738 ↗First posted 22 Oct 2024cs.LGcited by 0

On the map: Diffusion models → Diffusion and flow policies for robotics

also in 1 more
Abstract (arXiv)

In large deep neural networks that seem to perform surprisingly well on many tasks, we also observe a few failures related to accuracy, social biases, and alignment with human values, among others. Therefore, before deploying these models, it is crucial to characterize this failure landscape for engineers to debug or audit models. Nevertheless, it is infeasible to exhaustively test for all possible combinations of factors that could lead to a model's failure. In this paper, we improve the "Failures are fated, but can be faded" framework (arXiv:2406.07145)--a post-hoc method to explore and construct the failure landscape in pre-trained generative models--with a variety of deep reinforcement learning algorithms, screening tests, and LLM-based rewards and state generation. With the aid of limited human feedback, we then demonstrate how to restructure the failure landscape to be more desirable by moving away from the discovered failure modes. We empirically demonstrate the effectiveness of the proposed method on diffusion models. We also highlight the strengths and weaknesses of each algorithm in identifying failure modes.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Diffusion and flow policies for robotics, one of 14 directions in Diffusion models. Builds on 28 of its 31 references that are in the map. First posted on arXiv 22 Oct 2024.

Builds on

28 of 31 references.

References · 28

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2021High-Resolution Image Synthesis with Latent Diffusion Modelscited by 18,694
  3. 2017Proximal Policy Optimization Algorithmscited by 14,441
  4. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  5. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,326
  6. 2017Deep reinforcement learning from human preferencescited by 3,846
  7. 2016Asynchronous Methods for Deep Reinforcement Learningcited by 2,218
  8. 2022Code as Policies: Language Model Programs for Embodied Controlcited by 1,172
  9. 2022Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learnedcited by 702
  10. 2023Language to Rewards for Robotic Skill Synthesiscited by 307
  11. 2021Exploring the Limits of Out-of-Distribution Detectioncited by 283
  12. 2016Safety Verification of Deep Neural Networkscited by 262
16 more
  1. 2023Reward Design with Language Modelscited by 257
  2. 2021Deep learning: a statistical viewpointcited by 239
  3. 2017Deep Reinforcement Learning for De-Novo Drug Designcited by 161
  4. 2022Domino: Discovering Systematic Errors with Cross-Modal Embeddingscited by 155
  5. 2022Distilling Model Failures as Directions in Latent Spacecited by 82
  6. 2020A Survey of Algorithms for Black-Box Safety Validation of Cyber-Physical Systemscited by 71
  7. 2020Opportunities and Challenges in Deep Learning Adversarial Robustness: A Surveycited by 71
  8. 2020PatchAttack: A Black-box Texture-based Attack with Reinforcement Learningcited by 67
  9. 2023LANCE: Stress-testing Visual Models by Generating Language-guided Counterfactual Imagescited by 46
  10. 2020Out-of-Distribution Detection for Automotive Perceptioncited by 37
  11. 2022Single Model Uncertainty Estimation via Stochastic Data Centeringcited by 20
  12. 2022How Do We Fail? Stress Testing Perception in Autonomous Vehiclescited by 17
  13. 2021Reinforcement Learning Based Sparse Black-box Adversarial Attack on Video Recognition Modelscited by 12
  14. 2024Failures Are Fated, But Can Be Faded: Characterizing and Mitigating Unwanted Behaviors in Large-Scale Vision and Language Modelscited by 11
  15. 2022Out of Distribution Detection via Neural Network Anchoringcited by 6
  16. 2024DECIDER: Leveraging Foundation Model Priors for Improved Model Failure Detection and Explanationcited by 6

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025FailureAtlas:Mapping the Failure Landscape of T2I Models via Active Exploration
  2. 2023Identifying and Mitigating Model Failures through Few-shot CLIP-aided Diffusion Generation
  3. 2023Diffusion-C: Unveiling the Generative Challenges of Diffusion Models through Corrupted Data
  4. 2026Where Success Breaks: Failure-Boundary Learning for Robust Vision-Language-Action Models
  5. 2026Tracing Agentic Failure from the Flow of Success
  6. 2026Accurate Failure Prediction in Agents Does Not Imply Effective Failure Prevention
  7. 2025Why Do LLM Agents Fail in Exploring New Environments? A World-Modeling Perspective
  8. 2023Discovering Failure Modes of Text-guided Diffusion Models via Adversarial Search