# AART: AI-Assisted Red-Teaming with Diverse Data Generation for New LLM-powered Applications

arXiv: https://arxiv.org/abs/2311.08592 · HTML: https://mlsanity.com/paper/2311.08592/

## Key facts

- Cited by 54 papers in the map (data through 14 Oct 2026).
- Placed in Prompt injection and agent security, one of 12 directions in LLM safety and security.
- Builds on 23 of its 37 references that are in the map.
- First posted on arXiv 14 Nov 2023.

## Placed in

- LLM safety and security / [Prompt injection and agent security](https://mlsanity.com/safety/prompt-injection-and-agent-security/)

## Builds on (most cited first, 23 of 23)

- [Self-Consistency Improves Chain of Thought Reasoning in Language Models](https://mlsanity.com/paper/2203.11171/) (2022)
- [Large Language Models are Zero-Shot Reasoners](https://mlsanity.com/paper/2205.11916/) (2022)
- [Constitutional AI: Harmlessness from AI Feedback](https://mlsanity.com/paper/2212.08073/) (2022)
- [Self-Instruct: Aligning Language Models with Self-Generated Instructions](https://mlsanity.com/paper/2212.10560/) (2022)
- [RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Models](https://mlsanity.com/paper/2009.11462/) (2020)
- [LaMDA: Language Models for Dialog Applications](https://mlsanity.com/paper/2201.08239/) (2022)
- [WizardLM: Empowering large pre-trained language models to follow complex instructions](https://mlsanity.com/paper/2304.12244/) (2023)
- [Red Teaming Language Models with Language Models](https://mlsanity.com/paper/2202.03286/) (2022)
- [Ethical and social risks of harm from Language Models](https://mlsanity.com/paper/2112.04359/) (2021)
- [Improving alignment of dialogue agents via targeted human judgements](https://mlsanity.com/paper/2209.14375/) (2022)
- [Dynabench: Rethinking Benchmarking in NLP](https://mlsanity.com/paper/2104.14337/) (2021)
- [Principle-Driven Self-Alignment of Language Models from Scratch with Minimal Human Supervision](https://mlsanity.com/paper/2305.03047/) (2023)
- [Measurement and Fairness](https://mlsanity.com/paper/1912.05511/) (2019)
- [Process for Adapting Language Models to Society (PALMS) with Values-Targeted Datasets](https://mlsanity.com/paper/2106.10328/) (2021)
- [Build it Break it Fix it for Dialogue Safety: Robustness from Adversarial Human Attack](https://mlsanity.com/paper/1908.06083/) (2019)
- [Recipes for Safety in Open-domain Chatbots](https://mlsanity.com/paper/2010.07079/) (2020)
- [Sociotechnical Harms of Algorithmic Systems: Scoping a Taxonomy for Harm Reduction](https://mlsanity.com/paper/2210.05791/) (2022)
- [Faithful Reasoning Using Large Language Models](https://mlsanity.com/paper/2208.14271/) (2022)
- [Re-contextualizing Fairness in NLP: The Case of India](https://mlsanity.com/paper/2209.12226/) (2022)
- [Designing Disaggregated Evaluations of AI Systems: Choices, Considerations, and Tradeoffs](https://mlsanity.com/paper/2103.06076/) (2021)
- [Fairness for Unobserved Characteristics: Insights from Technological Impacts on Queer Communities](https://mlsanity.com/paper/2102.04257/) (2021)
- [Open-Domain Conversational Agents: Current Progress, Open Problems, and Future Directions](https://mlsanity.com/paper/2006.12442/) (2020)
- [Flexible text generation for counterfactual fairness probing](https://mlsanity.com/paper/2206.13757/) (2022)

## Built on by (most cited first, 30 of 54)

- [ShieldGemma: Generative AI Content Moderation Based on Gemma](https://mlsanity.com/paper/2407.21772/) (2024)
- [Aegis2.0: A Diverse AI Safety Dataset and Risks Taxonomy for Alignment of LLM Guardrails](https://mlsanity.com/paper/2501.09004/) (2025)
- [Red-Teaming for Generative AI: Silver Bullet or Security Theater?](https://mlsanity.com/paper/2401.15897/) (2024)
- [SafetyPrompts: a Systematic Review of Open Datasets for Evaluating and Improving Large Language Model Safety](https://mlsanity.com/paper/2404.05399/) (2024)
- [Operationalizing a Threat Model for Red-Teaming Large Language Models (LLMs)](https://mlsanity.com/paper/2407.14937/) (2024)
- [Agentic Web: Weaving the Next Web with AI Agents](https://mlsanity.com/paper/2507.21206/) (2025)
- [CulturalTeaming: AI-Assisted Interactive Red-Teaming for Challenging LLMs' (Lack of) Multicultural Knowledge](https://mlsanity.com/paper/2404.06664/) (2024)
- [Toward Generalizable Evaluation in the LLM Era: A Survey Beyond Benchmarks](https://mlsanity.com/paper/2504.18838/) (2025)
- [RAG LLMs are Not Safer: A Safety Analysis of Retrieval-Augmented Generation for Large Language Models](https://mlsanity.com/paper/2504.18041/) (2025)
- [Self-Tuning: Instructing LLMs to Effectively Acquire New Knowledge through Self-Teaching](https://mlsanity.com/paper/2406.06326/) (2024)
- [Raising the Bar: Investigating the Values of Large Language Models via Generative Evolving Testing](https://mlsanity.com/paper/2406.14230/) (2024)
- [Attack Atlas: A Practitioner's Perspective on Challenges and Pitfalls in Red Teaming GenAI](https://mlsanity.com/paper/2409.15398/) (2024)
- [MoPS: Modular Story Premise Synthesis for Open-Ended Automatic Story Generation](https://mlsanity.com/paper/2406.05690/) (2024)
- [STAR: SocioTechnical Approach to Red Teaming Language Models](https://mlsanity.com/paper/2406.11757/) (2024)
- [Automated Progressive Red Teaming](https://mlsanity.com/paper/2407.03876/) (2024)
- [Siren: A Learning-Based Multi-Turn Attack Framework for Simulating Real-World Human Jailbreak Behaviors](https://mlsanity.com/paper/2501.14250/) (2025)
- [Large Reasoning Models Are Autonomous Jailbreak Agents](https://mlsanity.com/paper/2508.04039/) (2025)
- [LexC-Gen: Generating Data for Extremely Low-Resource Languages with Large Language Models and Bilingual Lexicons](https://mlsanity.com/paper/2402.14086/) (2024)
- [Building Safe GenAI Applications: An End-to-End Overview of Red Teaming for Large Language Models](https://mlsanity.com/paper/2503.01742/) (2025)
- [CheckEmbed: Effective Verification of LLM Solutions to Open-Ended Tasks](https://mlsanity.com/paper/2406.02524/) (2024)
- [How Are LLMs Mitigating Stereotyping Harms? Learning from Search Engine Studies](https://mlsanity.com/paper/2407.11733/) (2024)
- [PrimeGuard: Safe and Helpful LLMs through Tuning-Free Routing](https://mlsanity.com/paper/2407.16318/) (2024)
- [SAGE-RT: Synthetic Alignment data Generation for Safety Evaluation and Red Teaming](https://mlsanity.com/paper/2408.11851/) (2024)
- [Active Learning for Robust and Representative LLM Generation in Safety-Critical Scenarios](https://mlsanity.com/paper/2410.11114/) (2024)
- [Diversity Helps Jailbreak Large Language Models](https://mlsanity.com/paper/2411.04223/) (2024)
- [Adversarial Prompt Evaluation: Systematic Benchmarking of Guardrails Against Prompt Input Attacks on LLMs](https://mlsanity.com/paper/2502.15427/) (2025)
- [A Survey on Responsible Generative AI: What to Generate and What Not](https://mlsanity.com/paper/2404.05783/) (2024)
- [SAGE: A Generic Framework for LLM Safety Evaluation](https://mlsanity.com/paper/2504.19674/) (2025)
- [Learning to Conceal Risk: Controllable Multi-turn Red Teaming for LLMs in the Financial Domain](https://mlsanity.com/paper/2509.10546/) (2025)
- [CoBia: Constructed Conversations Can Trigger Otherwise Concealed Societal Biases in LLMs](https://mlsanity.com/paper/2510.09871/) (2025)

## Abstract (arXiv)

Adversarial testing of large language models (LLMs) is crucial for their safe and responsible deployment. We introduce a novel approach for automated generation of adversarial evaluation datasets to test the safety of LLM generations on new downstream applications. We call it AI-assisted Red-Teaming (AART) - an automated alternative to current manual red-teaming efforts. AART offers a data generation and augmentation pipeline of reusable and customizable recipes that reduce human effort significantly and enable integration of adversarial testing earlier in new product development. AART generates evaluation datasets with high diversity of content characteristics critical for effective adversarial testing (e.g. sensitive and harmful concepts, specific to a wide range of cultural and geographic regions and application scenarios). The data generation is steered by AI-assisted recipes to define, scope and prioritize diversity within the application context. This feeds into a structured LLM-generation process that scales up evaluation priorities. Compared to some state-of-the-art tools, AART shows promising results in terms of concept coverage and data quality.