mlsanityData to 7 Oct 2026

AART: AI-Assisted Red-Teaming with Diverse Data Generation for New LLM-powered Applications

arXiv 2311.08592 ↗First posted 14 Nov 2023cs.SE cs.AI cs.CLcited by 53

On the map: LLM safety and security → Prompt injection and agent security

Abstract (arXiv)

Adversarial testing of large language models (LLMs) is crucial for their safe and responsible deployment. We introduce a novel approach for automated generation of adversarial evaluation datasets to test the safety of LLM generations on new downstream applications. We call it AI-assisted Red-Teaming (AART) - an automated alternative to current manual red-teaming efforts. AART offers a data generation and augmentation pipeline of reusable and customizable recipes that reduce human effort significantly and enable integration of adversarial testing earlier in new product development. AART generates evaluation datasets with high diversity of content characteristics critical for effective adversarial testing (e.g. sensitive and harmful concepts, specific to a wide range of cultural and geographic regions and application scenarios). The data generation is steered by AI-assisted recipes to define, scope and prioritize diversity within the application context. This feeds into a structured LLM-generation process that scales up evaluation priorities. Compared to some state-of-the-art tools, AART shows promising results in terms of concept coverage and data quality.

Key facts. Cited by 53 papers in the map (data through 7 Oct 2026). Placed in Prompt injection and agent security, one of 12 directions in LLM safety and security. Builds on 23 of its 37 references that are in the map. First posted on arXiv 14 Nov 2023.

Builds on

23 of 37 references.

Closest ancestors · 1

Same direction, newest first

  1. 2022Red Teaming Language Models with Language Models7 Feb

Other references · 22

  1. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  2. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  3. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  4. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
  5. 2020RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Modelscited by 1,235
  6. 2022LaMDA: Language Models for Dialog Applicationscited by 1,218
  7. 2023WizardLM: Empowering large pre-trained language models to follow complex instructionscited by 1,116
  8. 2021Ethical and social risks of harm from Language Modelscited by 957
  9. 2022Improving alignment of dialogue agents via targeted human judgementscited by 527
  10. 2021Dynabench: Rethinking Benchmarking in NLPcited by 447
  11. 2023Principle-Driven Self-Alignment of Language Models from Scratch with Minimal Human Supervisioncited by 318
  12. 2019Measurement and Fairnesscited by 299
10 more
  1. 2021Process for Adapting Language Models to Society (PALMS) with Values-Targeted Datasetscited by 179
  2. 2019Build it Break it Fix it for Dialogue Safety: Robustness from Adversarial Human Attackcited by 176
  3. 2020Recipes for Safety in Open-domain Chatbotscited by 168
  4. 2022Sociotechnical Harms of Algorithmic Systems: Scoping a Taxonomy for Harm Reductioncited by 137
  5. 2022Faithful Reasoning Using Large Language Modelscited by 130
  6. 2022Re-contextualizing Fairness in NLP: The Case of Indiacited by 55
  7. 2021Designing Disaggregated Evaluations of AI Systems: Choices, Considerations, and Tradeoffscited by 48
  8. 2021Fairness for Unobserved Characteristics: Insights from Technological Impacts on Queer Communitiescited by 40
  9. 2020Open-Domain Conversational Agents: Current Progress, Open Problems, and Future Directionscited by 34
  10. 2022Flexible text generation for counterfactual fairness probingcited by 13

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 23; 2025: 21; 2026: 9.

53 papers.

Sort:

By area:

  1. 2024ShieldGemma: Generative AI Content Moderation Based on Gemmacited by 224
  2. 2025Aegis2.0: A Diverse AI Safety Dataset and Risks Taxonomy for Alignment of LLM Guardrailscited by 122
  3. 2024Red-Teaming for Generative AI: Silver Bullet or Security Theater?cited by 81
  4. 2024SafetyPrompts: a Systematic Review of Open Datasets for Evaluating and Improving Large Language Model Safetycited by 75
  5. 2024Operationalizing a Threat Model for Red-Teaming Large Language Models (LLMs)cited by 44
  6. 2025Agentic Web: Weaving the Next Web with AI Agentscited by 43
  7. 2024CulturalTeaming: AI-Assisted Interactive Red-Teaming for Challenging LLMs' (Lack of) Multicultural Knowledgecited by 28
  8. 2025Toward Generalizable Evaluation in the LLM Era: A Survey Beyond Benchmarkscited by 26
  9. 2025RAG LLMs are Not Safer: A Safety Analysis of Retrieval-Augmented Generation for Large Language Modelscited by 23
  10. 2024Self-Tuning: Instructing LLMs to Effectively Acquire New Knowledge through Self-Teachingcited by 17
  11. 2024Raising the Bar: Investigating the Values of Large Language Models via Generative Evolving Testingcited by 17
  12. 2024Attack Atlas: A Practitioner's Perspective on Challenges and Pitfalls in Red Teaming GenAIcited by 14
38 more
  1. 2024MoPS: Modular Story Premise Synthesis for Open-Ended Automatic Story Generationcited by 13
  2. 2024STAR: SocioTechnical Approach to Red Teaming Language Modelscited by 13
  3. 2024Automated Progressive Red Teamingcited by 12
  4. 2025Siren: A Learning-Based Multi-Turn Attack Framework for Simulating Real-World Human Jailbreak Behaviorscited by 12
  5. 2025Large Reasoning Models Are Autonomous Jailbreak Agentscited by 12
  6. 2024LexC-Gen: Generating Data for Extremely Low-Resource Languages with Large Language Models and Bilingual Lexiconscited by 11
  7. 2025Building Safe GenAI Applications: An End-to-End Overview of Red Teaming for Large Language Modelscited by 11
  8. 2024CheckEmbed: Effective Verification of LLM Solutions to Open-Ended Taskscited by 10
  9. 2024PrimeGuard: Safe and Helpful LLMs through Tuning-Free Routingcited by 10
  10. 2024SAGE-RT: Synthetic Alignment data Generation for Safety Evaluation and Red Teamingcited by 10
  11. 2024Active Learning for Robust and Representative LLM Generation in Safety-Critical Scenarioscited by 10
  12. 2024How Are LLMs Mitigating Stereotyping Harms? Learning from Search Engine Studiescited by 9
  13. 2024Diversity Helps Jailbreak Large Language Modelscited by 9
  14. 2024A Survey on Responsible Generative AI: What to Generate and What Notcited by 8
  15. 2025Adversarial Prompt Evaluation: Systematic Benchmarking of Guardrails Against Prompt Input Attacks on LLMscited by 7
  16. 2025SAGE: A Generic Framework for LLM Safety Evaluationcited by 7
  17. 2025Learning to Conceal Risk: Controllable Multi-turn Red Teaming for LLMs in the Financial Domaincited by 6
  18. 2025CoBia: Constructed Conversations Can Trigger Otherwise Concealed Societal Biases in LLMscited by 5
  19. 2025TRIDENT: Enhancing Large Language Model Safety with Tri-Dimensional Diversified Red-Teaming Data Synthesiscited by 4
  20. 2024Automated Adversarial Discovery for Safety Classifierscited by 3
  21. 2025RAG Makes Guardrails Unsafe? Investigating Robustness of Guardrails under RAG-style Contextscited by 3
  22. 2024Debiasing Text Safety Classifiers through a Fairness-Aware Ensemblecited by 2
  23. 2024Libra-Leaderboard: Towards Responsible AI through a Balanced Leaderboard of Safety and Capabilitycited by 2
  24. 2025Libra: Large Chinese-based Safeguard for AI Contentcited by 2
  25. 2025Anecdoctoring: Automated Red-Teaming Across Language and Placecited by 2
  26. 2025How Catastrophic is Your LLM? Certifying Risk in Conversationcited by 2
  27. 2026ExpGuard: LLM Content Moderation in Specialized Domainscited by 2
  28. 2024Guardrails for avoiding harmful medical product recommendations and off-label promotion in generative AI modelscited by 1
  29. 2026From Secure Agentic AI to Secure Agentic Web: Challenges, Threats, and Future Directionscited by 1
  30. 2026Going PLACES: Participatory Localized Red Teaming for Text-to-Image Safety in the Global Southcited by 1
  31. 2025RedTopic: Toward Topic-Diverse Red Teaming of Large Language Modelscited by 0
  32. 2025From Seed to Harvest: Augmenting Human Creativity with AI for Red-teaming Text-to-Image Modelscited by 0
  33. 2025GRAID: Synthetic Data Generation with Geometric Constraints and Multi-Agentic Reflection for Harmful Content Detectioncited by 0
  34. 2025Responsible AI Technical Reportcited by 0
  35. 2025Telling Speculative Stories to Help Humans Imagine the Harms of Healthcare AIcited by 0
  36. 2026BARRED: Synthetic Training of Custom Policy Guardrails via Asymmetric Debatecited by 0
  37. 2026NodeSynth: Socially Aligned Synthetic Data for AI Evaluationcited by 0
  38. 2026ESLD (External Surrogate Latent Defense): A Latent-Space Architecture for Faster, Stronger Prompt-Injection Defensecited by 0

50 most cited of 53 All 53 (JSON)