mlsanityData to 14 Oct 2026

CoP: Agentic Red-teaming for Large Language Models using Composition of Principles

arXiv 2506.00781 ↗First posted 1 Jun 2025cs.AIcited by 13

On the map: LLM agents and tool use → Agent threats, guardrails and governance

also in 1 more
Abstract (arXiv)

Recent advances in Large Language Models (LLMs) have spurred transformative applications in various domains, ranging from open-source to proprietary LLMs. However, jailbreak attacks, which aim to break safety alignment and user compliance by tricking the target LLMs into answering harmful and risky responses, are becoming an urgent concern. The practice of red-teaming for LLMs is to proactively explore potential risks and error-prone instances before the release of frontier AI technology. This paper proposes an agentic workflow to automate and scale the red-teaming process of LLMs through the Composition-of-Principles (CoP) framework, where human users provide a set of red-teaming principles as instructions to an AI agent to automatically orchestrate effective red-teaming strategies and generate jailbreak prompts. Distinct from existing red-teaming methods, our CoP framework provides a unified and extensible framework to encompass and orchestrate human-provided red-teaming principles to enable the automated discovery of new red-teaming strategies. When tested against leading LLMs, CoP reveals unprecedented safety risks by finding novel jailbreak prompts and improving the best-known single-turn attack success rate by up to 19.0 times.

Key facts. Cited by 13 papers in the map (data through 14 Oct 2026). Placed in Agent threats, guardrails and governance, one of 16 directions in LLM agents and tool use. Builds on 19 of its 26 references that are in the map. First posted on arXiv 1 Jun 2025.

Builds on

19 of 26 references.

Closest ancestors · 2

Same direction, newest first

  1. 2025X-Teaming: Multi-Turn Jailbreaks and Defenses with Adaptive Multi-Agents15 Apr
  2. 2024Automated Red Teaming with GOAT: the Generative Offensive Agent Tester2 Oct

Other references · 17

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2024The Llama 3 Herd of Modelscited by 14,715
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  4. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 3,104
  5. 2023Jailbroken: How Does LLM Safety Training Fail?cited by 1,651
  6. 2024HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusalcited by 1,429
  7. 2023Jailbreaking Black Box Large Language Models in Twenty Queriescited by 1,420
  8. 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,055
  9. 2023AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Modelscited by 785
  10. 2023Tree of Attacks: Jailbreaking Black-Box LLMs Automaticallycited by 672
  11. 2024How Johnny Can Persuade LLMs to Jailbreak Them: Rethinking Persuasion to Challenge AI Safety by Humanizing LLMscited by 573
  12. 2023GPTFUZZER: Red Teaming Large Language Models with Auto-Generated Jailbreak Promptscited by 560
5 more
  1. 2024JailbreakBench: An Open Robustness Benchmark for Jailbreaking Large Language Modelscited by 536
  2. 2024Improving Alignment and Robustness with Circuit Breakerscited by 322
  3. 2024Rainbow Teaming: Open-Ended Generation of Diverse Adversarial Promptscited by 205
  4. 2024AutoDAN-Turbo: A Lifelong Agent for Strategy Self-Exploration to Jailbreak LLMscited by 199
  5. 2024Preference Fine-Tuning of LLMs Should Leverage Suboptimal, On-Policy Datacited by 185

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 2; 2026: 11.

13 papers.

Sort:

By area:

  1. 2026Steering Externalities: Benign Activation Steering Unintentionally Increases Jailbreak Risk for Large Language Modelscited by 8
  2. 2025The Trojan Knowledge: Bypassing Commercial LLM Guardrails via Harmless Prompt Weaving and Adaptive Tree Searchcited by 7
  3. 2026AgenticRed: Evolving Agentic Systems for Red-Teamingcited by 5
  4. 2026RvB: Automating AI System Hardening via Iterative Red-Blue Gamescited by 3
  5. 2026CONSCIENTIA: Can LLM Agents Learn to Strategize? Emergent Deception and Trust in a Multi-Agent NYC Simulationcited by 2
  6. 2026Agent Hacks Agent: Autoresearch for Production-Agent Red-Teamingcited by 2
  7. 2026Read, Grep, and Synthesize: Diagnosing Cross-Domain Seed Exposure for LLM Research Ideationcited by 1
  8. 2025Mass-Scale Analysis of In-the-Wild Conversations Reveals Complexity Bounds on LLM Jailbreakingcited by 0
  9. 2026Persona-Conditioned Adversarial Prompting: Multi-Identity Red-Teaming for Adversarial Discovery and Mitigationcited by 0
  10. 2026Persona-Conditioned Adversarial Prompting (PCAP): Multi-Identity Red-Teaming for Enhanced Adversarial Prompt Discoverycited by 0
  11. 2026MJ: Multi-turn LLM Jailbreaking via Decomposed Credit Assignmentcited by 0
  12. 2026Adversarial Stress Testing of Role-Playing Language Agents using Multi-Agent Evaluationcited by 0
1 more
  1. 2026SIR: Self-improving Red-teaming for Compute Use Agentscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024RedAgent: Red Teaming Large Language Models with Context-aware Autonomous Language Agent
  2. 2024Recent advancements in LLM Red-Teaming: Techniques, Defenses, and Ethical Considerations
  3. 2025Jailbreak-Zero: A Path to Pareto Optimal Red Teaming for Large Language Models
  4. 2025Building Safe GenAI Applications: An End-to-End Overview of Red Teaming for Large Language Models
  5. 2024Operationalizing a Threat Model for Red-Teaming Large Language Models (LLMs)
  6. 2025Automatic LLM Red Teaming
  7. 2025RedTWIZ: Diverse LLM Red Teaming via Adaptive Attack Planning
  8. 2025Capability-Based Scaling Trends for LLM-Based Red-Teaming