mlsanityData to 7 Oct 2026

TRACE: Task-Aware Adaptive Self-Evolving Agentic Jailbreaking

arXiv 2605.30883 ↗First posted 29 May 2026cs.CRcited by 4

On the map: LLM agents and tool use → Agent threats, guardrails and governance

also in 1 more
Abstract (arXiv)

The rise of LLM agents introduces a new threat by enabling planning, coding, and even end-to-end execution of expert-level attack workflows. However, this threat remains underexplored and underestimated since (i) safety alignment prevents LLMs from directly generating harmful instructions, and (ii) most existing jailbreak methods cannot consistently induce agents to execute malicious operations. In this paper, we propose TRACE, a practical agentic jailbreaking framework to further reveal the risks of this threat surface. To conceal the malicious intent, TRACE decomposes a malicious task into multiple subtask sequences under different schemes and selects the sequence with the fewest explicitly harmful subtasks. TRACE then disguises the remaining harmful subtasks as benign-looking instructions by embedding them in task-aware scenarios with related roles, environments, directives, and heuristics. The scenarios are iteratively evolved through well-defined transformation actions, which are sampled by a Q-learning-inspired mechanism, for inducing the agent to execute on the harmful subtasks. Extensive evaluations on AgentHarm and AdvCUA show that TRACE consistently outperforms existing jailbreak baselines across multiple advanced LLM agents, achieving up to 100% bypass rate and 0.73 average success score. We also demonstrate the effectiveness of TRACE in controlled cyberattack instances. Our code and demos are available at https://github.com/ZJU-LLM-Safety/TRACE.git.

Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Placed in Agent threats, guardrails and governance, one of 16 directions in LLM agents and tool use. Builds on 44 of its 63 references that are in the map. First posted on arXiv 29 May 2026.

Builds on

44 of 63 references.

Closest ancestors · 7

Same direction, newest first

  1. 2026AgentHazard: A Benchmark for Evaluating Harmful Behavior in Computer-Use Agents3 Apr
  2. 2026Helpful to a Fault: Measuring Illicit Assistance in Multi-Turn, Multilingual LLM Agents18 Feb
  3. 2026Unsafer in Many Turns: Benchmarking and Defending Multi-Turn Safety Risks in Tool-Using Agents13 Feb
  4. 2026AgentDoG: A Diagnostic Guardrail Framework for AI Agent Safety and Security26 Jan
  5. 2025X-Teaming: Multi-Turn Jailbreaks and Defenses with Adaptive Multi-Agents15 Apr
  6. 2024AgentHarm: A Benchmark for Measuring Harmfulness of LLM Agents11 Oct
  7. 2024LLM Agents can Autonomously Hack Websites6 Feb

Other references · 37

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  3. 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
  4. 2024Qwen2.5 Technical Reportcited by 4,713
  5. 2024GPT-4o System Cardcited by 4,467
  6. 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,072
  7. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  8. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 2,914
  9. 2023A Survey on Large Language Model based Autonomous Agentscited by 2,203
  10. 2023WebArena: A Realistic Web Environment for Building Autonomous Agentscited by 1,908
  11. 2024SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineeringcited by 1,628
  12. 2023Jailbreaking Black Box Large Language Models in Twenty Queriescited by 1,326
25 more
  1. 2023Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!cited by 1,106
  2. 2026DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligencecited by 877
  3. 2023AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Modelscited by 738
  4. 2023Tree of Attacks: Jailbreaking Black-Box LLMs Automaticallycited by 619
  5. 2023Baseline Defenses for Adversarial Attacks Against Aligned Language Modelscited by 608
  6. 2023C-Pack: Packed Resources For General Chinese Embeddingscited by 556
  7. 2024How Johnny Can Persuade LLMs to Jailbreak Them: Rethinking Persuasion to Challenge AI Safety by Humanizing LLMscited by 533
  8. 2023Augmented Language Models: a Surveycited by 398
  9. 2024Great, Now Write an Article About That: The Crescendo Multi-Turn LLM Jailbreak Attackcited by 364
  10. 2023DeepInception: Hypnotize Large Language Model to Be Jailbreakercited by 337
  11. 2024TextGrad: Automatic "Differentiation" via Textcited by 250
  12. 2023A Wolf in Sheep's Clothing: Generalized Nested Jailbreak Prompts can Fool Large Language Models Easilycited by 242
  13. 2024ArtPrompt: ASCII Art-based Jailbreak Attacks against Aligned LLMscited by 197
  14. 2024AutoDAN-Turbo: A Lifelong Agent for Strategy Self-Exploration to Jailbreak LLMscited by 185
  15. 2024COLD-Attack: Jailbreaking LLMs with Stealthiness and Controllabilitycited by 162
  16. 2024When LLM Meets DRL: Advancing Jailbreaking Efficiency via DRL-guided Searchcited by 42
  17. 2018Distributionally Adversarial Attackcited by 24
  18. 2025Monitoring Decomposition Attacks in LLMs with Lightweight Sequential Monitorscited by 21
  19. 2025HarmMetric Eval: Benchmarking Metrics and Judges for LLM Harmfulness Assessmentcited by 12
  20. 2025MAJIC: Markovian Adaptive Jailbreaking via Iterative Composition of Diverse Innovative Strategiescited by 11
  21. 2025TROJail: Trajectory-Level Optimization for Multi-Turn Large Language Model Jailbreaks with Process Rewardscited by 11
  22. 2025DualBreach: Efficient Dual-Jailbreaking via Target-Driven Initialization and Multi-Target Optimizationcited by 9
  23. 2025MUSE: MCTS-Driven Red Teaming Framework for Enhanced Multi-Turn Dialogue Safety in Large Language Modelscited by 4
  24. 2025Dynamic Jailbreaking Attackcited by 3
  25. 2025Jailbreaking Commercial Black-Box LLMs with Explicitly Harmful Promptscited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 4.

4 papers.

Similar, not cited

Close in wording, no citation either way

  1. 2026A Self-Evolving Multi-Agent Framework Defense against LLM Jailbreak Attacks
  2. 2026SoK: Rethinking Jailbreaking in the Era of Agentic AI: Attacks, Defenses, and Practical Consideration
  3. 2026Proteus: A Self-Evolving Red Team for Agent Skill Ecosystems
  4. 2026LITMUS: Benchmarking Behavioral Jailbreaks of LLM Agents in Real OS Environments
  5. 2026LoopTrap: Termination Poisoning Attacks on LLM Agents
  6. 2026EVOMAL: Self-Poisoning in Self-Evolving Coding Agents
  7. 2025Breaking the Code: Security Assessment of AI Code Agents Through Systematic Jailbreaking Attacks
  8. 2026A Systematic Survey of Security Threats and Defenses in LLM-Based AI Agents: A Layered Attack Surface Framework