TRACE: Task-Aware Adaptive Self-Evolving Agentic Jailbreaking
On the map: LLM agents and tool use → Agent threats, guardrails and governance
also in 1 more
Abstract (arXiv)
The rise of LLM agents introduces a new threat by enabling planning, coding, and even end-to-end execution of expert-level attack workflows. However, this threat remains underexplored and underestimated since (i) safety alignment prevents LLMs from directly generating harmful instructions, and (ii) most existing jailbreak methods cannot consistently induce agents to execute malicious operations. In this paper, we propose TRACE, a practical agentic jailbreaking framework to further reveal the risks of this threat surface. To conceal the malicious intent, TRACE decomposes a malicious task into multiple subtask sequences under different schemes and selects the sequence with the fewest explicitly harmful subtasks. TRACE then disguises the remaining harmful subtasks as benign-looking instructions by embedding them in task-aware scenarios with related roles, environments, directives, and heuristics. The scenarios are iteratively evolved through well-defined transformation actions, which are sampled by a Q-learning-inspired mechanism, for inducing the agent to execute on the harmful subtasks. Extensive evaluations on AgentHarm and AdvCUA show that TRACE consistently outperforms existing jailbreak baselines across multiple advanced LLM agents, achieving up to 100% bypass rate and 0.73 average success score. We also demonstrate the effectiveness of TRACE in controlled cyberattack instances. Our code and demos are available at https://github.com/ZJU-LLM-Safety/TRACE.git.
Key facts. Cited by 4 papers in the map (data through 7 Oct 2026). Placed in Agent threats, guardrails and governance, one of 16 directions in LLM agents and tool use. Builds on 44 of its 63 references that are in the map. First posted on arXiv 29 May 2026.
Builds on
44 of 63 references.
Closest ancestors · 7
Same direction, newest first
- 2026AgentHazard: A Benchmark for Evaluating Harmful Behavior in Computer-Use Agents3 Apr
- 2026Helpful to a Fault: Measuring Illicit Assistance in Multi-Turn, Multilingual LLM Agents18 Feb
- 2026Unsafer in Many Turns: Benchmarking and Defending Multi-Turn Safety Risks in Tool-Using Agents13 Feb
- 2026AgentDoG: A Diagnostic Guardrail Framework for AI Agent Safety and Security26 Jan
- 2025X-Teaming: Multi-Turn Jailbreaks and Defenses with Adaptive Multi-Agents15 Apr
- 2024AgentHarm: A Benchmark for Measuring Harmfulness of LLM Agents11 Oct
- 2024LLM Agents can Autonomously Hack Websites6 Feb
Other references · 37
- 2022Training language models to follow instructions with human feedbackcited by 15,712
- 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
- 2022ReAct: Synergizing Reasoning and Acting in Language Modelscited by 8,355
- 2024Qwen2.5 Technical Reportcited by 4,713
- 2024GPT-4o System Cardcited by 4,467
- 2023Toolformer: Language Models Can Teach Themselves to Use Toolscited by 4,072
- 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
- 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 2,914
- 2023A Survey on Large Language Model based Autonomous Agentscited by 2,203
- 2023WebArena: A Realistic Web Environment for Building Autonomous Agentscited by 1,908
- 2024SWE-agent: Agent-Computer Interfaces Enable Automated Software Engineeringcited by 1,628
- 2023Jailbreaking Black Box Large Language Models in Twenty Queriescited by 1,326
25 more
- 2023Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!cited by 1,106
- 2026DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligencecited by 877
- 2023AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Modelscited by 738
- 2023Tree of Attacks: Jailbreaking Black-Box LLMs Automaticallycited by 619
- 2023Baseline Defenses for Adversarial Attacks Against Aligned Language Modelscited by 608
- 2023C-Pack: Packed Resources For General Chinese Embeddingscited by 556
- 2024How Johnny Can Persuade LLMs to Jailbreak Them: Rethinking Persuasion to Challenge AI Safety by Humanizing LLMscited by 533
- 2023Augmented Language Models: a Surveycited by 398
- 2024Great, Now Write an Article About That: The Crescendo Multi-Turn LLM Jailbreak Attackcited by 364
- 2023DeepInception: Hypnotize Large Language Model to Be Jailbreakercited by 337
- 2024TextGrad: Automatic "Differentiation" via Textcited by 250
- 2023A Wolf in Sheep's Clothing: Generalized Nested Jailbreak Prompts can Fool Large Language Models Easilycited by 242
- 2024ArtPrompt: ASCII Art-based Jailbreak Attacks against Aligned LLMscited by 197
- 2024AutoDAN-Turbo: A Lifelong Agent for Strategy Self-Exploration to Jailbreak LLMscited by 185
- 2024COLD-Attack: Jailbreaking LLMs with Stealthiness and Controllabilitycited by 162
- 2024When LLM Meets DRL: Advancing Jailbreaking Efficiency via DRL-guided Searchcited by 42
- 2018Distributionally Adversarial Attackcited by 24
- 2025Monitoring Decomposition Attacks in LLMs with Lightweight Sequential Monitorscited by 21
- 2025HarmMetric Eval: Benchmarking Metrics and Judges for LLM Harmfulness Assessmentcited by 12
- 2025MAJIC: Markovian Adaptive Jailbreaking via Iterative Composition of Diverse Innovative Strategiescited by 11
- 2025TROJail: Trajectory-Level Optimization for Multi-Turn Large Language Model Jailbreaks with Process Rewardscited by 11
- 2025DualBreach: Efficient Dual-Jailbreaking via Target-Driven Initialization and Multi-Target Optimizationcited by 9
- 2025MUSE: MCTS-Driven Red Teaming Framework for Enhanced Multi-Turn Dialogue Safety in Large Language Modelscited by 4
- 2025Dynamic Jailbreaking Attackcited by 3
- 2025Jailbreaking Commercial Black-Box LLMs with Explicitly Harmful Promptscited by 1
Built on by
Citing papers per year: 2026: 4.
4 papers.
- 2026Safin-1: Safety from Within through Memory-Native State Evolutioncited by 1
- 2026Gradient Immunity: Null-Space Resistance to Malicious Fine-Tuningcited by 0
- 2026SEAD: A State-Based Perspective on Attack and Defense in Tool-Using Agentscited by 0
- 2026ReSI: Recursive Safety Improvement toward Resistant and Resilient AIcited by 0
- 2026ReSI: Recursive Safety Improvement toward Resistant and Resilient AI8 Oct 2026
- 2026SEAD: A State-Based Perspective on Attack and Defense in Tool-Using Agents28 Sep 2026
- 2026Safin-1: Safety from Within through Memory-Native State Evolution31 Aug 2026
- 2026Gradient Immunity: Null-Space Resistance to Malicious Fine-Tuning5 Aug 2026
Similar, not cited
Close in wording, no citation either way
- 2026A Self-Evolving Multi-Agent Framework Defense against LLM Jailbreak Attacks
- 2026SoK: Rethinking Jailbreaking in the Era of Agentic AI: Attacks, Defenses, and Practical Consideration
- 2026Proteus: A Self-Evolving Red Team for Agent Skill Ecosystems
- 2026LITMUS: Benchmarking Behavioral Jailbreaks of LLM Agents in Real OS Environments
- 2026LoopTrap: Termination Poisoning Attacks on LLM Agents
- 2026EVOMAL: Self-Poisoning in Self-Evolving Coding Agents
- 2025Breaking the Code: Security Assessment of AI Code Agents Through Systematic Jailbreaking Attacks
- 2026A Systematic Survey of Security Threats and Defenses in LLM-Based AI Agents: A Layered Attack Surface Framework