Not All Tokens Are Created Equal: Query-Efficient Jailbreak Fuzzing for LLMs
On the map: LLM safety and security → Jailbreaking
Abstract (arXiv)
Large Language Models(LLMs) are widely deployed, yet are vulnerable to jailbreak prompts that elicit policy-violating outputs. Although prior studies have uncovered these risks, they typically treat all tokens as equally important during prompt mutation, overlooking the varying contributions of individual tokens to triggering model refusals. Consequently, these attacks introduce substantial redundant searching under query-constrained scenarios, reducing attack efficiency and hindering comprehensive vulnerability assessment. In this work, we conduct a token-level analysis of refusal behavior and observe that token contributions are highly skewed rather than uniform. Moreover, we find strong cross-model consistency in refusal tendencies, enabling the use of a surrogate model to estimate token-level contributions to the target model's refusals. Motivated by these findings, we propose TriageFuzz, a token-aware jailbreak fuzzing framework that adapts the fuzz testing approach with a series of customized designs. TriageFuzz leverages a surrogate model to estimate the contribution of individual tokens to refusal behaviors, enabling the identification of sensitive regions within the prompt. Furthermore, it incorporates a refusal-guided evolutionary strategy that adaptively weights candidate prompts with a lightweight scorer to steer the evolution toward bypassing safety constraints. Extensive experiments on six open-source LLMs and three commercial APIs demonstrate that TriageFuzz achieves comparable attack success rates (ASR) with significantly reduced query costs. Notably, it attains a 90% ASR with over 70% fewer queries compared to baselines. Even under an extremely restrictive budget of 25 queries, TriageFuzz outperforms existing methods, improving ASR by 20-40%.
Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Jailbreaking, one of 12 directions in LLM safety and security. Builds on 43 of its 51 references that are in the map. First posted on arXiv 24 Mar 2026.
Builds on
43 of 51 references.
Closest ancestors · 21
Same direction, newest first
- 2025TurboFuzzLLM: Turbocharging Mutation-based Fuzzing for Effectively Jailbreaking Large Language Models in Practice21 Feb
- 2024Exploiting the Index Gradients for Optimization-Based Jailbreaking on Large Language Models11 Dec
- 2024Jailbreak Attacks and Defenses Against Large Language Models: A Survey5 Jul
- 2024AmpleGCG: Learning a Universal and Transferable Generative Model of Adversarial Suffixes for Jailbreaking Both Open and Closed LLMs11 Apr
- 2024JailbreakBench: An Open Robustness Benchmark for Jailbreaking Large Language Models28 Mar
- 2024Semantic Mirror Jailbreak: Genetic Algorithm Based Jailbreak Prompts Against Open-source LLMs21 Feb
- 2024COLD-Attack: Jailbreaking LLMs with Stealthiness and Controllability13 Feb
- 2024Weak-to-Strong Jailbreaking on Large Language Models30 Jan
- 2024How Johnny Can Persuade LLMs to Jailbreak Them: Rethinking Persuasion to Challenge AI Safety by Humanizing LLMs12 Jan
- 2023Tree of Attacks: Jailbreaking Black-Box LLMs Automatically4 Dec
- 2023Jailbreaking Black Box Large Language Models in Twenty Queries12 Oct
- 2023SmoothLLM: Defending Large Language Models Against Jailbreaking Attacks5 Oct
9 more
- 2023AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Models3 Oct
- 2023GPTFUZZER: Red Teaming Large Language Models with Auto-Generated Jailbreak Prompts19 Sep
- 2023FuzzLLM: A Novel and Universal Fuzzing Framework for Proactively Discovering Jailbreak Vulnerabilities in Large Language Models11 Sep
- 2023Open Sesame! Universal Black Box Jailbreaking of Large Language Models4 Sep
- 2023Detecting Language Model Attacks with Perplexity27 Aug
- 2023GPT-4 Is Too Smart To Be Safe: Stealthy Chat with LLMs via Cipher12 Aug
- 2023"Do Anything Now": Characterizing and Evaluating In-The-Wild Jailbreak Prompts on Large Language Models7 Aug
- 2023Jailbroken: How Does LLM Safety Training Fail?5 Jul
- 2023Jailbreaking ChatGPT via Prompt Engineering: An Empirical Study23 May
Other references · 22
- 2017Attention Is All You Needcited by 60,361
- 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
- 2020Language Models are Few-Shot Learnerscited by 32,419
- 2023GPT-4 Technical Reportcited by 18,396
- 2024The Llama 3 Herd of Modelscited by 14,715
- 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
- 2024GPT-4o System Cardcited by 4,588
- 2023Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injectioncited by 1,443
- 2024HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusalcited by 1,429
- 2020Transformer Feed-Forward Layers Are Key-Value Memoriescited by 1,336
- 2023Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversationscited by 1,207
- 2019BERT Rediscovers the Classical NLP Pipelinecited by 1,059
10 more
- 2024Refusal in Language Models Is Mediated by a Single Directioncited by 1,032
- 2019What Does BERT Look At? An Analysis of BERT's Attentioncited by 1,003
- 2019Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Prunedcited by 856
- 2022Ignore Previous Prompt: Attack Techniques For Language Modelscited by 832
- 2019Universal Adversarial Triggers for Attacking and Analyzing NLPcited by 722
- 2019Revealing the Dark Secrets of BERTcited by 301
- 2023Exploiting Programmatic Behavior of LLMs: Dual-Use Through Standard Security Attackscited by 278
- 2024SALAD-Bench: A Hierarchical and Comprehensive Safety Benchmark for Large Language Modelscited by 228
- 2017What do Neural Machine Translation Models Learn about Morphology?cited by 166
- 2025Using Mechanistic Interpretability to Craft Adversarial Attacks against Large Language Modelscited by 10
Built on by
Citing papers per year: 2026: 1.
1 paper.
Similar, not cited
Close in wording, no citation either way
- 2026TAO-Attack: Toward Advanced Optimization-Based Jailbreak Attacks for Large Language Models
- 2024Towards Understanding Jailbreak Attacks in LLMs: A Representation Space Analysis
- 2023Tricking LLMs into Disobedience: Formalizing, Analyzing, and Detecting Jailbreaks
- 2024JailPO: A Novel Black-box Jailbreak Framework via Preference Optimization against Aligned LLMs
- 2024Mind the Inconspicuous: Revealing the Hidden Weakness in Aligned LLMs' Refusal Boundaries
- 2026Analysis of LLMs Against Prompt Injection and Jailbreak Attacks
- 2024PAPILLON: Efficient and Stealthy Fuzz Testing-Powered Jailbreaks for LLMs
- 2026Why Do Aligned LLMs Remain Jailbreakable: Refusal-Escape Directions, Operator-Level Sources, and Safety-Utility Trade-off