mlsanityData to 7 Oct 2026

AdvPrefix: An Objective for Nuanced LLM Jailbreaks

arXiv 2412.10321 ↗First posted 13 Dec 2024cs.LG cs.AI cs.CLcited by 25

On the map: LLM safety and security → Jailbreaking

Abstract (arXiv)

Many jailbreak attacks on large language models (LLMs) rely on a common objective: making the model respond with the prefix ``Sure, here is (harmful request)''. While straightforward, this objective has two limitations: limited control over model behaviors, yielding incomplete or unrealistic jailbroken responses, and a rigid format that hinders optimization. We introduce AdvPrefix, a plug-and-play prefix-forcing objective that selects one or more model-dependent prefixes by combining two criteria: high prefilling attack success rates and low negative log-likelihood. AdvPrefix integrates seamlessly into existing jailbreak attacks to mitigate the previous limitations for free. For example, replacing GCG's default prefixes on Llama-3 improves nuanced attack success rates from 14% to 80%, revealing that current safety alignment fails to generalize to new prefixes. Code and selected prefixes are released at github.com/facebookresearch/jailbreak-objectives.

Key facts. Cited by 25 papers in the map (data through 7 Oct 2026). Placed in Jailbreaking, one of 12 directions in LLM safety and security. Builds on 55 of its 72 references that are in the map. First posted on arXiv 13 Dec 2024.

Builds on

55 of 72 references.

Closest ancestors · 23

Same direction, newest first

  1. 2024Advancing Adversarial Suffix Transfer Learning on Aligned Large Language Models27 Aug
  2. 2024Jailbreaking as a Reward Misspecification Problem20 Jun
  3. 2024Improved Few-Shot Jailbreaking Can Circumvent Aligned Language Models and Their Defenses3 Jun
  4. 2024Improved Techniques for Optimization-Based Jailbreaking on Large Language Models31 May
  5. 2024Don't Say No: Jailbreaking LLM by Suppressing Refusal25 Apr
  6. 2024AdvPrompter: Fast Adaptive Adversarial Prompting for LLMs21 Apr
  7. 2024AmpleGCG: Learning a Universal and Transferable Generative Model of Adversarial Suffixes for Jailbreaking Both Open and Closed LLMs11 Apr
  8. 2024Jailbreaking Leading Safety-Aligned LLMs with Simple Adaptive Attacks2 Apr
  9. 2024Coercing LLMs to do and reveal (almost) anything21 Feb
  10. 2024A StrongREJECT for Empty Jailbreaks15 Feb
  11. 2024Attacking Large Language Models with Projected Gradient Descent14 Feb
  12. 2024COLD-Attack: Jailbreaking LLMs with Stealthiness and Controllability13 Feb
11 more
  1. 2024Weak-to-Strong Jailbreaking on Large Language Models30 Jan
  2. 2024How Johnny Can Persuade LLMs to Jailbreak Them: Rethinking Persuasion to Challenge AI Safety by Humanizing LLMs12 Jan
  3. 2023Tree of Attacks: Jailbreaking Black-Box LLMs Automatically4 Dec
  4. 2023AutoDAN: Interpretable Gradient-Based Adversarial Attacks on Large Language Models23 Oct
  5. 2023Jailbreaking Black Box Large Language Models in Twenty Queries12 Oct
  6. 2023Catastrophic Jailbreak of Open-source LLMs via Exploiting Generation10 Oct
  7. 2023Jailbreak and Guard Aligned Language Models with Only Few In-Context Demonstrations10 Oct
  8. 2023AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Models3 Oct
  9. 2023Open Sesame! Universal Black Box Jailbreaking of Large Language Models4 Sep
  10. 2023Universal and Transferable Adversarial Attacks on Aligned Language Models27 Jul
  11. 2023Jailbroken: How Does LLM Safety Training Fail?5 Jul

Other references · 32

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2024The Llama 3 Herd of Modelscited by 14,192
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  5. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  6. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  7. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
  8. 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,928
  9. 2016Concrete Problems in AI Safetycited by 1,582
  10. 2024HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusalcited by 1,330
  11. 2023Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversationscited by 1,124
  12. 2022Red Teaming Language Models with Language Modelscited by 983
20 more
  1. 2023BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Datasetcited by 828
  2. 2024Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Headscited by 821
  3. 2015Scheduled Sampling for Sequence Prediction with Recurrent Neural Networkscited by 779
  4. 2022Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learnedcited by 702
  5. 2023Safe RLHF: Safe Reinforcement Learning from Human Feedbackcited by 630
  6. 2023Baseline Defenses for Adversarial Attacks Against Aligned Language Modelscited by 608
  7. 2023Prompt Injection attack against LLM-integrated Applicationscited by 568
  8. 2024Safety Alignment Should Be Made More Than Just a Few Tokens Deepcited by 450
  9. 2024Better & Faster Large Language Models via Multi-token Predictioncited by 344
  10. 2024TrustLLM: Trustworthiness in Large Language Modelscited by 280
  11. 2021Gradient-based Adversarial Attacks against Text Transformerscited by 233
  12. 2024ShieldGemma: Generative AI Content Moderation Based on Gemmacited by 224
  13. 2024OR-Bench: An Over-Refusal Benchmark for Large Language Modelscited by 222
  14. 2023Future Lens: Anticipating Subsequent Tokens from a Single Hidden Statecited by 116
  15. 2022Why Exposure Bias Matters: An Imitation Learning Perspective of Error Accumulation in Language Generationcited by 106
  16. 2024Introducing v0.5 of the AI Safety Benchmark from MLCommonscited by 60
  17. 2024Do language models plan ahead for future tokens?cited by 47
  18. 2024Automatic Pseudo-Harmful Prompt Generation for Evaluating False Refusals in Large Language Modelscited by 35
  19. 2023Make Them Spill the Beans! Coercive Knowledge Extraction from (Production) LLMscited by 26
  20. 2024FLRT: Fluent Student-Teacher Redteamingcited by 9

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 15; 2026: 10.

25 papers.

Sort:

By area:

  1. 2025Advances and Challenges in Foundation Agents: From Brain-Inspired Intelligence to Evolutionary, Collaborative, and Safe Systemscited by 125
  2. 2025Security Concerns for Large Language Models: A Surveycited by 26
  3. 2025LLM-Safety Evaluations Lack Robustnesscited by 16
  4. 2025REINFORCE Adversarial Attacks on Large Language Models: An Adaptive, Distributional, and Semantic Objectivecited by 15
  5. 2025Where to Start Alignment? Diffusion Large Language Model May Demand a Distinct Positioncited by 11
  6. 2025Jailbreak Attack Initializations as Extractors of Compliance Directionscited by 10
  7. 2025MetaDefense: Defending Finetuning-based Jailbreak Attack Before and During Generationcited by 8
  8. 2025Token Buncher: Shielding LLMs from Harmful Reinforcement Learning Fine-Tuningcited by 7
  9. 2025AdversariaLLM: A Unified and Modular Toolbox for LLM Robustness Researchcited by 5
  10. 2025Jailbreak Distillation: Renewable Safety Benchmarkingcited by 4
  11. 2025Toward Principled LLM Safety Testing: Solving the Jailbreak Oracle Problemcited by 4
  12. 2026RouteHijack: Routing-Aware Attack on Mixture-of-Experts LLMscited by 4
13 more
  1. 2025Dynamic Jailbreaking Attackcited by 3
  2. 2025The Resurgence of GCG Adversarial Attacks on Large Language Modelscited by 2
  3. 2025NonTextual Target Attackcited by 2
  4. 2025Strong but Brittle: Simple Attacks Subvert Reasoning-based Safety Guardrailscited by 2
  5. 2026Hitting a Moving Target: Test-Time Adaptation for AI Text Detection under Continual Distribution Shiftcited by 1
  6. 2026Sockpuppetting: Jailbreaking LLMs by Combining Prefilling with Optimizationcited by 0
  7. 2026Adaptive Prompt Embedding Optimization for LLM Jailbreakingcited by 0
  8. 2026Why Do Aligned LLMs Remain Jailbreakable: Refusal-Escape Directions, Operator-Level Sources, and Safety-Utility Trade-offcited by 0
  9. 2026Adaptive Probe-based Steering for Robust LLM Jailbreakingcited by 0
  10. 2026Can LLMs Reliably Self-Report Adversarial Prefills, and How?cited by 0
  11. 2026Breaking Refusal in the First Half: A Mechanistic Study of the Prefill Jailbreakcited by 0
  12. 2026ProbGuard: Calibrated Safety Risk Estimation from LLM Output Distributionscited by 0
  13. 2026LLM Alignment--Utility Asymmetry under Semantic-Preserving Transformationscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024JailPO: A Novel Black-box Jailbreak Framework via Preference Optimization against Aligned LLMs
  2. 2024Iterative Self-Tuning LLMs for Enhanced Jailbreaking Capabilities
  3. 2024Pruning for Protection: Increasing Jailbreak Resistance in Aligned LLMs Without Fine-Tuning
  4. 2024Defensive Prompt Patch: A Robust and Interpretable Defense of LLMs against Jailbreak Attacks
  5. 2026Exploring and Developing a Pre-Model Safeguard with Draft Models
  6. 2024Stealthy Jailbreak Attacks on Large Language Models via Benign Data Mirroring
  7. 2025JULI: Jailbreak Large Language Models by Self-Introspection
  8. 2025One Model Transfer to All: On Robust Jailbreak Prompts Generation against LLMs