mlsanityData to 7 Oct 2026

Weak-to-Strong Jailbreaking on Large Language Models

arXiv 2401.17256 ↗First posted 30 Jan 2024cs.CLcited by 100

On the map: LLM safety and security → Jailbreaking

Abstract (arXiv)

Large language models (LLMs) are vulnerable to jailbreak attacks - resulting in harmful, unethical, or biased text generations. However, existing jailbreaking methods are computationally costly. In this paper, we propose the weak-to-strong jailbreaking attack, an efficient inference time attack for aligned LLMs to produce harmful text. Our key intuition is based on the observation that jailbroken and aligned models only differ in their initial decoding distributions. The weak-to-strong attack's key technical insight is using two smaller models (a safe and an unsafe one) to adversarially modify a significantly larger safe model's decoding probabilities. We evaluate the weak-to-strong attack on 5 diverse open-source LLMs from 3 organizations. The results show our method can increase the misalignment rate to over 99% on two datasets with just one forward pass per example. Our study exposes an urgent safety issue that needs to be addressed when aligning LLMs. As an initial attempt, we propose a defense strategy to protect against such attacks, but creating more advanced defenses remains challenging. The code for replicating the method is available at https://github.com/XuandongZhao/weak-to-strong

Key facts. Cited by 100 papers in the map (data through 7 Oct 2026). Placed in Jailbreaking, one of 12 directions in LLM safety and security. Builds on 68 of its 88 references that are in the map. First posted on arXiv 30 Jan 2024.

Builds on

68 of 88 references.

Closest ancestors · 18

Same direction, newest first

  1. 2024How Johnny Can Persuade LLMs to Jailbreak Them: Rethinking Persuasion to Challenge AI Safety by Humanizing LLMs12 Jan
  2. 2023Tree of Attacks: Jailbreaking Black-Box LLMs Automatically4 Dec
  3. 2023Cognitive Overload: Jailbreaking Large Language Models with Overloaded Logical Thinking16 Nov
  4. 2023Defending Large Language Models Against Jailbreaking Attacks Through Goal Prioritization15 Nov
  5. 2023Scalable and Transferable Black-Box Jailbreaks for Language Models via Persona Modulation6 Nov
  6. 2023DeepInception: Hypnotize Large Language Model to Be Jailbreaker6 Nov
  7. 2023Jailbreaking Black Box Large Language Models in Twenty Queries12 Oct
  8. 2023Catastrophic Jailbreak of Open-source LLMs via Exploiting Generation10 Oct
  9. 2023Multilingual Jailbreak Challenges in Large Language Models10 Oct
  10. 2023Jailbreak and Guard Aligned Language Models with Only Few In-Context Demonstrations10 Oct
  11. 2023AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Models3 Oct
  12. 2023Defending Against Alignment-Breaking Attacks via Robustly Aligned LLM18 Sep
6 more
  1. 2023Certifying LLM Safety against Adversarial Prompting6 Sep
  2. 2023Open Sesame! Universal Black Box Jailbreaking of Large Language Models4 Sep
  3. 2023GPT-4 Is Too Smart To Be Safe: Stealthy Chat with LLMs via Cipher12 Aug
  4. 2023"Do Anything Now": Characterizing and Evaluating In-The-Wild Jailbreak Prompts on Large Language Models7 Aug
  5. 2023Universal and Transferable Adversarial Attacks on Aligned Language Models27 Jul
  6. 2023Jailbroken: How Does LLM Safety Training Fail?5 Jul

Other references · 50

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  5. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,326
  6. 2021On the Opportunities and Risks of Foundation Modelscited by 3,961
  7. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  8. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
  9. 2023Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversationscited by 1,124
  10. 2023Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!cited by 1,106
  11. 2023Accelerating Large Language Model Decoding with Speculative Samplingcited by 975
  12. 2023Baichuan 2: Open Large-scale Language Modelscited by 664
38 more
  1. 2023Safe RLHF: Safe Reinforcement Learning from Human Feedbackcited by 630
  2. 2023Baseline Defenses for Adversarial Attacks Against Aligned Language Modelscited by 608
  3. 2022Contrastive Decoding: Open-ended Text Generation as Optimizationcited by 598
  4. 2022Transformer Feed-Forward Layers Build Predictions by Promoting Concepts in the Vocabulary Spacecited by 522
  5. 2023DecodingTrust: A Comprehensive Assessment of Trustworthiness in GPT Modelscited by 505
  6. 2023XSTest: A Test Suite for Identifying Exaggerated Safety Behaviours in Large Language Modelscited by 477
  7. 2019On Evaluating Adversarial Robustnesscited by 442
  8. 2021DExperts: Decoding-Time Controlled Text Generation with Experts and Anti-Expertscited by 434
  9. 2018Generating Natural Language Adversarial Examplescited by 422
  10. 2023Sheared LLaMA: Accelerating Language Model Pre-training via Structured Pruningcited by 408
  11. 2023Shadow Alignment: The Ease of Subverting Safely-Aligned Language Modelscited by 284
  12. 2023Poisoning Language Models During Instruction Tuningcited by 254
  13. 2023The Unlocking Spell on Base LLMs: Rethinking Alignment via In-Context Learningcited by 251
  14. 2023Specializing Smaller Language Models towards Multi-Step Reasoningcited by 249
  15. 2023Red-Teaming Large Language Models using Chain of Utterances for Safety-Alignmentcited by 213
  16. 2023AI Control: Improving Safety Despite Intentional Subversioncited by 207
  17. 2024A Mechanistic Understanding of Alignment Algorithms: A Case Study on DPO and Toxicitycited by 179
  18. 2023Removing RLHF Protections in GPT-4 via Fine-Tuningcited by 165
  19. 2023Generative Language Models and Automated Influence Operations: Emerging Threats and Potential Mitigationscited by 157
  20. 2023Fundamental Limitations of Alignment in Large Language Modelscited by 152
  21. 2023Backdooring Instruction-Tuned Large Language Models with Virtual Prompt Injectioncited by 143
  22. 2023Controlled Decoding from Language Modelscited by 133
  23. 2024Black-Box Access is Insufficient for Rigorous AI Auditscited by 123
  24. 2023Universal Jailbreak Backdoors from Poisoned Human Feedbackcited by 120
  25. 2023On the Exploitability of Instruction Tuningcited by 110
  26. 2024Knowledge Fusion of Large Language Modelscited by 110
  27. 2024The Language Barrier: Dissecting Safety Challenges of LLMs in Multilingual Contextscited by 92
  28. 2023Jatmo: Prompt Injection Defense by Task-Specific Finetuningcited by 89
  29. 2024Tuning Language Models by Proxycited by 88
  30. 2023Reward-Augmented Decoding: Efficient Controlled Text Generation With a Unidirectional Reward Modelcited by 79
  31. 2023Attack Prompt Generation for Red Teaming and Defending Large Language Modelscited by 73
  32. 2023Language Model Inversioncited by 66
  33. 2023An Emulator for Fine-Tuning Large Language Models using Small Language Modelscited by 61
  34. 2023Inference-Time Policy Adapters (IPA): Tailoring Extreme-Scale LMs without Fine-tuningcited by 33
  35. 2023CombLM: Adapting Black-Box Language Models through Small Fine-Tuned Modelscited by 26
  36. 2023On the Safety of Open-Sourced Large Language Models: Does Alignment Really Prevent Them From Being Misused?cited by 25
  37. 2023Scaling Laws for Adversarial Attacks on Language Model Activationscited by 18
  38. 2023Safety Alignment in NLP Tasks: Weakly Aligned Summarization as an In-Context Attackcited by 10

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 37; 2025: 38; 2026: 25.

100 papers.

Sort:

By area:

  1. 2024Safety Alignment Should Be Made More Than Just a Few Tokens Deepcited by 450
  2. 2024Jailbreak Attacks and Defenses Against Large Language Models: A Surveycited by 215
  3. 2024AmpleGCG: Learning a Universal and Transferable Generative Model of Adversarial Suffixes for Jailbreaking Both Open and Closed LLMscited by 160
  4. 2024AdvPrompter: Fast Adaptive Adversarial Prompting for LLMscited by 146
  5. 2025A Comprehensive Survey in LLM(-Agent) Full Stack Safety: Data, Training and Deploymentcited by 110
  6. 2024Improved Techniques for Optimization-Based Jailbreaking on Large Language Modelscited by 98
  7. 2025H-CoT: Hijacking the Chain-of-Thought Safety Reasoning Mechanism to Jailbreak Large Reasoning Models, Including OpenAI o1/o3, DeepSeek-R1, and Gemini 2.0 Flash Thinkingcited by 94
  8. 2024Decoding-Time Language Model Alignment with Multiple Objectivescited by 90
  9. 2024Harmful Fine-tuning Attacks and Defenses for Large Language Models: A Surveycited by 86
  10. 2024When LLMs Meet Cybersecurity: A Systematic Literature Reviewcited by 70
  11. 2025Safety at Scale: A Comprehensive Survey of Large Model and Agent Safetycited by 63
  12. 2024LLMmap: Fingerprinting For Large Language Modelscited by 56
38 more
  1. 2024Don't Say No: Jailbreaking LLM by Suppressing Refusalcited by 53
  2. 2024Immune: Improving Safety Against Jailbreaks in Multi-modal LLMs via Inference-Time Alignmentcited by 42
  3. 2025Improving LLM Safety Alignment with Dual-Objective Optimizationcited by 40
  4. 2024Cascade Reward Sampling for Efficient Decoding-Time Alignmentcited by 38
  5. 2024Offset Unlearning for Large Language Modelscited by 33
  6. 2024Jailbreaking and Mitigation of Vulnerabilities in Large Language Modelscited by 33
  7. 2024JailbreakEval: An Integrated Toolkit for Evaluating Jailbreak Attempts Against Large Language Modelscited by 28
  8. 2025We Should Identify and Mitigate Third-Party Safety Risks in MCP-Powered Agent Systemscited by 27
  9. 2024How Susceptible are Large Language Models to Ideological Manipulation?cited by 26
  10. 2024How (un)ethical are instruction-centric responses of LLMs? Unveiling the vulnerabilities of safety guardrails to harmful queriescited by 26
  11. 2024Protecting Your LLMs with Information Bottleneckcited by 26
  12. 2025Security Concerns for Large Language Models: A Surveycited by 26
  13. 2024AdvPrefix: An Objective for Nuanced LLM Jailbreakscited by 25
  14. 2025Multilingual and Multi-Accent Jailbreaking of Audio LLMscited by 25
  15. 2025FlippedRAG: Black-Box Opinion Manipulation Adversarial Attacks to Retrieval-Augmented Generation Modelscited by 24
  16. 2024Attention Prompting on Image for Large Vision-Language Modelscited by 23
  17. 2025Understanding and Enhancing the Transferability of Jailbreaking Attackscited by 22
  18. 2024Bileve: Securing Text Provenance in Large Language Models Against Spoofing with Bi-level Signaturecited by 21
  19. 2024Mission Impossible: A Statistical Perspective on Jailbreaking LLMscited by 21
  20. 2024MACPO: Weak-to-Strong Alignment via Multi-Agent Contrastive Preference Optimizationcited by 20
  21. 2025SafeKey: Amplifying Aha-Moment Insights for Safety Reasoningcited by 20
  22. 2024Permute-and-Flip: An optimally stable and watermarkable decoder for LLMscited by 17
  23. 2024MoGU: A Framework for Enhancing Safety of Open-Sourced LLMs While Preserving Their Usabilitycited by 14
  24. 2025A Survey of Attacks on Large Language Modelscited by 13
  25. 2024Learning Fine-Grained Grounded Citations for Attributed Large Language Modelscited by 12
  26. 2024Jailbreaking as a Reward Misspecification Problemcited by 11
  27. 2024Black-Box Opinion Manipulation Attacks to Retrieval-Augmented Generation of Large Language Modelscited by 11
  28. 2024Root Defence Strategies: Ensuring Safety of LLM at the Decoding Levelcited by 11
  29. 2025Bias Beware: The Impact of Cognitive Biases on LLM-Driven Product Recommendationscited by 10
  30. 2024Diversity Helps Jailbreak Large Language Modelscited by 9
  31. 2025Robust Multi-Objective Controlled Decoding of Large Language Modelscited by 8
  32. 2024Defending LVLMs Against Vision Attacks through Partial-Perception Supervisioncited by 7
  33. 2025Bridging the Safety Gap: A Guardrail Pipeline for Trustworthy LLM Inferencescited by 7
  34. 2025AutoRAN: Automated Hijacking of Safety Reasoning in Large Reasoning Modelscited by 7
  35. 2025SafeTy Reasoning Elicitation Alignment for Multi-Turn Dialoguescited by 7
  36. 2026Weak-to-Strong On-Policy Distillationcited by 7
  37. 2025Proactive defense against LLM Jailbreakcited by 6
  38. 2025AutoPrompt: Automated Red-Teaming of Text-to-Image Models via LLM-Driven Adversarial Promptscited by 6

50 most cited of 100 All 100 (JSON)