mlsanityData to 7 Oct 2026

Pruning for Protection: Increasing Jailbreak Resistance in Aligned LLMs Without Fine-Tuning

arXiv 2401.10862 ↗First posted 19 Jan 2024cs.LG cs.AI cs.CLcited by 24

On the map: LLM safety and security → Jailbreaking

Abstract (arXiv)

This paper investigates the impact of model compression on the way Large Language Models (LLMs) process prompts, particularly concerning jailbreak resistance. We show that moderate WANDA pruning can enhance resistance to jailbreaking attacks without fine-tuning, while maintaining performance on standard benchmarks. To systematically evaluate this safety enhancement, we introduce a dataset of 225 harmful tasks across five categories. Our analysis of LLaMA-2 Chat, Vicuna 1.3, and Mistral Instruct v0.2 reveals that pruning benefits correlate with initial model safety levels. We interpret these results by examining changes in attention patterns and perplexity shifts, demonstrating that pruned models exhibit sharper attention and increased sensitivity to artificial jailbreak constructs. We extend our evaluation to the AdvBench harmful behavior tasks and the GCG attack method. We find that LLaMA-2 is much safer on AdvBench prompts than on our dataset when evaluated with manual jailbreak attempts, and that pruning is effective against both automated attacks and manual jailbreaking on Advbench.

Key facts. Cited by 24 papers in the map (data through 7 Oct 2026). Placed in Jailbreaking, one of 12 directions in LLM safety and security. Builds on 31 of its 47 references that are in the map. First posted on arXiv 19 Jan 2024.

Builds on

31 of 47 references.

Closest ancestors · 7

Same direction, newest first

  1. 2023Jailbreaking Black Box Large Language Models in Twenty Queries12 Oct
  2. 2023Catastrophic Jailbreak of Open-source LLMs via Exploiting Generation10 Oct
  3. 2023Defending Against Alignment-Breaking Attacks via Robustly Aligned LLM18 Sep
  4. 2023Certifying LLM Safety against Adversarial Prompting6 Sep
  5. 2023Universal and Transferable Adversarial Attacks on Aligned Language Models27 Jul
  6. 2023Jailbroken: How Does LLM Safety Training Fail?5 Jul
  7. 2023Jailbreaking ChatGPT via Prompt Engineering: An Empirical Study23 May

Other references · 24

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  4. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  5. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  6. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  7. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  8. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  9. 2016Pointer Sentinel Mixture Modelscited by 2,992
  10. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  11. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
  12. 2015Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Codingcited by 2,291
12 more
  1. 2020A Primer in BERTology: What we know about how BERT workscited by 807
  2. 2023A Simple and Effective Pruning Approach for Large Language Modelscited by 776
  3. 2023Baseline Defenses for Adversarial Attacks Against Aligned Language Modelscited by 608
  4. 2019Bias in Bios: A Case Study of Semantic Representation Bias in a High-Stakes Settingcited by 346
  5. 2019Analyzing the Structure of Attention in a Transformer Language Modelcited by 241
  6. 2023The Truth is in There: Improving Reasoning in Language Models with Layer-Selective Rank Reductioncited by 120
  7. 2018Sparse DNNs with Improved Adversarial Robustnesscited by 71
  8. 2019Model Compression with Adversarial Robustness: A Unified Optimization Frameworkcited by 71
  9. 2023Giraffe: Adventures in Expanding Context Lengths in LLMscited by 32
  10. 2022Pruning's Effect on Generalization Through the Lens of Training and Regularizationcited by 31
  11. 2021On the Adversarial Robustness of Quantized Neural Networkscited by 11
  12. 2023Relationship between Model Compression and Adversarial Robustness: A Review of Current Evidencecited by 6

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 13; 2025: 7; 2026: 4.

24 papers.

Sort:

By area:

  1. 2025NVIDIA Nemotron Nano 2: An Accurate and Efficient Hybrid Mamba-Transformer Reasoning Modelcited by 94
  2. 2024Coercing LLMs to do and reveal (almost) anythingcited by 88
  3. 2025Nemotron 3 Nano: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoningcited by 85
  4. 2024JailbreakZoo: Survey, Landscapes, and Horizons in Jailbreaking Large Language and Vision-Language Modelscited by 67
  5. 2024Decoding Compressed Trust: Scrutinizing the Trustworthiness of Efficient LLMs Under Compressioncited by 54
  6. 2024Risks of AI Scientists: Prioritizing Safeguarding Over Autonomycited by 49
  7. 2024Bag of Tricks: Benchmarking of Jailbreak Attacks on LLMscited by 47
  8. 2024Operationalizing a Threat Model for Red-Teaming Large Language Models (LLMs)cited by 44
  9. 2024JAILJUDGE: A Comprehensive Jailbreak Judge Benchmark with Multi-Agent Enhanced Explanation Evaluation Frameworkcited by 44
  10. 2024Jailbreaking and Mitigation of Vulnerabilities in Large Language Modelscited by 33
  11. 2026Nemotron 3 Super: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoningcited by 33
  12. 2024JailbreakEval: An Integrated Toolkit for Evaluating Jailbreak Attempts Against Large Language Modelscited by 28
12 more
  1. 2024Adversarial Tuning: Defending Against Jailbreak Attacks for LLMscited by 23
  2. 2024Model Surgery: Modulating LLM's Behavior Via Simple Parameter Editingcited by 23
  3. 2024Mission Impossible: A Statistical Perspective on Jailbreaking LLMscited by 21
  4. 2025SecurityLingua: Efficient Defense of LLM Jailbreak Attacks via Security-Aware Prompt Compressioncited by 12
  5. 2026Lost in the Noise: How Reasoning Models Fail with Contextual Distractorscited by 8
  6. 2024SoK: Prompt Hacking of Large Language Modelscited by 5
  7. 2025Hierarchical Safety Realignment: Lightweight Restoration of Safety in Pruned Large Vision-Language Modelscited by 4
  8. 2025Targeting Alignment: Extracting Safety Classifiers of Aligned LLMscited by 3
  9. 2025Can Small Language Models Reliably Resist Jailbreak Attacks? A Comprehensive Evaluationcited by 3
  10. 2025LLM in the Middle: A Systematic Review of Threats and Mitigations to Real-World LLM-based Systemscited by 1
  11. 2026Safety-Potential Pruning for Enhancing Safety Prompts Against VLM Jailbreaking Without Retrainingcited by 0
  12. 2026Securing LLMs in the Wild: Privacy and Security Challenges at the Edgecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024JailPO: A Novel Black-box Jailbreak Framework via Preference Optimization against Aligned LLMs
  2. 2026Exploring and Developing a Pre-Model Safeguard with Draft Models
  3. 2025PandaGuard: Systematic Evaluation of LLM Safety against Jailbreaking Attacks
  4. 2026MultiBreak: A Scalable and Diverse Multi-turn Jailbreak Benchmark for Evaluating LLM Safety
  5. 2025Speak Easy: Eliciting Harmful Jailbreaks from LLMs with Simple Interactions
  6. 2026Defending Large Language Models Against Jailbreak Attacks via In-Decoding Safety-Awareness Probing
  7. 2024No Free Lunch for Defending Against Prefilling Attack by In-Context Learning
  8. 2024Defending Large Language Models Against Jailbreak Attacks via Layer-specific Editing