mlsanityData to 14 Oct 2026

Jailbreak Instruction-Tuned LLMs via end-of-sentence MLP Re-weighting

arXiv 2410.10150 ↗First posted 14 Oct 2024cs.CL cs.AIcited by 5

On the map: LLM safety and security → Jailbreaking

Abstract (arXiv)

In this paper, we investigate the safety mechanisms of instruction fine-tuned large language models (LLMs). We discover that re-weighting MLP neurons can significantly compromise a model's safety, especially for MLPs in end-of-sentence inferences. We hypothesize that LLMs evaluate the harmfulness of prompts during end-of-sentence inferences, and MLP layers plays a critical role in this process. Based on this hypothesis, we develop 2 novel white-box jailbreak methods: a prompt-specific method and a prompt-general method. The prompt-specific method targets individual prompts and optimizes the attack on the fly, while the prompt-general method is pre-trained offline and can generalize to unseen harmful prompts. Our methods demonstrate robust performance across 7 popular open-source LLMs, size ranging from 2B to 72B. Furthermore, our study provides insights into vulnerabilities of instruction-tuned LLM's safety and deepens the understanding of the internal mechanisms of LLMs.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Placed in Jailbreaking, one of 12 directions in LLM safety and security. Builds on 34 of its 37 references that are in the map. First posted on arXiv 14 Oct 2024.

Builds on

34 of 37 references.

Closest ancestors · 6

Same direction, newest first

  1. 2024A StrongREJECT for Empty Jailbreaks15 Feb
  2. 2023Jailbreaking Black Box Large Language Models in Twenty Queries12 Oct
  3. 2023Catastrophic Jailbreak of Open-source LLMs via Exploiting Generation10 Oct
  4. 2023AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Models3 Oct
  5. 2023"Do Anything Now": Characterizing and Evaluating In-The-Wild Jailbreak Prompts on Large Language Models7 Aug
  6. 2023Jailbroken: How Does LLM Safety Training Fail?5 Jul

Other references · 28

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2022Training language models to follow instructions with human feedbackcited by 15,971
  3. 2024The Llama 3 Herd of Modelscited by 14,715
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  5. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  6. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  7. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  8. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
  9. 2022Scaling Instruction-Finetuned Language Modelscited by 3,024
  10. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,975
  11. 2024HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusalcited by 1,429
  12. 2023Representation Engineering: A Top-Down Approach to AI Transparencycited by 1,336
16 more
  1. 2023Inference-Time Intervention: Eliciting Truthful Answers from a Language Modelcited by 1,302
  2. 2023Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!cited by 1,154
  3. 2024Refusal in Language Models Is Mediated by a Single Directioncited by 1,032
  4. 2022Toy Models of Superpositioncited by 971
  5. 2022Transformer Feed-Forward Layers Build Predictions by Promoting Concepts in the Vocabulary Spacecited by 532
  6. 2023Are aligned neural networks adversarially aligned?cited by 315
  7. 2020AutoPrompt: Eliciting Knowledge from Language Models with Automatically Generated Promptscited by 249
  8. 2023Label Words are Anchors: An Information Flow Perspective for Understanding In-Context Learningcited by 236
  9. 2024Assessing the Brittleness of Safety Alignment via Pruning and Low-Rank Modificationscited by 231
  10. 2023Red-Teaming Large Language Models using Chain of Utterances for Safety-Alignmentcited by 223
  11. 2023LoRA Fine-tuning Efficiently Undoes Safety Training in Llama 2-Chat 70Bcited by 195
  12. 2024A Mechanistic Understanding of Alignment Algorithms: A Case Study on DPO and Toxicitycited by 185
  13. 2023Removing RLHF Protections in GPT-4 via Fine-Tuningcited by 171
  14. 2024Safety Layers in Aligned Large Language Models: The Key to LLM Securitycited by 134
  15. 2023RLCD: Reinforcement Learning from Contrastive Distillation for Language Model Alignmentcited by 43
  16. 2024Model Editing as a Robust and Denoised variant of DPO: A Case Study on Toxicitycited by 35

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 3; 2026: 1.

5 papers.

  1. 2025Advances and Challenges in Foundation Agents: From Brain-Inspired Intelligence to Evolutionary, Collaborative, and Safe Systemscited by 126
  2. 2024Large Language Model Safety: A Holistic Surveycited by 59
  3. 2025Refusal Falls off a Cliff: How Safety Alignment Fails in Reasoning?cited by 11
  4. 2025Why Safeguarded Ships Run Aground? Aligned Large Language Models' Safety Mechanisms Tend to Be Anchored in The Template Regioncited by 6
  5. 2026A Single Neuron Is Sufficient to Bypass Safety Alignment in Large Language Modelscited by 3

Similar, not cited

Close in wording, no citation either way

  1. 2024Defending Large Language Models Against Jailbreak Attacks via Layer-specific Editing
  2. 2024Harnessing Task Overload for Scalable Jailbreak Attacks on Large Language Models
  3. 2025Paper Summary Attack: Jailbreaking LLMs through LLM Safety Papers
  4. 2025Rewrite to Jailbreak: Discover Learnable and Transferable Implicit Harmfulness Instruction
  5. 2026MultiBreak: A Scalable and Diverse Multi-turn Jailbreak Benchmark for Evaluating LLM Safety
  6. 2025Unraveling LLM Jailbreaks Through Safety Knowledge Neurons
  7. 2026Circuit Discovery Helps Detect LLM Jailbreaking: A Mechanistic Interpretability Study
  8. 2026Dynamic Deep Prompt Optimization for Defending Against Jailbreak Attacks on LLMs