mlsanityData to 7 Oct 2026

Safety Alignment via Constrained Knowledge Unlearning

arXiv 2505.18588 ↗First posted 24 May 2025cs.CL cs.AIcited by 13

On the map: LLM alignment / RLHF / preference optimisation → Safety alignment and jailbreak attacks

also in 1 more
Abstract (arXiv)

Despite significant progress in safety alignment, large language models (LLMs) remain susceptible to jailbreak attacks. Existing defense mechanisms have not fully deleted harmful knowledge in LLMs, which allows such attacks to bypass safeguards and produce harmful outputs. To address this challenge, we propose a novel safety alignment strategy, Constrained Knowledge Unlearning (CKU), which focuses on two primary objectives: knowledge localization and retention, and unlearning harmful knowledge. CKU works by scoring neurons in specific multilayer perceptron (MLP) layers to identify a subset U of neurons associated with useful knowledge. During the unlearning process, CKU prunes the gradients of neurons in U to preserve valuable knowledge while effectively mitigating harmful content. Experimental results demonstrate that CKU significantly enhances model safety without compromising overall performance, offering a superior balance between safety and utility compared to existing methods. Additionally, our analysis of neuron knowledge sensitivity across various MLP layers provides valuable insights into the mechanics of safety alignment and model knowledge editing.

Key facts. Cited by 13 papers in the map (data through 7 Oct 2026). Placed in Safety alignment and jailbreak attacks, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 45 of its 64 references that are in the map. First posted on arXiv 24 May 2025.

Builds on

45 of 64 references.

Closest ancestors · 4

Same direction, newest first

  1. 2024Assessing the Brittleness of Safety Alignment via Pruning and Low-Rank Modifications7 Feb
  2. 2023Safe RLHF: Safe Reinforcement Learning from Human Feedback19 Oct
  3. 2023Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!5 Oct
  4. 2023BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Dataset10 Jul

Other references · 41

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2024The Llama 3 Herd of Modelscited by 14,192
  3. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  4. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  5. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  6. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  7. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,747
  8. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,114
  9. 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,793
  10. 2023Jailbreaking Black Box Large Language Models in Twenty Queriescited by 1,326
  11. 2020Transformer Feed-Forward Layers Are Key-Value Memoriescited by 1,307
  12. 2023Representation Engineering: A Top-Down Approach to AI Transparencycited by 1,239
29 more
  1. 2019Machine Unlearningcited by 1,020
  2. 2019Are Sixteen Heads Really Better than One?cited by 803
  3. 2018SNIP: Single-shot Network Pruning based on Connection Sensitivitycited by 802
  4. 2024How Johnny Can Persuade LLMs to Jailbreak Them: Rethinking Persuasion to Challenge AI Safety by Humanizing LLMscited by 533
  5. 2023Catastrophic Jailbreak of Open-source LLMs via Exploiting Generationcited by 439
  6. 2022Knowledge Unlearning for Mitigating Privacy Risks in Language Modelscited by 424
  7. 2023Understanding the Effects of RLHF on LLM Generalisation and Diversitycited by 386
  8. 2022A Holistic Approach to Undesired Content Detection in the Real Worldcited by 337
  9. 2024Improving Alignment and Robustness with Circuit Breakerscited by 302
  10. 2023Jailbreak in pieces: Compositional Adversarial Attacks on Multi-Modal Language Modelscited by 260
  11. 2022Quark: Controllable Text Generation with Reinforced Unlearningcited by 236
  12. 2023LLM Self Defense: By Self Examination, LLMs Know They Are Being Trickedcited by 226
  13. 2023Unlearn What You Want to Forget: Efficient Unlearning for LLMscited by 211
  14. 2024Improved Techniques for Optimization-Based Jailbreaking on Large Language Modelscited by 98
  15. 2022Debiasing isn't enough! -- On the Effectiveness of Debiasing MLMs and their Social Biases in Downstream Taskscited by 48
  16. 2024Eraser: Jailbreaking Defense in Large Language Models via Unlearning Harmful Knowledgecited by 48
  17. 2024Multimodal Reasoning with Multimodal Knowledge Graphcited by 43
  18. 2023Fake Alignment: Are LLMs Really Aligned Well?cited by 39
  19. 2024Improving the Robustness of Large Language Models via Consistency Alignmentcited by 36
  20. 2023Understanding the Effect of Model Compression on Social Bias in Large Language Modelscited by 19
  21. 2023Explore-Instruct: Enhancing Domain-Specific Instruction Coverage through Active Explorationcited by 18
  22. 2023A Two-Stage Adaptation of Large Language Models for Text Rankingcited by 17
  23. 2021A Novel Global Feature-Oriented Relational Triple Extraction Model based on Table Fillingcited by 16
  24. 2024Knowledge Fusion By Evolving Weights of Language Modelscited by 16
  25. 2025ReaderLM-v2: Small Language Model for HTML to Markdown and JSONcited by 15
  26. 2021A Simple but Effective Bidirectional Framework for Relational Triple Extractioncited by 14
  27. 2024Knowledge Editing with Dynamic Knowledge Graphs for Multi-Hop Question Answeringcited by 10
  28. 2023Topic-Selective Graph Network for Topic-Focused Summarizationcited by 4
  29. 2024User Modeling Challenges in Interactive AI Assistant Systemscited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 5; 2026: 8.

13 papers.

Sort:

By area:

  1. 2025Multi-objective Large Language Model Alignment with Hierarchical Expertscited by 10
  2. 2025Rethinking Jailbreak Detection of Large Vision Language Models with Representational Contrastive Scoringcited by 9
  3. 2026SafeNeuron: Neuron-Level Safety Alignment for Large Language Modelscited by 8
  4. 2025Adaptive Detoxification: Safeguarding General Capabilities of LLMs through Toxicity-Aware Knowledge Editingcited by 5
  5. 2025SafeLLM: Unlearning Harmful Outputs from Large Language Models against Jailbreak Attackscited by 4
  6. 2026Backdoors in RLVR: Jailbreak Backdoors in LLMs From Verifiable Rewardcited by 3
  7. 2026Skill Weaving: Efficient LLM Improvement via Modular Skillpackscited by 1
  8. 2025SoK: a Comprehensive Causality Analysis Framework for Large Language Model Securitycited by 0
  9. 2026JPU: Bridging Jailbreak Defense and Unlearning via On-Policy Path Rectificationcited by 0
  10. 2026Team-Based Self-Play With Dual Adaptive Weighting for Fine-Tuning LLMscited by 0
  11. 2026EVA: Editing for Versatile Alignment against Jailbreakscited by 0
  12. 2026The Tokens Remember: When Tokenization Bypasses Knowledge Editing and Unlearningcited by 0
1 more
  1. 2026Faithful Dual-constrained Erasure for Robust LLM Safety Alignmentcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024Towards Understanding Safety Alignment: A Mechanistic Perspective from Safety Neurons
  2. 2025NeuRel-Attack: Neuron Relearning for Safety Disalignment in Large Language Models
  3. 2025NeuronTune: Fine-Grained Neuron Modulation for Balanced Safety-Utility Alignment in LLMs
  4. 2026NeuronGuard: Robust LLM Safety Alignment via Ablation-Aware Safety Signal Redistribution
  5. 2026A Single Neuron Is Sufficient to Bypass Safety Alignment in Large Language Models
  6. 2024NLSR: Neuron-Level Safety Realignment of Large Language Models Against Harmful Fine-Tuning
  7. 2025Safety Alignment Should Be Made More Than Just A Few Attention Heads
  8. 2026NeST: Neuron Selective Tuning for LLM Safety