mlsanityData to 14 Oct 2026

Mitigating the Safety Alignment Tax with Null-Space Constrained Policy Optimization

arXiv 2512.11391 ↗First posted 12 Dec 2025cs.LGcited by 11

On the map: LLM alignment / RLHF / preference optimisation → Safety alignment and jailbreak attacks

Abstract (arXiv)

As Large Language Models (LLMs) are increasingly deployed in real-world applications, it is important to ensure their behaviors align with human values, societal norms, and ethical principles. However, safety alignment under Reinforcement Learning (RL) often suffers from forgetting learned general abilities, which is also known as the alignment tax. To address this issue, we introduce Null-Space constrained Policy Optimization (NSPO), a novel RL framework for LLM safety alignment while preserving their core abilities. The safety policy gradients are geometrically projected into the null space of general tasks, thereby mitigating the safety alignment tax. In addition, we theoretically prove that NSPO preserves the model's original core capabilities, while still guaranteeing a descent direction for effective safety alignment. Extensive experiments demonstrate that NSPO outperforms existing methods by a large margin, achieving state-of-the-art safety performance without sacrificing accuracy on general tasks, including math, code, and instruction-following tasks. Notably, NSPO is data-efficient and only requires 40% of public human-annotated safety data from PKU-SafeRLHF to achieve promising safety performance, without a large amount of mixed general tasks data in existing alignment methods.

Key facts. Cited by 11 papers in the map (data through 14 Oct 2026). Placed in Safety alignment and jailbreak attacks, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 52 of its 65 references that are in the map. First posted on arXiv 12 Dec 2025.

Builds on

52 of 65 references.

Closest ancestors · 12

Same direction, newest first

  1. 2025AlphaAlign: Incentivizing Safety Alignment with Extremely Simplified Reinforcement Learning20 Jul
  2. 2025SafeDPO: A Simple Approach to Direct Preference Optimization with Enhanced Safety26 May
  3. 2025Improving LLM Safety Alignment with Dual-Objective Optimization5 Mar
  4. 2025Safety Tax: Safety Alignment Makes Your Large Reasoning Models Less Reasonable1 Mar
  5. 2024Enhancing Safety in Reinforcement Learning with Human Feedback via Rectified Policy Optimization25 Oct
  6. 2024PKU-SafeRLHF: Towards Multi-Level Safety Alignment for LLMs with Human Preference20 Jun
  7. 2024SORRY-Bench: Systematically Evaluating Large Language Model Safety Refusal20 Jun
  8. 2024Safety Alignment Should Be Made More Than Just a Few Tokens Deep10 Jun
  9. 2024One-Shot Safety Alignment for Large Language Models via Optimal Dualization29 May
  10. 2023AutoDAN: Interpretable Gradient-Based Adversarial Attacks on Large Language Models23 Oct
  11. 2023Safe RLHF: Safe Reinforcement Learning from Human Feedback19 Oct
  12. 2023Universal and Transferable Adversarial Attacks on Aligned Language Models27 Jul

Other references · 40

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2017Proximal Policy Optimization Algorithmscited by 15,215
  3. 2024The Llama 3 Herd of Modelscited by 14,715
  4. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  5. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  6. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  7. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  8. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  9. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 2,057
  10. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 2,004
  11. 2023Jailbroken: How Does LLM Safety Training Fail?cited by 1,651
  12. 2024HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusalcited by 1,429
28 more
  1. 2024DeepSeek-Coder: When the Large Language Model Meets Programming -- The Rise of Code Intelligencecited by 1,181
  2. 2024OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problemscited by 1,136
  3. 2017Constrained Policy Optimizationcited by 983
  4. 2019The Woman Worked as a Babysitter: On Biases in Language Generationcited by 541
  5. 2024JailbreakBench: An Open Robustness Benchmark for Jailbreaking Large Language Modelscited by 536
  6. 2023DecodingTrust: A Comprehensive Assessment of Trustworthiness in GPT Modelscited by 518
  7. 2024Jailbreaking Leading Safety-Aligned LLMs with Simple Adaptive Attackscited by 486
  8. 2018Scalable agent alignment via reward modeling: a research directioncited by 447
  9. 2018Reward Constrained Policy Optimizationcited by 417
  10. 2024AlphaEdit: Null-Space Constrained Knowledge Editing for Language Modelscited by 236
  11. 2023Red-Teaming Large Language Models using Chain of Utterances for Safety-Alignmentcited by 223
  12. 2020Projection-Based Constrained Policy Optimizationcited by 203
  13. 2023Mitigating the Alignment Tax of RLHFcited by 183
  14. 2019IPO: Interior-point Policy Optimization under Constraintscited by 132
  15. 2023Beyond One-Preference-Fits-All Alignment: Multi-Objective Direct Preference Optimizationcited by 124
  16. 2020Learning to Utilize Shaping Rewards: A New Approach of Reward Shapingcited by 114
  17. 2022Why Should Adversarial Perturbations be Imperceptible? Rethink the Research Paradigm in Adversarial NLPcited by 102
  18. 2025LeetCodeDataset: A Temporal Dataset for Robust Evaluation and Efficient Training of Code LLMscited by 70
  19. 2025Injecting Domain-Specific Knowledge into Large Language Models: A Comprehensive Surveycited by 51
  20. 2023Do Models Really Learn to Follow Instructions? An Empirical Study of Instruction Tuningcited by 49
  21. 2024Enhancing LLM Safety via Constrained Direct Preference Optimizationcited by 49
  22. 2020Constrained Markov Decision Processes via Backward Value Functionscited by 33
  23. 2024Bi-Factorial Preference Optimization: Balancing Safety-Helpfulness in Language Modelscited by 27
  24. 2024Shaping the Safety Boundaries: Understanding and Defending Against Jailbreaks in Large Language Modelscited by 23
  25. 2025Alignment and Safety in Large Language Models: Safety Mechanisms, Training Paradigms, and Emerging Challengescited by 15
  26. 2025Learning Safety Constraints for Large Language Modelscited by 13
  27. 2024Unlocking Decoding-time Controllability: Gradient-Free Multi-Objective Alignment with Contrastive Promptscited by 9
  28. 2025On the Limitations of Steering in Language Model Alignmentcited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 11.

11 papers.

  1. 2026Revisiting Robustness for LLM Safety Alignment via Selective Geometry Controlcited by 4
  2. 2026SafeSteer: Localized On-Policy Distillation for Efficient Safety Alignmentcited by 4
  3. 2026Beyond Alignment: Expanding Reasoning Capacity via Manifold-Reshaping Policy Optimizationcited by 3
  4. 2026What Is the Alignment Tax?cited by 1
  5. 2026Explaining and Breaking the Safety-Helpfulness Ceiling via Preference Dimensional Expansioncited by 1
  6. 2026Constitutional On-Policy Safe Distillationcited by 1
  7. 2026PolicyAlign: Direct Policy-Based Safety Alignment for Large Language Modelscited by 1
  8. 2026Behind Harmful Compliance: Behavioral and Mechanistic Divergence Across LLM Jailbreakscited by 0
  9. 2026Dysco: Dynamic Subspace Boosting to Mitigate LoRA Interference in Federated Learningcited by 0
  10. 2026MergeHEIR: Mitigating Multimodal Hallucinations as the Tax of Model Mergingcited by 0
  11. 2026TRACE: Trajectory Return Attribution and Contrastive Erasure for Multi-Turn Safetycited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026BSO: Safety Alignment Is Density Ratio Matching
  2. 2025Constrained Language Model Policy Optimization via Risk-aware Stepwise Alignment
  3. 2026Improving Safety Alignment via Balanced Direct Preference Optimization
  4. 2025Efficient Safety Alignment of Large Language Models via Preference Re-ranking and Representation-based Reward Modeling
  5. 2026Oyster-II: Reinforcement Learning for Constructive Safety Alignment in Large Language Models
  6. 2026Curriculum Learning for Safety Alignment
  7. 2024Stepwise Alignment for Constrained Language Model Policy Optimization
  8. 2026TriPlay-RL: Tri-Role Self-Play Reinforcement Learning for LLM Safety Alignment