# SafeSteer: Localized On-Policy Distillation for Efficient Safety Alignment

arXiv: https://arxiv.org/abs/2606.02530 · HTML: https://mlsanity.com/paper/2606.02530/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Placed in Safety alignment and jailbreak attacks, one of 10 directions in LLM alignment / RLHF / preference optimisation.
- Builds on 26 of its 35 references that are in the map.
- First posted on arXiv 1 Jun 2026.

## Placed in

- LLM alignment / RLHF / preference optimisation / [Safety alignment and jailbreak attacks](https://mlsanity.com/alignment/safety-alignment-and-jailbreak-attacks/)
- LLM safety and security / [Safety alignment, refusal and guardrails](https://mlsanity.com/safety/safety-alignment-refusal-and-guardrails/)
- Knowledge distillation / [On-policy and self-distillation for LLMs](https://mlsanity.com/distill/on-policy-and-self-distillation-for-llms/)

## Builds on (most cited first, 26 of 26)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [Let's Verify Step by Step](https://mlsanity.com/paper/2305.20050/) (2023)
- [Universal and Transferable Adversarial Attacks on Aligned Language Models](https://mlsanity.com/paper/2307.15043/) (2023)
- [HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusal](https://mlsanity.com/paper/2402.04249/) (2024)
- [Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!](https://mlsanity.com/paper/2310.03693/) (2023)
- [DeepSeek-V4: Towards Highly Efficient Million-Token Context Intelligence](https://mlsanity.com/paper/2606.19348/) (2026)
- [Refusal in Language Models Is Mediated by a Single Direction](https://mlsanity.com/paper/2406.11717/) (2024)
- [On-Policy Distillation of Language Models: Learning from Self-Generated Mistakes](https://mlsanity.com/paper/2306.13649/) (2023)
- [Safe RLHF: Safe Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2310.12773/) (2023)
- [GLM-5: from Vibe Coding to Agentic Engineering](https://mlsanity.com/paper/2602.15763/) (2026)
- [Safety Alignment Should Be Made More Than Just a Few Tokens Deep](https://mlsanity.com/paper/2406.05946/) (2024)
- [Improving Alignment and Robustness with Circuit Breakers](https://mlsanity.com/paper/2406.04313/) (2024)
- [Rethinking On-Policy Distillation of Large Language Models: Phenomenology, Mechanism, and Recipe](https://mlsanity.com/paper/2604.13016/) (2026)
- [Red-Teaming Large Language Models using Chain of Utterances for Safety-Alignment](https://mlsanity.com/paper/2308.09662/) (2023)
- [A Survey of On-Policy Distillation for Large Language Models](https://mlsanity.com/paper/2604.00626/) (2026)
- [Safety Tax: Safety Alignment Makes Your Large Reasoning Models Less Reasonable](https://mlsanity.com/paper/2503.00555/) (2025)
- [ALERT: A Comprehensive Benchmark for Assessing Large Language Models' Safety through Red Teaming](https://mlsanity.com/paper/2404.08676/) (2024)
- [LLMs Encode Harmfulness and Refusal Separately](https://mlsanity.com/paper/2507.11878/) (2025)
- [On the Role of Attention Heads in Large Language Model Safety](https://mlsanity.com/paper/2410.13708/) (2024)
- [Improving LLM Safety Alignment with Dual-Objective Optimization](https://mlsanity.com/paper/2503.03710/) (2025)
- [Bi-Factorial Preference Optimization: Balancing Safety-Helpfulness in Language Models](https://mlsanity.com/paper/2408.15313/) (2024)
- [One-Shot Safety Alignment for Large Language Models via Optimal Dualization](https://mlsanity.com/paper/2405.19544/) (2024)
- [Layer-Aware Representation Filtering: Purifying Finetuning Data to Preserve LLM Safety Alignment](https://mlsanity.com/paper/2507.18631/) (2025)
- [X-Boundary: Establishing Exact Safety Boundary to Shield LLMs from Multi-Turn Jailbreaks without Compromising Usability](https://mlsanity.com/paper/2502.09990/) (2025)
- [Mitigating the Safety Alignment Tax with Null-Space Constrained Policy Optimization](https://mlsanity.com/paper/2512.11391/) (2025)

## Built on by (most cited first, 4 of 4)

- [SafeEvolve: Harness-Policy Co-Evolution from Agent Experience for Safety Alignment](https://mlsanity.com/paper/2609.02786/) (2026)
- [Activation-Conditioned Self-Distillation](https://mlsanity.com/paper/2609.38342/) (2026)
- [Look Before You Select: Rethinking Vocabulary Sparsification in On-Policy Distillation](https://mlsanity.com/paper/2609.34386/) (2026)
- [Does On-Policy Distillation for Safety Pose Backdoor Risks?](https://mlsanity.com/paper/2610.07654/) (2026)

## Abstract (arXiv)

Aligning Large Language Models (LLMs) with human values often degrades their general capabilities, termed the alignment tax. Existing methods mitigate this by balancing dual objectives, which heavily rely on massive general-purpose data or auxiliary reward models. In this paper, we argue that, because safety features are inherently sparse within the output distribution, alignment requires localized modifications rather than global trade-offs. To this end, we propose SafeSteer, which performs on-policy distillation confined to safety tokens. First, we construct a safety teacher via activation steering. Based on this teacher, we develop a safety token selection algorithm. Consequently, SafeSteer restricts the reverse KL penalty to these tokens during training to preserve general capabilities. Experimental results across diverse models show that our SafeSteer achieves a superior trade-off between safety and general capability compared with existing methods, attaining strong safety performance on seven safety benchmarks with only minimal degradation on five general capability benchmarks. Notably, SafeSteer requires only 100 harmful samples without using any general-purpose data, less than 1% of what previous baselines used, considerably reducing alignment cost. More details are on our project page at https://anjingkun.github.io/SafeSteer.