# Self-Guided Adaptive Safety Alignment: Synthesizing and Internalizing Guidelines in Reasoning Models

arXiv: https://arxiv.org/abs/2511.21214 · HTML: https://mlsanity.com/paper/2511.21214/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Reasoning for agents, planning and code, one of 17 directions in LLM reasoning.
- Builds on 24 of its 44 references that are in the map.
- First posted on arXiv 26 Nov 2025.

## Placed in

- LLM reasoning / [Reasoning for agents, planning and code](https://mlsanity.com/lr/reasoning-for-agents-planning-and-code/)
- LLM alignment / RLHF / preference optimisation / [Safety alignment and jailbreak attacks](https://mlsanity.com/alignment/safety-alignment-and-jailbreak-attacks/)
- LLM safety and security / [Safety alignment, refusal and guardrails](https://mlsanity.com/safety/safety-alignment-refusal-and-guardrails/)

## Builds on (most cited first, 24 of 24)

- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Qwen3 Technical Report](https://mlsanity.com/paper/2505.09388/) (2025)
- [DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning](https://mlsanity.com/paper/2501.12948/) (2025)
- [Universal and Transferable Adversarial Attacks on Aligned Language Models](https://mlsanity.com/paper/2307.15043/) (2023)
- [Constitutional AI: Harmlessness from AI Feedback](https://mlsanity.com/paper/2212.08073/) (2022)
- [HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusal](https://mlsanity.com/paper/2402.04249/) (2024)
- [Jailbreaking Black Box Large Language Models in Twenty Queries](https://mlsanity.com/paper/2310.08419/) (2023)
- [OpenAI o1 System Card](https://mlsanity.com/paper/2412.16720/) (2024)
- [AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Models](https://mlsanity.com/paper/2310.04451/) (2023)
- [A StrongREJECT for Empty Jailbreaks](https://mlsanity.com/paper/2402.10260/) (2024)
- [From System 1 to System 2: A Survey of Reasoning Large Language Models](https://mlsanity.com/paper/2502.17419/) (2025)
- [Deliberative Alignment: Reasoning Enables Safer Language Models](https://mlsanity.com/paper/2412.16339/) (2024)
- [The Unlocking Spell on Base LLMs: Rethinking Alignment via In-Context Learning](https://mlsanity.com/paper/2312.01552/) (2023)
- [WildTeaming at Scale: From In-the-Wild Jailbreaks to (Adversarially) Safer Language Models](https://mlsanity.com/paper/2406.18510/) (2024)
- [Constitutional Classifiers: Defending against Universal Jailbreaks across Thousands of Hours of Red Teaming](https://mlsanity.com/paper/2501.18837/) (2025)
- [SafeChain: Safety of Language Models with Long Chain-of-Thought Reasoning Capabilities](https://mlsanity.com/paper/2502.12025/) (2025)
- [Best Practices and Lessons Learned on Synthetic Data](https://mlsanity.com/paper/2404.07503/) (2024)
- [H-CoT: Hijacking the Chain-of-Thought Safety Reasoning Mechanism to Jailbreak Large Reasoning Models, Including OpenAI o1/o3, DeepSeek-R1, and Gemini 2.0 Flash Thinking](https://mlsanity.com/paper/2502.12893/) (2025)
- [The Hidden Risks of Large Reasoning Models: A Safety Assessment of R1](https://mlsanity.com/paper/2502.12659/) (2025)
- [Safety in Large Reasoning Models: A Survey](https://mlsanity.com/paper/2504.17704/) (2025)
- [Safety is Not Only About Refusal: Reasoning-Enhanced Fine-tuning for Interpretable LLM Safety](https://mlsanity.com/paper/2503.05021/) (2025)
- [Jailbreaking Large Language Models with Symbolic Mathematics](https://mlsanity.com/paper/2409.11445/) (2024)
- [Ensuring Safe and High-Quality Outputs: A Guideline Library Approach for Language Models](https://mlsanity.com/paper/2403.11838/) (2024)
- [Don't Command, Cultivate: An Exploratory Study of System-2 Alignment](https://mlsanity.com/paper/2411.17075/) (2024)

## Abstract (arXiv)

Explicit safety policies can improve reasoning-model safety, but their effective coverage may lag behind evolving jailbreak strategies. We study whether a reasoning model can synthesize and internalize a task-specific safety guideline from a small set of harmful and benign examples. We introduce Self-Guided Adaptive Safety Alignment (SGASA). The model generates a guideline, refines it on its own errors, and selects a version by self-evaluation, which can then be applied in context or distilled into the model for guideline-free inference. Across two adversarial prompt datasets and three Qwen3 scales, in-context guidelines improve a combined safety and non-over-refusal score by 20.0-45.5 points on the 8B and 14B models. Self-evaluation selects the externally best refinement round in five of six settings, while guideline generation and utilization show distinct scaling patterns. Using alignment supervision derived only from WildJailbreak, internalized models retain 13.3-14.1 point gains on WildJailbreak without an inference-time guideline. These results support self-generated guidelines as a useful intermediate representation for low-resource safety adaptation.