# STAR-1: Safer Alignment of Reasoning LLMs with 1K Data

arXiv: https://arxiv.org/abs/2504.01903 · HTML: https://mlsanity.com/paper/2504.01903/

## Key facts

- Cited by 68 papers in the map (data through 14 Oct 2026).
- Placed in Reasoning for agents, planning and code, one of 17 directions in LLM reasoning.
- Builds on 48 of its 69 references that are in the map.
- First posted on arXiv 2 Apr 2025.

## Placed in

- LLM reasoning / [Reasoning for agents, planning and code](https://mlsanity.com/lr/reasoning-for-agents-planning-and-code/)

## Builds on (most cited first, 30 of 48)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks](https://mlsanity.com/paper/1908.10084/) (2019)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning](https://mlsanity.com/paper/2501.12948/) (2025)
- [Qwen2.5 Technical Report](https://mlsanity.com/paper/2412.15115/) (2024)
- [Overcoming catastrophic forgetting in neural networks](https://mlsanity.com/paper/1612.00796/) (2016)
- [Let's Verify Step by Step](https://mlsanity.com/paper/2305.20050/) (2023)
- [Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2204.05862/) (2022)
- [Universal and Transferable Adversarial Attacks on Aligned Language Models](https://mlsanity.com/paper/2307.15043/) (2023)
- [Constitutional AI: Harmlessness from AI Feedback](https://mlsanity.com/paper/2212.08073/) (2022)
- [GPQA: A Graduate-Level Google-Proof Q&A Benchmark](https://mlsanity.com/paper/2311.12022/) (2023)
- [MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark](https://mlsanity.com/paper/2406.01574/) (2024)
- [Jailbroken: How Does LLM Safety Training Fail?](https://mlsanity.com/paper/2307.02483/) (2023)
- [HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusal](https://mlsanity.com/paper/2402.04249/) (2024)
- [s1: Simple test-time scaling](https://mlsanity.com/paper/2501.19393/) (2025)
- [ZeRO: Memory Optimizations Toward Training Trillion Parameter Models](https://mlsanity.com/paper/1910.02054/) (2019)
- [Kimi k1.5: Scaling Reinforcement Learning with LLMs](https://mlsanity.com/paper/2501.12599/) (2025)
- [Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!](https://mlsanity.com/paper/2310.03693/) (2023)
- [BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Dataset](https://mlsanity.com/paper/2307.04657/) (2023)
- [AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Models](https://mlsanity.com/paper/2310.04451/) (2023)
- [Generalizing to Unseen Domains: A Survey on Domain Generalization](https://mlsanity.com/paper/2103.03097/) (2021)
- [Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learned](https://mlsanity.com/paper/2209.07858/) (2022)
- [WildChat: 1M ChatGPT Interaction Logs in the Wild](https://mlsanity.com/paper/2405.01470/) (2024)
- [GPTFUZZER: Red Teaming Large Language Models with Auto-Generated Jailbreak Prompts](https://mlsanity.com/paper/2309.10253/) (2023)
- [JailbreakBench: An Open Robustness Benchmark for Jailbreaking Large Language Models](https://mlsanity.com/paper/2404.01318/) (2024)
- [XSTest: A Test Suite for Identifying Exaggerated Safety Behaviours in Large Language Models](https://mlsanity.com/paper/2308.01263/) (2023)
- [LIMO: Less is More for Reasoning](https://mlsanity.com/paper/2502.03387/) (2025)
- [LMSYS-Chat-1M: A Large-Scale Real-World LLM Conversation Dataset](https://mlsanity.com/paper/2309.11998/) (2023)
- [Catastrophic Jailbreak of Open-source LLMs via Exploiting Generation](https://mlsanity.com/paper/2310.06987/) (2023)
- [Safety-Tuned LLaMAs: Lessons From Improving the Safety of Large Language Models that Follow Instructions](https://mlsanity.com/paper/2309.07875/) (2023)

## Built on by (most cited first, 30 of 68)

- [OpenThoughts: Data Recipes for Reasoning Models](https://mlsanity.com/paper/2506.04178/) (2025)
- [Safety in Large Reasoning Models: A Survey](https://mlsanity.com/paper/2504.17704/) (2025)
- [RealSafe-R1: Safety-Aligned DeepSeek-R1 without Compromising Reasoning Capability](https://mlsanity.com/paper/2504.10081/) (2025)
- [Mitigating Deceptive Alignment via Self-Monitoring](https://mlsanity.com/paper/2505.18807/) (2025)
- [A Comprehensive Survey on Trustworthiness in Reasoning with Large Language Models](https://mlsanity.com/paper/2509.03871/) (2025)
- [SAFEPATH: Preventing Harmful Reasoning in Chain-of-Thought via Early Alignment](https://mlsanity.com/paper/2505.14667/) (2025)
- [Reasoning as an Adaptive Defense for Safety](https://mlsanity.com/paper/2507.00971/) (2025)
- [SafeKey: Amplifying Aha-Moment Insights for Safety Reasoning](https://mlsanity.com/paper/2505.16186/) (2025)
- [How Should We Enhance the Safety of Large Reasoning Models: An Empirical Study](https://mlsanity.com/paper/2505.15404/) (2025)
- [Towards Concise and Adaptive Thinking in Large Reasoning Models: A Survey](https://mlsanity.com/paper/2507.09662/) (2025)
- [Alignment and Safety in Large Language Models: Safety Mechanisms, Training Paradigms, and Emerging Challenges](https://mlsanity.com/paper/2507.19672/) (2025)
- [Knowledge or Reasoning? A Close Look at How LLMs Think Across Domains](https://mlsanity.com/paper/2506.02126/) (2025)
- [Reasoning Model Unlearning: Forgetting Traces, Not Just Answers, While Preserving Reasoning Skills](https://mlsanity.com/paper/2506.12963/) (2025)
- [Large Reasoning Models Learn Better Alignment from Flawed Thinking](https://mlsanity.com/paper/2510.00938/) (2025)
- [Self-Jailbreaking: Language Models Can Reason Themselves Out of Safety Alignment After Benign Reasoning Training](https://mlsanity.com/paper/2510.20956/) (2025)
- [Towards Safe Reasoning in Large Reasoning Models via Corrective Intervention](https://mlsanity.com/paper/2509.24393/) (2025)
- [Reducing the Safety Tax in LLM Safety Alignment with On-Policy Self-Distillation](https://mlsanity.com/paper/2605.15239/) (2026)
- [ARMOR: Aligning Secure and Safe Large Language Models via Meticulous Reasoning](https://mlsanity.com/paper/2507.11500/) (2025)
- [Beyond Safe Answers: A Benchmark for Evaluating True Risk Awareness in Large Reasoning Models](https://mlsanity.com/paper/2505.19690/) (2025)
- [MSR-Align: Policy-Grounded Multimodal Alignment for Safety-Aware Reasoning in Vision-Language Models](https://mlsanity.com/paper/2506.19257/) (2025)
- [UnsafeChain: Enhancing Reasoning Model Safety via Hard Cases](https://mlsanity.com/paper/2507.21652/) (2025)
- [R1-ACT: Efficient Reasoning Model Safety Alignment by Activating Safety Knowledge](https://mlsanity.com/paper/2508.00324/) (2025)
- [When Models Outthink Their Safety: Unveiling and Mitigating Self-Jailbreak in Large Reasoning Models](https://mlsanity.com/paper/2510.21285/) (2025)
- [SAFER: Advancing Safety Alignment via Efficient Ex-Ante Reasoning](https://mlsanity.com/paper/2504.02725/) (2025)
- [Reasoning over Boundaries: Enhancing Specification Alignment via Test-time Deliberation](https://mlsanity.com/paper/2509.14760/) (2025)
- [AdvChain: Adversarial Chain-of-Thought Tuning for Robust Safety Alignment of Large Reasoning Models](https://mlsanity.com/paper/2509.24269/) (2025)
- [STAR-S: Improving Safety Alignment through Self-Taught Reasoning on Safety Rules](https://mlsanity.com/paper/2601.03537/) (2026)
- [Mitigating Safety Tax via Distribution-Grounded Refinement in Large Reasoning Models](https://mlsanity.com/paper/2602.02136/) (2026)
- [Internalizing Safety Understanding in Large Reasoning Models via Verification](https://mlsanity.com/paper/2605.08930/) (2026)
- [Do Thinking Tokens Help with Safety?](https://mlsanity.com/paper/2606.25013/) (2026)

## Abstract (arXiv)

This paper introduces STAR-1, a high-quality, just-1k-scale safety dataset specifically designed for large reasoning models (LRMs) like DeepSeek-R1. Built on three core principles -- diversity, deliberative reasoning, and rigorous filtering -- STAR-1 aims to address the critical needs for safety alignment in LRMs. Specifically, we begin by integrating existing open-source safety datasets from diverse sources. Then, we curate safety policies to generate policy-grounded deliberative reasoning samples. Lastly, we apply a GPT-4o-based safety scoring system to select training examples aligned with best practices. Experimental results show that fine-tuning LRMs with STAR-1 leads to an average 40% improvement in safety performance across four benchmarks, while only incurring a marginal decrease (e.g., an average of 1.1%) in reasoning ability measured across five reasoning tasks. Extensive ablation studies further validate the importance of our design principles in constructing STAR-1 and analyze its efficacy across both LRMs and traditional LLMs. Our project page is https://ucsc-vlaa.github.io/STAR-1.