mlsanityData to 7 Oct 2026

STAR-1: Safer Alignment of Reasoning LLMs with 1K Data

arXiv 2504.01903 ↗First posted 2 Apr 2025cs.CL cs.AIcited by 67

On the map: LLM reasoning → Reasoning for agents, planning and code

Abstract (arXiv)

This paper introduces STAR-1, a high-quality, just-1k-scale safety dataset specifically designed for large reasoning models (LRMs) like DeepSeek-R1. Built on three core principles -- diversity, deliberative reasoning, and rigorous filtering -- STAR-1 aims to address the critical needs for safety alignment in LRMs. Specifically, we begin by integrating existing open-source safety datasets from diverse sources. Then, we curate safety policies to generate policy-grounded deliberative reasoning samples. Lastly, we apply a GPT-4o-based safety scoring system to select training examples aligned with best practices. Experimental results show that fine-tuning LRMs with STAR-1 leads to an average 40% improvement in safety performance across four benchmarks, while only incurring a marginal decrease (e.g., an average of 1.1%) in reasoning ability measured across five reasoning tasks. Extensive ablation studies further validate the importance of our design principles in constructing STAR-1 and analyze its efficacy across both LRMs and traditional LLMs. Our project page is https://ucsc-vlaa.github.io/STAR-1.

Key facts. Cited by 67 papers in the map (data through 7 Oct 2026). Placed in Reasoning for agents, planning and code, one of 17 directions in LLM reasoning. Builds on 48 of its 69 references that are in the map. First posted on arXiv 2 Apr 2025.

Builds on

48 of 69 references.

Closest ancestors · 3

Same direction, newest first

  1. 2025The Hidden Risks of Large Reasoning Models: A Safety Assessment of R118 Feb
  2. 2025SafeChain: Safety of Language Models with Long Chain-of-Thought Reasoning Capabilities17 Feb
  3. 2024Deliberative Alignment: Reasoning Enables Safer Language Models20 Dec

Other references · 45

  1. 2024The Llama 3 Herd of Modelscited by 14,192
  2. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  3. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  4. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,146
  5. 2016Overcoming catastrophic forgetting in neural networkscited by 4,718
  6. 2024Qwen2.5 Technical Reportcited by 4,713
  7. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  8. 2023Let's Verify Step by Stepcited by 3,598
  9. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  10. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 2,914
  11. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,748
  12. 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,758
33 more
  1. 2023Jailbroken: How Does LLM Safety Training Fail?cited by 1,565
  2. 2024HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusalcited by 1,330
  3. 2025s1: Simple test-time scalingcited by 1,311
  4. 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,278
  5. 2025Kimi k1.5: Scaling Reinforcement Learning with LLMscited by 1,193
  6. 2023Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!cited by 1,106
  7. 2023BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Datasetcited by 828
  8. 2021Generalizing to Unseen Domains: A Survey on Domain Generalizationcited by 762
  9. 2023AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Modelscited by 738
  10. 2022Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learnedcited by 702
  11. 2024WildChat: 1M ChatGPT Interaction Logs in the Wildcited by 598
  12. 2023GPTFUZZER: Red Teaming Large Language Models with Auto-Generated Jailbreak Promptscited by 520
  13. 2024JailbreakBench: An Open Robustness Benchmark for Jailbreaking Large Language Modelscited by 494
  14. 2025LIMO: Less is More for Reasoningcited by 488
  15. 2023XSTest: A Test Suite for Identifying Exaggerated Safety Behaviours in Large Language Modelscited by 477
  16. 2023LMSYS-Chat-1M: A Large-Scale Real-World LLM Conversation Datasetcited by 449
  17. 2023Catastrophic Jailbreak of Open-source LLMs via Exploiting Generationcited by 439
  18. 2023Safety-Tuned LLaMAs: Lessons From Improving the Safety of Large Language Models that Follow Instructionscited by 377
  19. 2024A StrongREJECT for Empty Jailbreakscited by 352
  20. 2024SALAD-Bench: A Hierarchical and Comprehensive Safety Benchmark for Large Language Modelscited by 225
  21. 2023Red-Teaming Large Language Models using Chain of Utterances for Safety-Alignmentcited by 213
  22. 2023Do-Not-Answer: A Dataset for Evaluating Safeguards in LLMscited by 176
  23. 2025LIMR: Less is More for RL Scalingcited by 136
  24. 2024Controllable Preference Optimization: Toward Controllable Multi-Objective Alignmentcited by 105
  25. 2023How Many Unicorns Are in This Image? A Safety Evaluation Benchmark for Vision LLMscited by 103
  26. 2024ALERT: A Comprehensive Benchmark for Assessing Large Language Models' Safety through Red Teamingcited by 96
  27. 2023SimpleSafetyTests: a Test Suite for Identifying Critical Safety Risks in Large Language Modelscited by 72
  28. 2024A Preliminary Study of o1 in Medicine: Are We Closer to an AI Doctor?cited by 33
  29. 2024AttnGCG: Enhancing Jailbreaking Attacks on LLMs with Attention Manipulationcited by 31
  30. 2023Unveiling Safety Vulnerabilities of Large Language Modelscited by 26
  31. 2024A Deep Dive into the Trade-Offs of Parameter-Efficient Preference Alignment Techniquescited by 16
  32. 2023The Poison of Alignmentcited by 10
  33. 2024Only-IF:Revealing the Decisive Effect of Instruction Diversity on Generalizationcited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 36; 2026: 31.

67 papers.

Sort:

By area:

  1. 2025OpenThoughts: Data Recipes for Reasoning Modelscited by 287
  2. 2025RealSafe-R1: Safety-Aligned DeepSeek-R1 without Compromising Reasoning Capabilitycited by 58
  3. 2025Safety in Large Reasoning Models: A Surveycited by 58
  4. 2025Mitigating Deceptive Alignment via Self-Monitoringcited by 25
  5. 2025SAFEPATH: Preventing Harmful Reasoning in Chain-of-Thought via Early Alignmentcited by 24
  6. 2025Reasoning as an Adaptive Defense for Safetycited by 22
  7. 2025A Comprehensive Survey on Trustworthiness in Reasoning with Large Language Modelscited by 22
  8. 2025SafeKey: Amplifying Aha-Moment Insights for Safety Reasoningcited by 20
  9. 2025How Should We Enhance the Safety of Large Reasoning Models: An Empirical Studycited by 18
  10. 2025Towards Concise and Adaptive Thinking in Large Reasoning Models: A Surveycited by 18
  11. 2025Knowledge or Reasoning? A Close Look at How LLMs Think Across Domainscited by 14
  12. 2025Alignment and Safety in Large Language Models: Safety Mechanisms, Training Paradigms, and Emerging Challengescited by 14
38 more
  1. 2025Large Reasoning Models Learn Better Alignment from Flawed Thinkingcited by 13
  2. 2025Reasoning Model Unlearning: Forgetting Traces, Not Just Answers, While Preserving Reasoning Skillscited by 12
  3. 2025Towards Safe Reasoning in Large Reasoning Models via Corrective Interventioncited by 11
  4. 2025Self-Jailbreaking: Language Models Can Reason Themselves Out of Safety Alignment After Benign Reasoning Trainingcited by 9
  5. 2025ARMOR: Aligning Secure and Safe Large Language Models via Meticulous Reasoningcited by 8
  6. 2025Beyond Safe Answers: A Benchmark for Evaluating True Risk Awareness in Large Reasoning Modelscited by 7
  7. 2025MSR-Align: Policy-Grounded Multimodal Alignment for Safety-Aware Reasoning in Vision-Language Modelscited by 7
  8. 2025UnsafeChain: Enhancing Reasoning Model Safety via Hard Casescited by 7
  9. 2025R1-ACT: Efficient Reasoning Model Safety Alignment by Activating Safety Knowledgecited by 6
  10. 2026Reducing the Safety Tax in LLM Safety Alignment with On-Policy Self-Distillationcited by 6
  11. 2025When Models Outthink Their Safety: Unveiling and Mitigating Self-Jailbreak in Large Reasoning Modelscited by 5
  12. 2025SAFER: Advancing Safety Alignment via Efficient Ex-Ante Reasoningcited by 4
  13. 2025AdvChain: Adversarial Chain-of-Thought Tuning for Robust Safety Alignment of Large Reasoning Modelscited by 4
  14. 2025Reasoning over Boundaries: Enhancing Specification Alignment via Test-time Deliberationcited by 3
  15. 2026Mitigating Safety Tax via Distribution-Grounded Refinement in Large Reasoning Modelscited by 3
  16. 2026Internalizing Safety Understanding in Large Reasoning Models via Verificationcited by 3
  17. 2026Do Thinking Tokens Help with Safety?cited by 3
  18. 2025Does More Inference-Time Compute Really Help Robustness?cited by 2
  19. 2025PRISM: Robust VLM Alignment with Principled Reasoning for Integrated Safety in Multimodalitycited by 2
  20. 2025EASE: Practical and Efficient Safety Alignment for Small Language Modelscited by 2
  21. 2025MetaGDPO: Alleviating Catastrophic Forgetting with Metacognitive Knowledge through Group Direct Preference Optimizationcited by 2
  22. 2025Beyond SFT: Reinforcement Learning for Safer Large Reasoning Models with Better Reasoning Abilitycited by 2
  23. 2026STAR-S: Improving Safety Alignment through Self-Taught Reasoning on Safety Rulescited by 2
  24. 2026Mitigating the Safety-utility Trade-off in LLM Alignment via Adaptive Safe Context Learningcited by 2
  25. 2026Invisible Safety Threat: Malicious Finetuning for LLM via Steganographycited by 2
  26. 2026Contrastive Reasoning Alignment: Reinforcement Learning from Hidden Representationscited by 2
  27. 2025Annotating the Chain-of-Thought: A Behavior-Labeled Dataset for AI Safetycited by 1
  28. 2025Rebellion: Noise-Robust Reasoning Training for Audio Reasoning Modelscited by 1
  29. 2025Affect, Body, Cognition, Demographics, and Emotion: The ABCDE of Text Features for Computational Affective Sciencecited by 1
  30. 2026RAPO: Risk-Aware Preference Optimization for Generalizable Safe Reasoningcited by 1
  31. 2026Subspace Control: Turning Constrained Model Steering into Controllable Spectral Optimizationcited by 1
  32. 2026Deliberative Alignment is Deep, but Uncertainty Remains: Inference time safety improvement in reasoning via attribution of unsafe behavior to base modelcited by 1
  33. 2026Reasoning Structure Matters for Safety Alignment of Reasoning Modelscited by 1
  34. 2026Self-ReSET: Learning to Self-Recover from Unsafe Reasoning Trajectoriescited by 1
  35. 2026Explaining and Breaking the Safety-Helpfulness Ceiling via Preference Dimensional Expansioncited by 1
  36. 2026Reasoning Quality Emerges Early: Data Curation for Reasoning Modelscited by 1
  37. 2026Safin-1: Safety from Within through Memory-Native State Evolutioncited by 1
  38. 2025Concept Concentration for Faithful Representation Interventioncited by 0

50 most cited of 67 All 67 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025LoRA is All You Need for Safety Alignment of Reasoning LLMs
  2. 2026Towards Safer Large Reasoning Models by Promoting Safety Decision-Making before Chain-of-Thought Generation
  3. 2026Finding and Reactivating Post-Trained LLMs' Hidden Safety Mechanisms
  4. 2025Safety Tax: Safety Alignment Makes Your Large Reasoning Models Less Reasonable
  5. 2025SafeRBench: Dissecting the Reasoning Safety of Large Language Models
  6. 2026Reasoning over Precedents Alongside Statutes: Case-Augmented Deliberative Alignment for LLM Safety
  7. 2025SaRO: Enhancing LLM Safety through Reasoning-based Alignment
  8. 2025Are Smarter LLMs Safer? Exploring Safety-Reasoning Trade-offs in Prompting and Fine-Tuning