mlsanityData to 7 Oct 2026

SafeChain: Safety of Language Models with Long Chain-of-Thought Reasoning Capabilities

arXiv 2502.12025 ↗First posted 17 Feb 2025cs.AI cs.CLcited by 133

On the map: LLM reasoning → Reasoning for agents, planning and code

Abstract (arXiv)

Emerging large reasoning models (LRMs), such as DeepSeek-R1 models, leverage long chain-of-thought (CoT) reasoning to generate structured intermediate steps, enhancing their reasoning capabilities. However, long CoT does not inherently guarantee safe outputs, potentially leading to harmful consequences such as the introduction of security vulnerabilities in code or the spread of misinformation. Current research on large language model (LLM) safety usually focuses on short-answer responses, overlooking the long CoT style outputs of LRMs. To bridge this gap, we conduct a systematic study of LRM safety. First, we investigate safety evaluators calibrated against human annotations. Using our newly developed metrics, we thoroughly assess the safety of 12 state-of-the-art LRMs on StrongReject and WildJailbreak datasets. Our results show that LRMs are not safe compared to their reasoning advance. Further, we perform a fine-grained analysis of the reasoning trace and final answer. We find that three decoding strategies-ZeroThink, LessThink, and MoreThink-can improve model safety without additional training. However, these strategies either use constrained reasoning traces or incur high inference costs. To better strengthen LRM safety, we introduce SafeChain, the first-of-its-kind safety training dataset in CoT style. We fine-tune two LRMs with SafeChain, showing that it not only enhances model safety but also preserves performance across 6 reasoning benchmarks.

Key facts. Cited by 133 papers in the map (data through 7 Oct 2026). Placed in Reasoning for agents, planning and code, one of 17 directions in LLM reasoning. Builds on 37 of its 58 references that are in the map. First posted on arXiv 17 Feb 2025.

Builds on

37 of 58 references.

Closest ancestors · 2

Same direction, newest first

  1. 2024Deliberative Alignment: Reasoning Enables Safer Language Models20 Dec
  2. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Code12 Mar

Other references · 35

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  4. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  5. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  6. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  7. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,146
  8. 2022Large Language Models are Zero-Shot Reasonerscited by 4,323
  9. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  10. 2023Let's Verify Step by Stepcited by 3,598
  11. 2021Program Synthesis with Large Language Modelscited by 3,509
  12. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,155
23 more
  1. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  2. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 2,914
  3. 2023Jailbroken: How Does LLM Safety Training Fail?cited by 1,565
  4. 2024LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Modelscited by 1,442
  5. 2022Least-to-Most Prompting Enables Complex Reasoning in Large Language Modelscited by 1,397
  6. 2024HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusalcited by 1,330
  7. 2025s1: Simple test-time scalingcited by 1,311
  8. 2025Kimi k1.5: Scaling Reinforcement Learning with LLMscited by 1,193
  9. 2023Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversationscited by 1,124
  10. 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,023
  11. 2023Enhancing Chat Language Models by Scaling High-quality Instructional Conversationscited by 825
  12. 2023Large Language Modelscited by 774
  13. 2023AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Modelscited by 738
  14. 2024Safety Alignment Should Be Made More Than Just a Few Tokens Deepcited by 450
  15. 2024MLE-bench: Evaluating Machine Learning Agents on Machine Learning Engineeringcited by 429
  16. 2024Training Language Models to Self-Correct via Reinforcement Learningcited by 405
  17. 2024A StrongREJECT for Empty Jailbreakscited by 352
  18. 2023Alphazero-like Tree-Search can Guide Large Language Model Decoding and Trainingcited by 291
  19. 2024SafeDecoding: Defending against Jailbreak Attacks via Safety-Aware Decodingcited by 233
  20. 2024WildBench: Benchmarking LLMs with Challenging Tasks from Real Users in the Wildcited by 211
  21. 2024WildTeaming at Scale: From In-the-Wild Jailbreaks to (Adversarially) Safer Language Modelscited by 206
  22. 2024ArtPrompt: ASCII Art-based Jailbreak Attacks against Aligned LLMscited by 197
  23. 2024ChatBug: A Common Vulnerability of Aligned LLMs Induced by Chat Templatescited by 29

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 86; 2026: 47.

133 papers.

Sort:

By area:

  1. 2025Towards Reasoning Era: A Survey of Long Chain-of-Thought for Reasoning Large Language Modelscited by 381
  2. 2025From System 1 to System 2: A Survey of Reasoning Large Language Modelscited by 319
  3. 2025Phi-4-reasoning Technical Reportcited by 146
  4. 2025Safety Tax: Safety Alignment Makes Your Large Reasoning Models Less Reasonablecited by 120
  5. 2025A Comprehensive Survey in LLM(-Agent) Full Stack Safety: Data, Training and Deploymentcited by 110
  6. 2025STAR-1: Safer Alignment of Reasoning LLMs with 1K Datacited by 67
  7. 2025Effectively Controlling Reasoning Models through Thinking Interventioncited by 63
  8. 2025RealSafe-R1: Safety-Aligned DeepSeek-R1 without Compromising Reasoning Capabilitycited by 58
  9. 2025Safety is Not Only About Refusal: Reasoning-Enhanced Fine-tuning for Interpretable LLM Safetycited by 38
  10. 2025VChain: Chain-of-Visual-Thought for Reasoning in Video Generationcited by 31
  11. 2025100 Days After DeepSeek-R1: A Survey on Replication Studies and More Directions for Reasoning Language Modelscited by 29
  12. 2025We Should Identify and Mitigate Third-Party Safety Risks in MCP-Powered Agent Systemscited by 27
38 more
  1. 2025SaRO: Enhancing LLM Safety through Reasoning-based Alignmentcited by 26
  2. 2025Mitigating Deceptive Alignment via Self-Monitoringcited by 25
  3. 2025Leaky Thoughts: Large Reasoning Models Are Not Private Thinkerscited by 25
  4. 2025SAFEPATH: Preventing Harmful Reasoning in Chain-of-Thought via Early Alignmentcited by 24
  5. 2025Reasoning as an Adaptive Defense for Safetycited by 22
  6. 2025A Comprehensive Survey on Trustworthiness in Reasoning with Large Language Modelscited by 22
  7. 2025Trade-offs in Large Reasoning Models: An Empirical Analysis of Deliberative and Adaptive Reasoning over Foundational Capabilitiescited by 20
  8. 2025SafeKey: Amplifying Aha-Moment Insights for Safety Reasoningcited by 20
  9. 2025Think in Safety: Unveiling and Mitigating Safety Alignment Collapse in Multimodal Large Reasoning Modelcited by 18
  10. 2025How Should We Enhance the Safety of Large Reasoning Models: An Empirical Studycited by 18
  11. 2025Towards Concise and Adaptive Thinking in Large Reasoning Models: A Surveycited by 18
  12. 2025Can We Predict Alignment Before Models Finish Thinking? Towards Monitoring Misaligned Reasoning Modelscited by 16
  13. 2025Generative AI Act II: Test Time Scaling Drives Cognition Engineeringcited by 14
  14. 2025Steering the CensorShip: Uncovering Representation Vectors for LLM "Thought" Controlcited by 14
  15. 2025Safety Through Reasoning: An Empirical Study of Reasoning Guardrail Modelscited by 14
  16. 2025Thinking in Character: Advancing Role-Playing Agents with Role-Aware Reasoningcited by 14
  17. 2025Alignment and Safety in Large Language Models: Safety Mechanisms, Training Paradigms, and Emerging Challengescited by 14
  18. 2025Reinforcement Fine-Tuning Powers Reasoning Capability of Multimodal Large Language Modelscited by 13
  19. 2025Lifelong Safety Alignment for Language Modelscited by 13
  20. 2025Large Reasoning Models Learn Better Alignment from Flawed Thinkingcited by 13
  21. 2025Reasoning Model Unlearning: Forgetting Traces, Not Just Answers, While Preserving Reasoning Skillscited by 12
  22. 2025VisCRA: A Visual Chain Reasoning Attack for Jailbreaking Multimodal Large Language Modelscited by 11
  23. 2025SoSBench: Benchmarking Safety Alignment on Six Scientific Domainscited by 11
  24. 2025Position: The Hidden Costs and Measurement Gaps of Reinforcement Learning with Verifiable Rewardscited by 11
  25. 2025Towards Safe Reasoning in Large Reasoning Models via Corrective Interventioncited by 11
  26. 2025Refusal Falls off a Cliff: How Safety Alignment Fails in Reasoning?cited by 11
  27. 2025SafeGRPO: Self-Rewarded Multimodal Safety Alignment via Rule-Governed Policy Optimizationcited by 11
  28. 2025A Survey of Scaling in Large Language Model Reasoningcited by 10
  29. 2025Exploring the System 1 Thinking Capability of Large Reasoning Modelscited by 10
  30. 2025ReasoningShield: Safety Detection over Reasoning Traces of Large Reasoning Modelscited by 9
  31. 2025Self-Jailbreaking: Language Models Can Reason Themselves Out of Safety Alignment After Benign Reasoning Trainingcited by 9
  32. 2025R-TOFU: Unlearning in Large Reasoning Modelscited by 8
  33. 2025ARMOR: Aligning Secure and Safe Large Language Models via Meticulous Reasoningcited by 8
  34. 2025Distributional AGI Safetycited by 8
  35. 2025Practical Reasoning Interruption Attacks on Reasoning Large Language Modelscited by 7
  36. 2025AutoRAN: Automated Hijacking of Safety Reasoning in Large Reasoning Modelscited by 7
  37. 2025Beyond Safe Answers: A Benchmark for Evaluating True Risk Awareness in Large Reasoning Modelscited by 7
  38. 2025UnsafeChain: Enhancing Reasoning Model Safety via Hard Casescited by 7

50 most cited of 133 All 133 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025The Hidden Risks of Large Reasoning Models: A Safety Assessment of R1
  2. 2026Beyond Content Safety: Real-Time Monitoring for Reasoning Vulnerabilities in Large Language Models
  3. 2025SafeMLRM: Demystifying Safety in Multi-modal Large Reasoning Models
  4. 2026SafeSci: Safety Evaluation of Large Language Models in Science Domains and Beyond
  5. 2026SFCoT: Safer Chain-of-Thought via Active Safety Evaluation and Calibration
  6. 2025H-CoT: Hijacking the Chain-of-Thought Safety Reasoning Mechanism to Jailbreak Large Reasoning Models, Including OpenAI o1/o3, DeepSeek-R1, and Gemini 2.0 Flash Thinking
  7. 2024BadChain: Backdoor Chain-of-Thought Prompting for Large Language Models
  8. 2026Chain of Risk: Safety Failures in Large Reasoning Models and Mitigation via Adaptive Multi-Principle Steering