mlsanityData to 7 Oct 2026

Challenges in Ensuring AI Safety in DeepSeek-R1 Models: The Shortcomings of Reinforcement Learning Strategies

arXiv 2501.17030 ↗First posted 28 Jan 2025cs.LG cs.AI cs.CLcited by 11

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

Large Language Models (LLMs) have achieved remarkable progress in reasoning, alignment, and task-specific performance. However, ensuring harmlessness in these systems remains a critical challenge, particularly in advanced models like DeepSeek-R1. This paper examines the limitations of Reinforcement Learning (RL) as the primary approach for reducing harmful outputs in DeepSeek-R1 and compares it with Supervised Fine-Tuning (SFT). While RL improves reasoning capabilities, it faces challenges such as reward hacking, generalization failures, language mixing, and high computational costs. We propose hybrid training approaches combining RL and SFT to achieve robust harmlessness reduction. Usage recommendations and future directions for deploying DeepSeek-R1 responsibly are also presented.

Key facts. Cited by 11 papers in the map (data through 7 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 0 of its 21 references that are in the map. First posted on arXiv 28 Jan 2025.

Builds on

0 of 21 references.

No references here yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 8; 2026: 3.

11 papers.

  1. 2025Towards Reasoning Era: A Survey of Long Chain-of-Thought for Reasoning Large Language Modelscited by 381
  2. 2025From System 1 to System 2: A Survey of Reasoning Large Language Modelscited by 319
  3. 2025SafeMLRM: Demystifying Safety in Multi-modal Large Reasoning Modelscited by 29
  4. 2025Towards Understanding the Safety Boundaries of DeepSeek Models: Evaluation and Findingscited by 24
  5. 2025Trade-offs in Large Reasoning Models: An Empirical Analysis of Deliberative and Adaptive Reasoning over Foundational Capabilitiescited by 20
  6. 2025Towards Robust ESG Analysis Against Greenwashing Risks: Aspect-Action Analysis with Cross-Category Generalizationcited by 7
  7. 2025Reasoning LLMs in the Medical Domain: A Literature Surveycited by 5
  8. 2025Improving Large Language Models Function Calling and Interpretability via Guided-Structured Templatescited by 3
  9. 2026Beyond Alignment: Expanding Reasoning Capacity via Manifold-Reshaping Policy Optimizationcited by 3
  10. 2026Safety Recovery in Reasoning Models Is Only a Few Early Steering Steps Awaycited by 3
  11. 2026Beyond the pale: Assessing prevalence and contents of extremist speech in LLM training datacited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Breaking the Safety-Capability Tradeoff: Reinforcement Learning with Verifiable Rewards Maintains Safety Guardrails in LLMs
  2. 2025100 Days After DeepSeek-R1: A Survey on Replication Studies and More Directions for Reasoning Language Models
  3. 2024Evaluating Defences against Unsafe Feedback in RLHF
  4. 2025Beyond SFT: Reinforcement Learning for Safer Large Reasoning Models with Better Reasoning Ability
  5. 2023Safe RLHF: Safe Reinforcement Learning from Human Feedback
  6. 2025A Survey of Reinforcement Learning for Large Reasoning Models
  7. 2025Large Language Models Reasoning Abilities Under Non-Ideal Conditions After RL-Fine-Tuning
  8. 2025Equilibrate RLHF: Towards Balancing Helpfulness-Safety Trade-off in Large Language Models