mlsanityData to 14 Oct 2026

Safety Reasoning with Guidelines

arXiv 2502.04040 ↗First posted 6 Feb 2025cs.LG cs.AI cs.CLcited by 15

On the map: LLM reasoning → Reasoning for agents, planning and code

Abstract (arXiv)

Training safe LLMs remains a critical challenge. The most widely used method, Refusal Training (RT), struggles to generalize against various Out-of-Distribution (OOD) jailbreaking attacks. Although various advanced methods have been proposed to address this issue, we instead question whether OOD attacks inherently surpass the capability of vanilla RT. Evaluations using Best-of-N (BoN) reveal significant safety improvements as N increases, indicating models possess adequate latent safety knowledge but RT fails to consistently elicit it under OOD scenarios. Further domain adaptation analysis reveals that direct RT causes reliance on superficial shortcuts, resulting in non-generalizable representation mappings. Inspired by our findings, we propose training model to perform safety reasoning for each query. Specifically, we synthesize reasoning supervision aligned with specified guidelines that reflect diverse perspectives on safety knowledge. This encourages model to engage in deeper reasoning, explicitly eliciting and utilizing latent safety knowledge for each query. Extensive experiments show that our method significantly improves model generalization against OOD attacks.

Key facts. Cited by 15 papers in the map (data through 14 Oct 2026). Placed in Reasoning for agents, planning and code, one of 17 directions in LLM reasoning. Builds on 58 of its 69 references that are in the map. First posted on arXiv 6 Feb 2025.

Builds on

58 of 69 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024Deliberative Alignment: Reasoning Enables Safer Language Models20 Dec
  2. 2024Recursive Introspection: Teaching Language Model Agents How to Self-Improve25 Jul
  3. 2024How Johnny Can Persuade LLMs to Jailbreak Them: Rethinking Persuasion to Challenge AI Safety by Humanizing LLMs12 Jan

Other references · 55

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2022Training language models to follow instructions with human feedbackcited by 15,971
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  4. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  5. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  6. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  7. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  8. 2024GPT-4o System Cardcited by 4,588
  9. 2023Let's Verify Step by Stepcited by 3,789
  10. 2021Program Synthesis with Large Language Modelscited by 3,637
  11. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,442
  12. 2023Qwen Technical Reportcited by 3,344
38 more
  1. 2023Mistral 7Bcited by 3,164
  2. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 3,104
  3. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 3,050
  4. 2020Learning to summarize from human feedbackcited by 2,588
  5. 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,998
  6. 2023Jailbroken: How Does LLM Safety Training Fail?cited by 1,651
  7. 2016Concrete Problems in AI Safetycited by 1,642
  8. 2024LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Modelscited by 1,464
  9. 2024HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusalcited by 1,429
  10. 2023Jailbreaking Black Box Large Language Models in Twenty Queriescited by 1,420
  11. 2023Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generationcited by 1,391
  12. 2023Representation Engineering: A Top-Down Approach to AI Transparencycited by 1,336
  13. 2025Kimi k1.5: Scaling Reinforcement Learning with LLMscited by 1,210
  14. 2022Scaling Laws for Reward Model Overoptimizationcited by 1,095
  15. 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,055
  16. 2022Solving math word problems with process- and outcome-based feedbackcited by 901
  17. 2023Enhancing Chat Language Models by Scaling High-quality Instructional Conversationscited by 839
  18. 2023AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Modelscited by 785
  19. 2022Fine-Tuning can Distort Pretrained Features and Underperform Out-of-Distributioncited by 739
  20. 2023XSTest: A Test Suite for Identifying Exaggerated Safety Behaviours in Large Language Modelscited by 505
  21. 2020Underspecification Presents Challenges for Credibility in Modern Machine Learningcited by 496
  22. 2023"Do Anything Now": Characterizing and Evaluating In-The-Wild Jailbreak Prompts on Large Language Modelscited by 489
  23. 2024Safety Alignment Should Be Made More Than Just a Few Tokens Deepcited by 472
  24. 2023Scaling Relationship on Learning Mathematical Reasoning with Large Language Modelscited by 427
  25. 2023GPT-4 Is Too Smart To Be Safe: Stealthy Chat with LLMs via Ciphercited by 423
  26. 2023Safety-Tuned LLaMAs: Lessons From Improving the Safety of Large Language Models that Follow Instructionscited by 387
  27. 2024Improving Alignment and Robustness with Circuit Breakerscited by 322
  28. 2025rStar-Math: Small LLMs Can Master Math Reasoning with Self-Evolved Deep Thinkingcited by 313
  29. 2024PKU-SafeRLHF: Towards Multi-Level Safety Alignment for LLMs with Human Preferencecited by 261
  30. 2023Do-Not-Answer: A Dataset for Evaluating Safeguards in LLMscited by 184
  31. 2024Latent Adversarial Training Improves Robustness to Persistent Harmful Behaviors in LLMscited by 173
  32. 2024Orca-Math: Unlocking the potential of SLMs in Grade School Mathcited by 151
  33. 2022On Feature Learning in the Presence of Spurious Correlationscited by 145
  34. 2024Common 7B Language Models Already Possess Strong Math Capabilitiescited by 109
  35. 2024Rule Based Rewards for Language Model Safetycited by 108
  36. 2024Attacking Large Language Models with Projected Gradient Descentcited by 94
  37. 2023Overthinking the Truth: Understanding how Language Models Process False Demonstrationscited by 81
  38. 2024Does Refusal Training in LLMs Generalize to the Past Tense?cited by 79

50 most cited of 55 All 55 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 12; 2026: 3.

15 papers.

Sort:

By area:

  1. 2025Towards Reasoning Era: A Survey of Long Chain-of-Thought for Reasoning Large Language Modelscited by 384
  2. 2025Reinforcement Learning with Rubric Anchorscited by 113
  3. 2025Safety is Not Only About Refusal: Reasoning-Enhanced Fine-tuning for Interpretable LLM Safetycited by 39
  4. 2025SafeMLRM: Demystifying Safety in Multi-modal Large Reasoning Modelscited by 29
  5. 2025Generative AI Act II: Test Time Scaling Drives Cognition Engineeringcited by 15
  6. 2025Lifelong Safety Alignment for Language Modelscited by 15
  7. 2025Reinforcement Learning Meets Masked Generative Models: Mask-GRPO for Text-to-Image Generationcited by 14
  8. 2025UnsafeChain: Enhancing Reasoning Model Safety via Hard Casescited by 7
  9. 2025Strong but Brittle: Simple Attacks Subvert Reasoning-based Safety Guardrailscited by 5
  10. 2025Reasoning Up the Instruction Ladder for Controllable Language Modelscited by 5
  11. 2026Safety Recovery in Reasoning Models Is Only a Few Early Steering Steps Awaycited by 3
  12. 2025Why Does Weak-OOD Help? A Further Step Towards Understanding Jailbreaking VLMscited by 1
3 more
  1. 2025Concept Concentration for Faithful Representation Interventioncited by 0
  2. 2026Addressing Over-Refusal in LLMs with Competing Rewardscited by 0
  3. 2026ACTR: Aligning Thoughts and Responses for Multilingual Safety in Reasoning LLMscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Refusal Falls off a Cliff: How Safety Alignment Fails in Reasoning?
  2. 2026Beyond Shallow Alignment: How Post-Training Methods Determine Refusal Circuits And Steering Robustness
  3. 2025FalseReject: A Resource for Improving Contextual Safety and Mitigating Over-Refusals in LLMs via Structured Reasoning
  4. 2026Not All Refusals Are Equal: How Safety Alignment Fails Cybersecurity at Scale
  5. 2026STAR-S: Improving Safety Alignment through Self-Taught Reasoning on Safety Rules
  6. 2025Enhancing Model Defense Against Jailbreaks with Proactive Safety Reasoning
  7. 2026The Refusal--Compliance Tradeoff: A Large-Scale Safety Behavior Audit of Large Language Models
  8. 2025When Models Outthink Their Safety: Unveiling and Mitigating Self-Jailbreak in Large Reasoning Models