mlsanityData to 14 Oct 2026

GuardReasoner: Towards Reasoning-based LLM Safeguards

arXiv 2501.18492 ↗First posted 30 Jan 2025cs.CR cs.AI cs.LGcited by 100

On the map: LLM reasoning → Reasoning for agents, planning and code

also in 1 more
Abstract (arXiv)

As LLMs increasingly impact safety-critical applications, ensuring their safety using guardrails remains a key challenge. This paper proposes GuardReasoner, a new safeguard for LLMs, by guiding the guard model to learn to reason. Concretely, we first create the GuardReasonerTrain dataset, which consists of 127K samples with 460K detailed reasoning steps. Then, we introduce reasoning SFT to unlock the reasoning capability of guard models. In addition, we present hard sample DPO to further strengthen their reasoning ability. In this manner, GuardReasoner achieves better performance, explainability, and generalizability. Extensive experiments and analyses on 13 benchmarks of 3 guardrail tasks demonstrate its superiority. Remarkably, GuardReasoner 8B surpasses GPT-4o+CoT by 5.74% and LLaMA Guard 3 8B by 20.84% F1 score on average. We release the training data, code, and models with different scales (1B, 3B, 8B) of GuardReasoner : https://github.com/yueliu1999/GuardReasoner/.

Key facts. Cited by 100 papers in the map (data through 14 Oct 2026). Placed in Reasoning for agents, planning and code, one of 17 directions in LLM reasoning. Builds on 75 of its 85 references that are in the map. First posted on arXiv 30 Jan 2025.

Builds on

75 of 85 references.

Closest ancestors · 5

Same direction, newest first

  1. 2024Deliberative Alignment: Reasoning Enables Safer Language Models20 Dec
  2. 2024Qwen2.5-Coder Technical Report18 Sep
  3. 2024R2-Guard: Robust Reasoning Enabled LLM Guardrail via Knowledge-Enhanced Logical Reasoning8 Jul
  4. 2024GuardAgent: Safeguard LLM Agents by a Guard Agent via Knowledge-Enabled Reasoning13 Jun
  5. 2023Encouraging Divergent Thinking in Large Language Models through Multi-Agent Debate30 May

Other references · 70

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2022Training language models to follow instructions with human feedbackcited by 15,971
  3. 2024The Llama 3 Herd of Modelscited by 14,715
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  5. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  6. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  7. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  8. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,330
  9. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  10. 2022Large Language Models are Zero-Shot Reasonerscited by 4,374
  11. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,224
  12. 2023Mistral 7Bcited by 3,164
38 more
  1. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,724
  2. 2019Fine-Tuning Language Models from Human Preferencescited by 2,150
  3. 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,998
  4. 2024Mixtral of Expertscited by 1,823
  5. 2023Improving Factuality and Reasoning in Language Models through Multiagent Debatecited by 1,784
  6. 2024LlamaFactory: Unified Efficient Fine-Tuning of 100+ Language Modelscited by 1,464
  7. 2024HarmBench: A Standardized Evaluation Framework for Automated Red Teaming and Robust Refusalcited by 1,429
  8. 2025Kimi k1.5: Scaling Reinforcement Learning with LLMscited by 1,210
  9. 2023Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversationscited by 1,207
  10. 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,055
  11. 2022Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Taskscited by 868
  12. 2023BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Datasetcited by 860
  13. 2024Training Large Language Models to Reason in a Continuous Latent Spacecited by 797
  14. 2022Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learnedcited by 729
  15. 2023Safe RLHF: Safe Reinforcement Learning from Human Feedbackcited by 656
  16. 2024Do NOT Think That Much for 2+3=? On the Overthinking of o1-Like LLMscited by 570
  17. 2023Plan-and-Solve Prompting: Improving Zero-Shot Chain-of-Thought Reasoning by Large Language Modelscited by 560
  18. 2023XSTest: A Test Suite for Identifying Exaggerated Safety Behaviours in Large Language Modelscited by 505
  19. 2024WildGuard: Open One-Stop Moderation Tools for Safety Risks, Jailbreaks, and Refusals of LLMscited by 496
  20. 2024Training Language Models to Self-Correct via Reinforcement Learningcited by 426
  21. 2023NeMo Guardrails: A Toolkit for Controllable and Safe LLM Applications with Programmable Railscited by 365
  22. 2022A Holistic Approach to Undesired Content Detection in the Real Worldcited by 356
  23. 2021Understanding Dataset Difficulty with V-Usable Informationcited by 291
  24. 2021Recursively Summarizing Books with Human Feedbackcited by 290
  25. 2022PromptSource: An Integrated Development Environment and Repository for Natural Language Promptscited by 289
  26. 2023Think before you speak: Training Language Models With Pause Tokenscited by 264
  27. 2023AI Alignment: A Comprehensive Surveycited by 263
  28. 2023Pretraining Language Models with Human Preferencescited by 247
  29. 2024ShieldGemma: Generative AI Content Moderation Based on Gemmacited by 243
  30. 2023ToxicChat: Unveiling Hidden Challenges of Toxicity Detection in Real-World User-AI Conversationcited by 231
  31. 2024SALAD-Bench: A Hierarchical and Comprehensive Safety Benchmark for Large Language Modelscited by 228
  32. 2022A New Generation of Perspective API: Efficient Multilingual Character-level Transformerscited by 185
  33. 2021Process for Adapting Language Models to Society (PALMS) with Values-Targeted Datasetscited by 181
  34. 2024COLD-Attack: Jailbreaking LLMs with Stealthiness and Controllabilitycited by 175
  35. 2021Challenges in Detoxifying Language Modelscited by 169
  36. 2024AEGIS: Online Adaptive AI Content Safety Moderation with Ensemble of LLM Expertscited by 136
  37. 2025Aegis2.0: A Diverse AI Safety Dataset and Risks Taxonomy for Alignment of LLM Guardrailscited by 132
  38. 2024Llama Guard 3 Vision: Safeguarding Human-AI Image Understanding Conversationscited by 129

50 most cited of 70 All 70 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 49; 2026: 51.

100 papers.

Sort:

By area:

  1. 2025Towards Reasoning Era: A Survey of Long Chain-of-Thought for Reasoning Large Language Modelscited by 384
  2. 2025From System 1 to System 2: A Survey of Reasoning Large Language Modelscited by 320
  3. 2025Qwen3Guard Technical Reportcited by 155
  4. 2025RoboBrain: A Unified Brain Model for Robotic Manipulation from Abstract to Concretecited by 135
  5. 2025A Comprehensive Survey in LLM(-Agent) Full Stack Safety: Data, Training and Deploymentcited by 116
  6. 2025FlowReasoner: Reinforcing Query-Level Meta-Agentscited by 63
  7. 2025Efficient Inference for Large Reasoning Models: A Surveycited by 61
  8. 2025RedTeamCUA: Realistic Adversarial Testing of Computer-Use Agents in Hybrid Web-OS Environmentscited by 46
  9. 2025VPI-Bench: Visual Prompt Injection Attacks for Computer-Use Agentscited by 45
  10. 2025Agentic Web: Weaving the Next Web with AI Agentscited by 45
  11. 2025GuardReasoner-VL: Safeguarding VLMs via Reinforced Reasoningcited by 42
  12. 2025Safety is Not Only About Refusal: Reasoning-Enhanced Fine-tuning for Interpretable LLM Safetycited by 39
38 more
  1. 2025EvoTest: Evolutionary Test-Time Learning for Self-Improving Agentic Systemscited by 37
  2. 2025Reasoning-to-Defend: Safety-Aware Reasoning Can Defend Large Language Models from Jailbreakingcited by 34
  3. 2025Contextual Integrity in LLMs via Reasoning and Reinforcement Learningcited by 31
  4. 2025100 Days After DeepSeek-R1: A Survey on Replication Studies and More Directions for Reasoning Language Modelscited by 30
  5. 2025A Comprehensive Survey on Trustworthiness in Reasoning with Large Language Modelscited by 25
  6. 2025SAFEPATH: Preventing Harmful Reasoning in Chain-of-Thought via Early Alignmentcited by 24
  7. 2025SoK: Evaluating Jailbreak Guardrails for Large Language Modelscited by 21
  8. 2025Steering Dialogue Dynamics for Robustness against Multi-turn Jailbreaking Attackscited by 20
  9. 2025DynaGuard: A Dynamic Guardian Model With User-Defined Policiescited by 19
  10. 2025Robustness via Referencing: Defending against Prompt Injection Attacks by Referencing the Executed Instructioncited by 16
  11. 2025Safety Through Reasoning: An Empirical Study of Reasoning Guardrail Modelscited by 16
  12. 2025X-Guard: Multilingual Guard Agent for Content Moderationcited by 13
  13. 2025DiffuGuard: How Intrinsic Safety is Lost and Found in Diffusion Large Language Modelscited by 12
  14. 2025MrGuard: A Multilingual Reasoning Guardrail for Universal LLM Safetycited by 9
  15. 2025The Eye of Sherlock Holmes: Uncovering User Private Attribute Profiling via Vision-Language Model Agentic Frameworkcited by 9
  16. 2025SECA: Semantically Equivalent and Coherent Attacks for Eliciting LLM Hallucinationscited by 9
  17. 2026EvoClinician: A Self-Evolving Agent for Multi-Turn Medical Diagnosis via Test-Time Evolutionary Learningcited by 8
  18. 2025ShieldVLM: Safeguarding the Multimodal Implicit Toxicity via Deliberative Reasoning with LVLMscited by 7
  19. 2025Beyond Safe Answers: A Benchmark for Evaluating True Risk Awareness in Large Reasoning Modelscited by 7
  20. 2025UnsafeChain: Enhancing Reasoning Model Safety via Hard Casescited by 7
  21. 2025TeleAI-Safety: A comprehensive LLM jailbreaking benchmark towards attacks, defenses, and evaluationscited by 7
  22. 2026What Matters For Safety Alignment?cited by 7
  23. 2026FlexGuard: Continuous Risk Scoring for Strictness-Adaptive LLM Content Moderationcited by 7
  24. 2026SafeAgent: A Runtime Protection Architecture for Agentic Systemscited by 7
  25. 2026FlowSteer: Prompt-Only Workflow Steering Exposes Planning-Time Vulnerabilities in Multi-Agent LLM Systemscited by 7
  26. 2025Reasoning-CV: Fine-tuning Powerful Reasoning LLMs for Knowledge-Assisted Claim Verificationcited by 6
  27. 2025Reasoned Safety Alignment: Ensuring Jailbreak Defense via Answer-Then-Checkcited by 6
  28. 2026SingGuard: A Policy-Adaptive Multimodal LLM Guardrail with Dynamic Reasoningcited by 6
  29. 2026DARWIN: Evolving Jailbreak Adversary and Guardrail for LLM Safety Evaluation and Protectioncited by 6
  30. 2025Geneshift: Impact of different scenario shift on Jailbreaking LLMcited by 5
  31. 2026LPG: Balancing Efficiency and Policy Reasoning in Latent Policy Guardrailscited by 5
  32. 2025Thought Purity: A Defense Framework For Chain-of-Thought Attackcited by 4
  33. 2025ReasoningGuard: Safeguarding Large Reasoning Models with Inference-time Safety Aha Momentscited by 4
  34. 2025IntentionReasoner: Facilitating Adaptive LLM Safeguards through Intent Reasoning and Selective Query Refinementcited by 4
  35. 2025CoPE: A Small Language Model for Steerable and Scalable Content Labelingcited by 4
  36. 2025ProGuard: Towards Proactive Multimodal Safeguardcited by 4
  37. 2026GLiGuard: Schema-Conditioned Classification for LLM Safeguardcited by 4
  38. 2025Reasoning's Razor: Reasoning Improves Accuracy but Can Hurt Recall at Critical Operating Points in Safety and Hallucination Detectioncited by 3

50 most cited of 100 All 100 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026PL-Guard: Probabilistic Logic Reasoning for LLM Guardrails
  2. 2025GSPR: Aligning LLM Safeguards as Generalizable Safety Policy Reasoners
  3. 2025Protect: Towards Robust Guardrailing Stack for Trustworthy Enterprise LLM Systems
  4. 2026Do Safety Guardrails Need to Reason? LeanGuard: A Fast and Light Approach for Robust Moderation
  5. 2026SafePyramid: A Hierarchical Benchmark for In-context Policy Guardrailing
  6. 2026TRACE: An Evidence-Grounded Benchmark for Safety Evaluation of Large Reasoning Models
  7. 2025Guard Vector: Beyond English LLM Guardrails with Task-Vector Composition and Streaming-Aware Prefix SFT
  8. 2025RSafe: Incentivizing proactive reasoning to build robust and adaptive LLM safeguards