mlsanityData to 7 Oct 2026

Deliberative Alignment: Reasoning Enables Safer Language Models

arXiv 2412.16339 ↗First posted 20 Dec 2024cs.CL cs.AI cs.CYcited by 292

On the map: LLM reasoning → Reasoning for agents, planning and code

Abstract (arXiv)

As large-scale language models increasingly impact safety-critical domains, ensuring their reliable adherence to well-defined principles remains a fundamental challenge. We introduce Deliberative Alignment, a new paradigm that directly teaches the model safety specifications and trains it to explicitly recall and accurately reason over the specifications before answering. We used this approach to align OpenAI's o-series models, and achieved highly precise adherence to OpenAI's safety policies, without requiring human-written chain-of-thoughts or answers. Deliberative Alignment pushes the Pareto frontier by simultaneously increasing robustness to jailbreaks while decreasing overrefusal rates, and also improves out-of-distribution generalization. We demonstrate that reasoning over explicitly specified policies enables more scalable, trustworthy, and interpretable alignment.

Key facts. Cited by 292 papers in the map (data through 7 Oct 2026). Placed in Reasoning for agents, planning and code, one of 17 directions in LLM reasoning. Builds on 28 of its 37 references that are in the map. First posted on arXiv 20 Dec 2024.

Builds on

28 of 37 references.

References · 28

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2024The Llama 3 Herd of Modelscited by 14,192
  4. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  5. 2017Deep reinforcement learning from human preferencescited by 3,846
  6. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,317
  7. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
  8. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  9. 2023Universal and Transferable Adversarial Attacks on Aligned Language Modelscited by 2,914
  10. 2023Jailbroken: How Does LLM Safety Training Fail?cited by 1,565
  11. 2023Jailbreaking Black Box Large Language Models in Twenty Queriescited by 1,326
  12. 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,023
16 more
  1. 2022Solving math word problems with process- and outcome-based feedbackcited by 871
  2. 2021BBQ: A Hand-Built Bias Benchmark for Question Answeringcited by 683
  3. 2024WildChat: 1M ChatGPT Interaction Logs in the Wildcited by 598
  4. 2024JailbreakBench: An Open Robustness Benchmark for Jailbreaking Large Language Modelscited by 494
  5. 2023XSTest: A Test Suite for Identifying Exaggerated Safety Behaviours in Large Language Modelscited by 477
  6. 2023"Do Anything Now": Characterizing and Evaluating In-The-Wild Jailbreak Prompts on Large Language Modelscited by 457
  7. 2024Jailbreaking Leading Safety-Aligned LLMs with Simple Adaptive Attackscited by 450
  8. 2024A StrongREJECT for Empty Jailbreakscited by 352
  9. 2022Generating Sequences by Learning to Self-Correctcited by 271
  10. 2023Automatically Correcting Large Language Models: Surveying the landscape of diverse self-correction strategiescited by 236
  11. 2022Learning by Distilling Contextcited by 128
  12. 2023Deception Abilities Emerged in Large Language Modelscited by 99
  13. 2024Refusal-Trained LLMs Are Easily Jailbroken As Browser Agentscited by 61
  14. 2024Uncovering Deceptive Tendencies in Language Models: A Simulated Company AI Assistantcited by 32
  15. 2024Backtracking Improves Generation Safetycited by 31
  16. 2023Large Language Model Programscited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 180; 2026: 111.

292 papers.

Sort:

By area:

  1. 2025gpt-oss-120b & gpt-oss-20b Model Cardcited by 1,258
  2. 2025Monitoring Reasoning Models for Misbehavior and the Risks of Promoting Obfuscationcited by 338
  3. 2025From System 1 to System 2: A Survey of Reasoning Large Language Modelscited by 319
  4. 2024HuatuoGPT-o1, Towards Medical Complex Reasoning with LLMscited by 257
  5. 2025MoBA: Mixture of Block Attention for Long-Context LLMscited by 188
  6. 2025Emergent Misalignment: Narrow finetuning can produce broadly misaligned LLMscited by 158
  7. 2025SafeChain: Safety of Language Models with Long Chain-of-Thought Reasoning Capabilitiescited by 133
  8. 2025A Survey of Efficient Reasoning for Large Reasoning Models: Language, Multimodality, and Beyondcited by 131
  9. 2025A Comprehensive Survey in LLM(-Agent) Full Stack Safety: Data, Training and Deploymentcited by 110
  10. 2025Reinforcement Learning with Verifiable Rewards: GRPO's Effective Loss, Dynamics, and Success Amplificationcited by 108
  11. 2025Reinforcement Learning with Rubric Anchorscited by 108
  12. 2025H-CoT: Hijacking the Chain-of-Thought Safety Reasoning Mechanism to Jailbreak Large Reasoning Models, Including OpenAI o1/o3, DeepSeek-R1, and Gemini 2.0 Flash Thinkingcited by 94
38 more
  1. 2025GuardReasoner: Towards Reasoning-based LLM Safeguardscited by 89
  2. 2025The Hidden Risks of Large Reasoning Models: A Safety Assessment of R1cited by 89
  3. 2025Subliminal Learning: Language models transmit behavioral traits via hidden signals in datacited by 89
  4. 2025STAIR: Improving Safety Alignment with Introspective Reasoningcited by 72
  5. 2025STAR-1: Safer Alignment of Reasoning LLMs with 1K Datacited by 67
  6. 2025Effectively Controlling Reasoning Models through Thinking Interventioncited by 63
  7. 2025Are Reasoning Models More Prone to Hallucination?cited by 62
  8. 2025RealSafe-R1: Safety-Aligned DeepSeek-R1 without Compromising Reasoning Capabilitycited by 58
  9. 2025Safety in Large Reasoning Models: A Surveycited by 58
  10. 2025A Comprehensive Survey of Reward Models: Taxonomy, Applications, Challenges, and Futurecited by 53
  11. 2025Reinforcement Learning from Human Feedbackcited by 53
  12. 2025Inverse Scaling in Test-Time Computecited by 49
  13. 2025From Hard Refusals to Safe-Completions: Toward Output-Centric Safety Trainingcited by 49
  14. 2025AlphaSteer: Learning Refusal Steering with Principled Null-Space Constraintcited by 45
  15. 2025The State of Multilingual LLM Safety Research: From Measuring the Language Gap to Mitigating Itcited by 42
  16. 2025A Mousetrap: Fooling Large Reasoning Models for Jailbreak with Chain of Iterative Chaoscited by 39
  17. 2025Teaching Language Models to Critique via Reinforcement Learningcited by 38
  18. 2025Safety is Not Only About Refusal: Reasoning-Enhanced Fine-tuning for Interpretable LLM Safetycited by 38
  19. 2025LlamaRL: A Distributed Asynchronous Reinforcement Learning Framework for Efficient Large-scale LLM Trainingcited by 37
  20. 2026Nemotron 3 Super: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoningcited by 33
  21. 2025Towards Understanding Subliminal Learning: When and How Hidden Biases Transfercited by 30
  22. 2025100 Days After DeepSeek-R1: A Survey on Replication Studies and More Directions for Reasoning Language Modelscited by 29
  23. 2025Adversarial Reasoning at Jailbreaking Timecited by 27
  24. 2025SaRO: Enhancing LLM Safety through Reasoning-based Alignmentcited by 26
  25. 2025Security Concerns for Large Language Models: A Surveycited by 26
  26. 2025Antidistillation Samplingcited by 24
  27. 2025Reasoning as an Adaptive Defense for Safetycited by 22
  28. 2025A Comprehensive Survey on Trustworthiness in Reasoning with Large Language Modelscited by 22
  29. 2025Reason4Rec: Deliberative User Preference Alignment of Large Language Models for Recommendationcited by 21
  30. 2025Logarithmic Regret for Online KL-Regularized Reinforcement Learningcited by 20
  31. 2025SafeKey: Amplifying Aha-Moment Insights for Safety Reasoningcited by 20
  32. 2025FORTRESS: Frontier Risk Evaluation for National Security and Public Safetycited by 20
  33. 2026GenAgent: Scaling Text-to-Image Generation via Agentic Multimodal Reasoningcited by 20
  34. 2026How RLHF Amplifies Sycophancycited by 20
  35. 2025Oyster-I: Beyond Refusal -- Constructive Safety Alignment for Responsible Language Modelscited by 19
  36. 2025PropensityBench: Evaluating Latent Safety Risks in Large Language Models via an Agentic Approachcited by 19
  37. 2025Think in Safety: Unveiling and Mitigating Safety Alignment Collapse in Multimodal Large Reasoning Modelcited by 18
  38. 2025Compromising Honesty and Harmlessness in Language Models via Deception Attackscited by 17

50 most cited of 292 All 292 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Dialectics of Alignment: Harnessing Unsafe Knowledge for Dynamic Safety Routing
  2. 2025Vulnerability Mitigation for Safety-Aligned Language Models via Debiasing
  3. 2026Beyond Safe Data: Pretraining-Stage Alignment with Regular Safety Reflection
  4. 2026MOSAIC: Composable Safety Alignment with Modular Control Tokens
  5. 2026PolicyAlign: Direct Policy-Based Safety Alignment for Large Language Models
  6. 2026SafeSteer: Localized On-Policy Distillation for Efficient Safety Alignment
  7. 2024Superficial Safety Alignment Hypothesis
  8. 2024Controllable Safety Alignment: Inference-Time Adaptation to Diverse Safety Requirements