mlsanityData to 7 Oct 2026

Safe RLHF: Safe Reinforcement Learning from Human Feedback

arXiv 2310.12773 ↗First posted 19 Oct 2023cs.AI cs.LGcited by 630

On the map: LLM alignment / RLHF / preference optimisation → Safety alignment and jailbreak attacks

Abstract (arXiv)

With the development of large language models (LLMs), striking a balance between the performance and safety of AI systems has never been more critical. However, the inherent tension between the objectives of helpfulness and harmlessness presents a significant challenge during LLM training. To address this issue, we propose Safe Reinforcement Learning from Human Feedback (Safe RLHF), a novel algorithm for human value alignment. Safe RLHF explicitly decouples human preferences regarding helpfulness and harmlessness, effectively avoiding the crowdworkers' confusion about the tension and allowing us to train separate reward and cost models. We formalize the safety concern of LLMs as an optimization task of maximizing the reward function while satisfying specified cost constraints. Leveraging the Lagrangian method to solve this constrained problem, Safe RLHF dynamically adjusts the balance between the two objectives during fine-tuning. Through a three-round fine-tuning using Safe RLHF, we demonstrate a superior ability to mitigate harmful responses while enhancing model performance compared to existing value-aligned algorithms. Experimentally, we fine-tuned the Alpaca-7B using Safe RLHF and aligned it with collected human preferences, significantly improving its helpfulness and harmlessness according to human evaluations.

Key facts. Cited by 630 papers in the map (data through 7 Oct 2026). Placed in Safety alignment and jailbreak attacks, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 51 of its 102 references that are in the map. First posted on arXiv 19 Oct 2023.

Builds on

51 of 102 references.

Closest ancestors · 2

Same direction, newest first

  1. 2023BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Dataset10 Jul
  2. 2022Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learned23 Aug

Other references · 49

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2017Proximal Policy Optimization Algorithmscited by 14,441
  5. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  6. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  7. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  8. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  9. 2018Soft Actor-Critic: Off-Policy Maximum Entropy Deep Reinforcement Learning with a Stochastic Actorcited by 4,635
  10. 2017Deep reinforcement learning from human preferencescited by 3,846
  11. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  12. 2022Scaling Instruction-Finetuned Language Modelscited by 2,997
37 more
  1. 2020Learning to summarize from human feedbackcited by 2,562
  2. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,968
  3. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 1,916
  4. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,462
  5. 2020RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Modelscited by 1,235
  6. 2018Off-Policy Deep Reinforcement Learning without Explorationcited by 1,227
  7. 2023PaLM 2 Technical Reportcited by 1,154
  8. 2022Scaling Laws for Reward Model Overoptimizationcited by 1,056
  9. 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,023
  10. 2021Ethical and social risks of harm from Language Modelscited by 957
  11. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 743
  12. 2023Can Large Language Models Be an Alternative to Human Evaluations?cited by 729
  13. 2022PAL: Program-aided Language Modelscited by 635
  14. 2022Improving alignment of dialogue agents via targeted human judgementscited by 527
  15. 2023RRHF: Rank Responses to Align Language Models with Human Feedback without tearscited by 454
  16. 2020Artificial Intelligence, Values and Alignmentcited by 443
  17. 2023Fine-Grained Human Feedback Gives Better Rewards for Language Model Trainingcited by 399
  18. 2018AI safety via debatecited by 384
  19. 2021Persistent Anti-Muslim Bias in Large Language Modelscited by 376
  20. 2023Toxicity in ChatGPT: Analyzing Persona-assigned Language Modelscited by 356
  21. 2015Risk-Constrained Reinforcement Learning with Percentile Risk Criteriacited by 320
  22. 2023Principle-Driven Self-Alignment of Language Models from Scratch with Minimal Human Supervisioncited by 318
  23. 2021Recursively Summarizing Books with Human Feedbackcited by 287
  24. 2023Preference Ranking Optimization for Human Alignmentcited by 283
  25. 2022GatorTron: A Large Clinical Language Model to Unlock Patient Information from Unstructured Electronic Health Recordscited by 238
  26. 2023ChatGPT: Jack of all trades, master of nonecited by 204
  27. 2021Societal Biases in Language Generation: Progress and Challengescited by 201
  28. 2023Model evaluation for extreme riskscited by 167
  29. 2023Safety Assessment of Chinese Large Language Modelscited by 82
  30. 2023Preference Transformer: Modeling Human Preferences using Transformers for RLcited by 77
  31. 2023OmniSafe: An Infrastructure for Accelerating Safe Reinforcement Learning Researchcited by 76
  32. 2022An Empirical Survey on Long Document Summarization: Datasets, Models and Metricscited by 74
  33. 2022Saute RL: Almost Surely Safe Reinforcement Learning Using State Augmentationcited by 63
  34. 2023All Languages Matter: On the Multilingual Safety of Large Language Modelscited by 51
  35. 2023SafeDreamer: Safe Reinforcement Learning with World Modelscited by 50
  36. 2022Characteristics of Harmful Text: Towards Rigorous Benchmarking of Language Modelscited by 49
  37. 2022Evaluating Model-free Reinforcement Learning toward Safety-critical Taskscited by 18

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 9; 2024: 186; 2025: 247; 2026: 188.

630 papers.

Sort:

By area:

  1. 2024HybridFlow: A Flexible and Efficient RLHF Frameworkcited by 1,980
  2. 2024SimPO: Simple Preference Optimization with a Reference-Free Rewardcited by 951
  3. 2024RewardBench: Evaluating Reward Models for Language Modelingcited by 480
  4. 2024WildGuard: Open One-Stop Moderation Tools for Safety Risks, Jailbreaks, and Refusals of LLMscited by 460
  5. 2024R-Judge: Benchmarking Safety Risk Awareness for LLM Agentscited by 268
  6. 2024Is DPO Superior to PPO for LLM Alignment? A Comprehensive Studycited by 262
  7. 2024PKU-SafeRLHF: Towards Multi-Level Safety Alignment for LLMs with Human Preferencecited by 252
  8. 2024SALAD-Bench: A Hierarchical and Comprehensive Safety Benchmark for Large Language Modelscited by 225
  9. 2024SORRY-Bench: Systematically Evaluating Large Language Model Safety Refusalcited by 224
  10. 2024OR-Bench: An Over-Refusal Benchmark for Large Language Modelscited by 222
  11. 2024Assessing the Brittleness of Safety Alignment via Pruning and Low-Rank Modificationscited by 220
  12. 2024WildTeaming at Scale: From In-the-Wild Jailbreaks to (Adversarially) Safer Language Modelscited by 206
38 more
  1. 2026GDPO: Group reward-Decoupled Normalization Policy Optimization for Multi-reward RL Optimizationcited by 175
  2. 2024AmpleGCG: Learning a Universal and Transferable Generative Model of Adversarial Suffixes for Jailbreaking Both Open and Closed LLMscited by 160
  3. 2024Personalizing Reinforcement Learning from Human Feedback with Variational Preference Learningcited by 147
  4. 2024Aligning with Human Judgement: The Role of Pairwise Preference in Large Language Model Evaluatorscited by 146
  5. 2024OASIS: Open Agent Social Interaction Simulations with One Million Agentscited by 127
  6. 2024Safety Layers in Aligned Large Language Models: The Key to LLM Securitycited by 126
  7. 2023Distributional Preference Learning: Understanding and Accounting for Hidden Context in RLHFcited by 125
  8. 2025Advances and Challenges in Foundation Agents: From Brain-Inspired Intelligence to Evolutionary, Collaborative, and Safe Systemscited by 125
  9. 2024Attacks, Defenses and Evaluations for LLM Conversation Safety: A Surveycited by 124
  10. 2025Safety Tax: Safety Alignment Makes Your Large Reasoning Models Less Reasonablecited by 120
  11. 2024Know Your Limits: A Survey of Abstention in Large Language Modelscited by 116
  12. 2024Eyes Closed, Safety On: Protecting Multimodal LLMs via Image-to-Text Transformationcited by 114
  13. 2024Vaccine: Perturbation-aware Alignment for Large Language Models against Harmful Fine-tuning Attackcited by 113
  14. 2024MLLM-Protector: Ensuring MLLM's Safety without Hurting Performancecited by 111
  15. 2025Act Only When It Pays: Efficient Reinforcement Learning for LLM Reasoning via Selective Rolloutscited by 108
  16. 2024Rule Based Rewards for Language Model Safetycited by 106
  17. 2024Weak-to-Strong Jailbreaking on Large Language Modelscited by 100
  18. 2024The Art of Saying No: Contextual Noncompliance in Language Modelscited by 100
  19. 2024Aligner: Efficient Alignment by Learning to Correctcited by 96
  20. 2024Group Robust Preference Optimization in Reward-free RLHFcited by 94
  21. 2024OffsetBias: Leveraging Debiased Data for Tuning Evaluatorscited by 94
  22. 2024Booster: Tackling Harmful Fine-tuning for Large Language Models via Attenuating Harmful Perturbationcited by 92
  23. 2024RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMscited by 89
  24. 2024Lisa: Lazy Safety Alignment for Large Language Models against Harmful Fine-tuning Attackcited by 89
  25. 2025GuardReasoner: Towards Reasoning-based LLM Safeguardscited by 89
  26. 2025Reward Shaping to Mitigate Reward Hacking in RLHFcited by 88
  27. 2024Harmful Fine-tuning Attacks and Defenses for Large Language Models: A Surveycited by 86
  28. 2024Panacea: Pareto Alignment via Preference Adaptation for LLMscited by 78
  29. 2024FlipAttack: Jailbreak LLMs via Flippingcited by 73
  30. 2025InternLM-XComposer2.5-Reward: A Simple Yet Effective Multi-Modal Reward Modelcited by 73
  31. 2023SimpleSafetyTests: a Test Suite for Identifying Critical Safety Risks in Large Language Modelscited by 72
  32. 2025STAIR: Improving Safety Alignment with Introspective Reasoningcited by 72
  33. 2024Antidote: Post-fine-tuning Safety Alignment for Large Language Models against Harmful Fine-tuningcited by 71
  34. 2024On the Role of Attention Heads in Large Language Model Safetycited by 70
  35. 2024Evaluating Mathematical Reasoning Beyond Accuracycited by 69
  36. 2024Self-Generated Critiques Boost Reward Modeling for Language Modelscited by 69
  37. 2024Reinforcement Learning Enhanced LLMs: A Surveycited by 69
  38. 2025SafeVLA: Towards Safety Alignment of Vision-Language-Action Model via Constrained Learningcited by 69

50 most cited of 630 All 630 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Robust Reinforcement Learning from Human Feedback for Large Language Models Fine-Tuning
  2. 2024Towards Reliable Alignment: Uncertainty-aware RLHF
  3. 2026Reinforcement Learning from Human Feedback: A Statistical Perspective
  4. 2024AI Alignment through Reinforcement Learning from Human Feedback? Contradictions and Limitations
  5. 2023RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedback
  6. 2025Doubly Robust Alignment for Large Language Models
  7. 2025RLTHF: Targeted Human Feedback for LLM Alignment
  8. 2024Reward-Robust RLHF in LLMs