mlsanityData to 7 Oct 2026

PKU-SafeRLHF: Towards Multi-Level Safety Alignment for LLMs with Human Preference

arXiv 2406.15513 ↗First posted 20 Jun 2024cs.AI cs.CLcited by 252

On the map: LLM alignment / RLHF / preference optimisation → Safety alignment and jailbreak attacks

also in 1 more
Abstract (arXiv)

In this study, we introduce the safety human preference dataset, PKU-SafeRLHF, designed to promote research on safety alignment in large language models (LLMs). As a sibling project to SafeRLHF and BeaverTails, we separate annotations of helpfulness and harmlessness for question-answering pairs, providing distinct perspectives on these coupled attributes. Overall, we provide 44.6k refined prompts and 265k question-answer pairs with safety meta-labels for 19 harm categories and three severity levels ranging from minor to severe, with answers generated by Llama-family models. Based on this, we collected 166.8k preference data, including dual-preference (helpfulness and harmlessness decoupled) and single-preference data (trade-off the helpfulness and harmlessness from scratch), respectively. Using the large-scale annotation data, we further train severity-sensitive moderation for the risk control of LLMs and safety-centric RLHF algorithms for the safety alignment of LLMs. We believe this dataset will be a valuable resource for the community, aiding in the safe deployment of LLMs. Data is available at https://huggingface.co/datasets/PKU-Alignment/PKU-SafeRLHF.

Key facts. Cited by 252 papers in the map (data through 7 Oct 2026). Placed in Safety alignment and jailbreak attacks, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 43 of its 92 references that are in the map. First posted on arXiv 20 Jun 2024.

Builds on

43 of 92 references.

Closest ancestors · 5

Same direction, newest first

  1. 2023Safe RLHF: Safe Reinforcement Learning from Human Feedback19 Oct
  2. 2023Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!5 Oct
  3. 2023AutoDAN: Generating Stealthy Jailbreak Prompts on Aligned Large Language Models3 Oct
  4. 2023Universal and Transferable Adversarial Attacks on Aligned Language Models27 Jul
  5. 2023BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Dataset10 Jul

Other references · 38

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2023GPT-4 Technical Reportcited by 18,132
  3. 2022Training language models to follow instructions with human feedbackcited by 15,712
  4. 2017Proximal Policy Optimization Algorithmscited by 14,441
  5. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  6. 2019Sentence-BERT: Sentence Embeddings using Siamese BERT-Networkscited by 9,049
  7. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  8. 2017Deep reinforcement learning from human preferencescited by 3,846
  9. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  10. 2023Qwen Technical Reportcited by 3,273
  11. 2018UMAP: Uniform Manifold Approximation and Projection for Dimension Reductioncited by 2,841
  12. 2023A Survey on Large Language Model based Autonomous Agentscited by 2,203
26 more
  1. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 1,916
  2. 2023The Rise and Potential of Large Language Model Based Agents: A Surveycited by 1,422
  3. 2023PaLM 2 Technical Reportcited by 1,154
  4. 2023Llama Guard: LLM-based Input-Output Safeguard for Human-AI Conversationscited by 1,124
  5. 2023Towards Understanding Sycophancy in Language Modelscited by 988
  6. 2023A Multitask, Multilingual, Multimodal Evaluation of ChatGPT on Reasoning, Hallucination, and Interactivitycited by 888
  7. 2022Discovering Language Model Behaviors with Model-Written Evaluationscited by 805
  8. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 743
  9. 2023Baichuan 2: Open Large-scale Language Modelscited by 664
  10. 2023GPTFUZZER: Red Teaming Large Language Models with Auto-Generated Jailbreak Promptscited by 520
  11. 2023A Survey on Large Language Model (LLM) Security and Privacy: The Good, the Bad, and the Uglycited by 380
  12. 2021All That's 'Human' Is Not Gold: Evaluating Human Evaluation of Generated Textcited by 287
  13. 2023AI Alignment: A Comprehensive Surveycited by 256
  14. 2023AI Deception: A Survey of Examples, Risks, and Potential Solutionscited by 185
  15. 2022A Review of Safe Reinforcement Learning: Methods, Theory and Applicationscited by 169
  16. 2022Repairing the Cracked Foundation: A Survey of Obstacles in Evaluation Practices for Generated Textcited by 154
  17. 2023Safety-Gymnasium: A Unified Safe Reinforcement Learning Benchmarkcited by 135
  18. 2023Harms from Increasingly Agentic Algorithmic Systemscited by 134
  19. 2023On the Risk of Misinformation Pollution with Large Language Modelscited by 124
  20. 2023Black-Box Prompt Optimization: Aligning Large Language Models without Model Trainingcited by 122
  21. 2023Red teaming ChatGPT via Jailbreaking: Bias, Robustness, Reliability and Toxicitycited by 86
  22. 2023AlignBench: Benchmarking Chinese Alignment of Large Language Modelscited by 68
  23. 2023BiasAsker: Measuring the Bias in Conversational AI Systemcited by 52
  24. 2023All Languages Matter: On the Multilingual Safety of Large Language Modelscited by 51
  25. 2023Not All Countries Celebrate Thanksgiving: On the Cultural Dominance in Large Language Modelscited by 42
  26. 2022Trustworthy Reinforcement Learning Against Intrinsic Vulnerabilities: Robustness, Safety, and Generalizabilitycited by 32

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 14; 2025: 104; 2026: 134.

252 papers.

Sort:

By area:

  1. 2024LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation Methodscited by 451
  2. 2025Qwen3Guard Technical Reportcited by 125
  3. 2025A Comprehensive Survey in LLM(-Agent) Full Stack Safety: Data, Training and Deploymentcited by 110
  4. 2025AgenTracer: Who Is Inducing Failure in the LLM Agentic Systems?cited by 105
  5. 2025RealSafe-R1: Safety-Aligned DeepSeek-R1 without Compromising Reasoning Capabilitycited by 58
  6. 2025Your Agent May Misevolve: Emergent Risks in Self-evolving LLM Agentscited by 52
  7. 2024On Targeted Manipulation and Deception when Optimizing LLMs for User Feedbackcited by 45
  8. 2025A Survey of Direct Preference Optimizationcited by 35
  9. 2025Cultivating Pluralism In Algorithmic Monoculture: The Community Alignment Datasetcited by 32
  10. 2025From Language to Action: A Review of Large Language Models as Autonomous Agents and Tool Userscited by 32
  11. 2025MTSA: Multi-turn Safety Alignment for LLMs through Multi-round Red-teamingcited by 31
  12. 2024Rethinking Reward Model Evaluation: Are We Barking up the Wrong Tree?cited by 29
38 more
  1. 2025From 1,000,000 Users to Every User: Scaling Up Personalized Preference for User-level Alignmentcited by 28
  2. 2025Safe RLHF-V: Safe Reinforcement Learning from Multi-modal Human Feedbackcited by 27
  3. 2025PARM: Multi-Objective Test-Time Alignment via Preference-Aware Autoregressive Reward Modelcited by 27
  4. 2025RewardAnything: Generalizable Principle-Following Reward Modelscited by 27
  5. 2025SaRO: Enhancing LLM Safety through Reasoning-based Alignmentcited by 26
  6. 2025CARES: Comprehensive Evaluation of Safety and Adversarial Robustness in Medical LLMscited by 26
  7. 2026Noisy but Valid: Robust Statistical Evaluation of LLMs with Imperfect Judgescited by 23
  8. 2025Gradient-Adaptive Policy Optimization: Towards Multi-Objective Alignment of Large Language Modelscited by 22
  9. 2025What's In My Human Feedback? Learning Interpretable Descriptions of Preference Datacited by 22
  10. 2025AI Alignment at Your Discretioncited by 20
  11. 2025From Judgment to Interference: Early Stopping LLM Harmful Outputs via Streaming Content Monitoringcited by 20
  12. 2025Oyster-I: Beyond Refusal -- Constructive Safety Alignment for Responsible Language Modelscited by 19
  13. 2025MPO: Multilingual Safety Alignment via Reward Gap Optimizationcited by 18
  14. 2025Gradient-Based Multi-Objective Deep Learning: Algorithms, Theories, Applications, and Beyondcited by 15
  15. 2025Extragradient Preference Optimization (EGPO): Beyond Last-Iterate Convergence for Nash Learning from Human Feedbackcited by 15
  16. 2025Safety Reasoning with Guidelinescited by 14
  17. 2025RSafe: Incentivizing proactive reasoning to build robust and adaptive LLM safeguardscited by 14
  18. 2026YuFeng-XGuard: A Reasoning-Centric, Interpretable, and Flexible Guardrail Model for Large Language Modelscited by 14
  19. 2025Lifelong Safety Alignment for Language Modelscited by 13
  20. 2025Omni-SafetyBench: A Benchmark for Safety Evaluation of Audio-Visual Large Language Modelscited by 13
  21. 2025From Code Foundation Models to Agents and Applications: A Comprehensive Survey and Practical Guide to Code Intelligencecited by 13
  22. 2024Understanding Forgetting in LLM Supervised Fine-Tuning and Preference Learning -- A Convex Optimization Perspectivecited by 12
  23. 2025TinyR1-32B-Preview: Boosting Accuracy with Branch-Merge Distillationcited by 12
  24. 2026SCOPE: Selective Conformal Optimized Pairwise LLM Judgingcited by 12
  25. 2024LongSafety: Enhance Safety for Long-Context LLMscited by 11
  26. 2025Mitigating the Safety Alignment Tax with Null-Space Constrained Policy Optimizationcited by 11
  27. 2024RED: Unleashing Token-Level Rewards from Holistic Feedback via Reward Redistributioncited by 10
  28. 2026CARE: Confounder-Aware Aggregation for Reliable LLM Evaluationcited by 10
  29. 2024Sequence to Sequence Reward Modeling: Improving RLHF by Language Feedbackcited by 9
  30. 2024Playing Language Game with LLMs Leads to Jailbreakingcited by 9
  31. 2025SEA: Low-Resource Safety Alignment for Multimodal Large Language Models via Synthetic Embeddingscited by 9
  32. 2025DualBreach: Efficient Dual-Jailbreaking via Target-Driven Initialization and Multi-Target Optimizationcited by 9
  33. 2025Mining Intrinsic Rewards from LLM Hidden States for Efficient Best-of-N Samplingcited by 9
  34. 2025MiCRo: Mixture Modeling and Context-aware Routing for Personalized Preference Learningcited by 9
  35. 2025SafeLawBench: Towards Safe Alignment of Large Language Modelscited by 9
  36. 2025A2D: Any-Order, Any-Step Safety Alignment for Diffusion Language Modelscited by 9
  37. 2025Rethinking Jailbreak Detection of Large Vision Language Models with Representational Contrastive Scoringcited by 9
  38. 2026Outcome Accuracy is Not Enough: Aligning the Reasoning Process of Reward Modelscited by 9

50 most cited of 252 All 252 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025SafeTuneBed: A Toolkit for Benchmarking LLM Safety Alignment in Fine-Tuning
  2. 2024Safety Arithmetic: A Framework for Test-time Safety Alignment of Language Models by Steering Parameters and Activations
  3. 2023Red-Teaming Large Language Models using Chain of Utterances for Safety-Alignment
  4. 2026Can Safety Emerge from Weak Supervision? A Systematic Analysis of Small Language Models
  5. 2024Legend: Leveraging Representation Engineering to Annotate Safety Margin for Preference Datasets
  6. 2024Emulated Disalignment: Safety Alignment for Large Language Models May Backfire!
  7. 2024SafeWorld: Geo-Diverse Safety Alignment
  8. 2026SafeSci: Safety Evaluation of Large Language Models in Science Domains and Beyond