mlsanityData to 7 Oct 2026

BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Dataset

arXiv 2307.04657 ↗First posted 10 Jul 2023cs.CLcited by 828

On the map: LLM alignment / RLHF / preference optimisation → Safety alignment and jailbreak attacks

Abstract (arXiv)

In this paper, we introduce the BeaverTails dataset, aimed at fostering research on safety alignment in large language models (LLMs). This dataset uniquely separates annotations of helpfulness and harmlessness for question-answering pairs, thus offering distinct perspectives on these crucial attributes. In total, we have gathered safety meta-labels for 333,963 question-answer (QA) pairs and 361,903 pairs of expert comparison data for both the helpfulness and harmlessness metrics. We further showcase applications of BeaverTails in content moderation and reinforcement learning with human feedback (RLHF), emphasizing its potential for practical safety measures in LLMs. We believe this dataset provides vital resources for the community, contributing towards the safe development and deployment of LLMs. Our project page is available at the following URL: https://sites.google.com/view/pku-beavertails.

Key facts. Cited by 828 papers in the map (data through 7 Oct 2026). Placed in Safety alignment and jailbreak attacks, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 34 of its 70 references that are in the map. First posted on arXiv 10 Jul 2023.

Builds on

34 of 70 references.

Closest ancestors · 1

Same direction, newest first

  1. 2022Red Teaming Language Models to Reduce Harms: Methods, Scaling Behaviors, and Lessons Learned23 Aug

Other references · 33

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2017Proximal Policy Optimization Algorithmscited by 14,441
  4. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  5. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
  6. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
  7. 2022Self-Instruct: Aligning Language Models with Self-Generated Instructionscited by 2,674
  8. 2020Learning to summarize from human feedbackcited by 2,562
  9. 2022Beyond the Imitation Game: Quantifying and extrapolating the capabilities of language modelscited by 1,970
  10. 2020RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Modelscited by 1,235
  11. 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,023
  12. 2022Red Teaming Language Models with Language Modelscited by 983
21 more
  1. 2021Ethical and social risks of harm from Language Modelscited by 957
  2. 2019Solving Rubik's Cube with a Robot Handcited by 699
  3. 2021BBQ: A Hand-Built Bias Benchmark for Question Answeringcited by 683
  4. 2019Text Summarization with Pretrained Encoderscited by 553
  5. 2022Improving alignment of dialogue agents via targeted human judgementscited by 527
  6. 2022LM-Nav: Robotic Navigation with Large Pre-Trained Models of Language, Vision, and Actioncited by 435
  7. 2023Fine-Grained Human Feedback Gives Better Rewards for Language Model Trainingcited by 399
  8. 2023Toxicity in ChatGPT: Analyzing Persona-assigned Language Modelscited by 356
  9. 2022Learning robust perceptive locomotion for quadrupedal robots in the wildcited by 299
  10. 2023ChatGPT for Robotics: Design Principles and Model Abilitiescited by 296
  11. 2022GatorTron: A Large Clinical Language Model to Unlock Patient Information from Unstructured Electronic Health Recordscited by 238
  12. 2023TidyBot: Personalized Robot Assistance with Large Language Modelscited by 219
  13. 2022A New Generation of Perspective API: Efficient Multilingual Character-level Transformerscited by 181
  14. 2023Model evaluation for extreme riskscited by 167
  15. 2019Reducing Sentiment Bias in Language Models via Counterfactual Evaluationcited by 148
  16. 2023Safety Assessment of Chinese Large Language Modelscited by 82
  17. 2023The Wisdom of Hindsight Makes Language Models Better Instruction Followerscited by 59
  18. 2022Characteristics of Harmful Text: Towards Rigorous Benchmarking of Language Modelscited by 49
  19. 2023How Ready are Pre-trained Abstractive Models and LLMs for Legal Case Judgement Summarization?cited by 43
  20. 2023Proactive Multi-Camera Collaboration For 3D Human Pose Estimationcited by 13
  21. 2020AirCapRL: Autonomous Aerial Human Motion Capture using Deep Reinforcement Learningcited by 11

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 36; 2024: 259; 2025: 281; 2026: 252.

828 papers.

Sort:

By area:

  1. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,883
  2. 2024SimPO: Simple Preference Optimization with a Reference-Free Rewardcited by 951
  3. 2024Yi: Open Foundation Models by 01.AIcited by 706
  4. 2024Phi-4 Technical Reportcited by 675
  5. 2023Baichuan 2: Open Large-scale Language Modelscited by 664
  6. 2023Safe RLHF: Safe Reinforcement Learning from Human Feedbackcited by 630
  7. 2023Tree of Attacks: Jailbreaking Black-Box LLMs Automaticallycited by 619
  8. 2024Smarter, Better, Faster, Longer: A Modern Bidirectional Encoder for Fast, Memory Efficient, and Long Context Finetuning and Inferencecited by 527
  9. 2024Negative Preference Optimization: From Catastrophic Collapse to Effective Unlearningcited by 484
  10. 2024WildGuard: Open One-Stop Moderation Tools for Safety Risks, Jailbreaks, and Refusals of LLMscited by 460
  11. 2023UltraFeedback: Boosting Language Models with Scaled AI Feedbackcited by 394
  12. 2024Interpretable Preferences via Multi-Objective Reward Modeling and Mixture-of-Expertscited by 369
38 more
  1. 2024Skywork-Reward: Bag of Tricks for Reward Modeling in LLMscited by 322
  2. 2024Aya Model: An Instruction Finetuned Open-Access Multilingual Language Modelcited by 292
  3. 2023Large Language Model Unlearningcited by 288
  4. 2024A Survey on Knowledge Distillation of Large Language Modelscited by 276
  5. 2024R-Judge: Benchmarking Safety Risk Awareness for LLM Agentscited by 268
  6. 2024RLHF Workflow: From Reward Modeling to Online RLHFcited by 265
  7. 2024PKU-SafeRLHF: Towards Multi-Level Safety Alignment for LLMs with Human Preferencecited by 252
  8. 2024SALAD-Bench: A Hierarchical and Comprehensive Safety Benchmark for Large Language Modelscited by 225
  9. 2024ShieldGemma: Generative AI Content Moderation Based on Gemmacited by 224
  10. 2024OR-Bench: An Over-Refusal Benchmark for Large Language Modelscited by 222
  11. 2024Jailbreak Attacks and Defenses Against Large Language Models: A Surveycited by 215
  12. 2024VoiceBench: Benchmarking LLM-Based Voice Assistantscited by 192
  13. 2023Do-Not-Answer: A Dataset for Evaluating Safeguards in LLMscited by 176
  14. 2023Generative Judge for Evaluating Alignmentcited by 163
  15. 2024Personalizing Reinforcement Learning from Human Feedback with Variational Preference Learningcited by 147
  16. 2024Arithmetic Control of LLMs for Diverse User Preferences: Directional Preference Alignment with Multi-Objective Rewardscited by 145
  17. 2023HelpSteer: Multi-attribute Helpfulness Dataset for SteerLMcited by 140
  18. 2024Benchmark Data Contamination of Large Language Models: A Surveycited by 138
  19. 2023Safety-Gymnasium: A Unified Safe Reinforcement Learning Benchmarkcited by 135
  20. 2024Towards Safer Large Language Models through Machine Unlearningcited by 134
  21. 2025LLM Post-Training: A Deep Dive into Reasoning Large Language Modelscited by 129
  22. 2025Qwen3Guard Technical Reportcited by 125
  23. 2024Attacks, Defenses and Evaluations for LLM Conversation Safety: A Surveycited by 124
  24. 2023Black-Box Prompt Optimization: Aligning Large Language Models without Model Trainingcited by 122
  25. 2025Aegis2.0: A Diverse AI Safety Dataset and Risks Taxonomy for Alignment of LLM Guardrailscited by 122
  26. 2023Beyond One-Preference-Fits-All Alignment: Multi-Objective Direct Preference Optimizationcited by 121
  27. 2024Images are Achilles' Heel of Alignment: Exploiting Visual Vulnerabilities for Jailbreaking Multimodal Large Language Modelscited by 121
  28. 2025Safety Tax: Safety Alignment Makes Your Large Reasoning Models Less Reasonablecited by 120
  29. 2024Know Your Limits: A Survey of Abstention in Large Language Modelscited by 116
  30. 2024Vaccine: Perturbation-aware Alignment for Large Language Models against Harmful Fine-tuning Attackcited by 113
  31. 2024Relying on the Unreliable: The Impact of Language Models' Reluctance to Express Uncertaintycited by 106
  32. 2024Rule Based Rewards for Language Model Safetycited by 106
  33. 2024Taming Overconfidence in LLMs: Reward Calibration in RLHFcited by 105
  34. 2024SOUL: Unlocking the Power of Second-Order Optimization for LLM Unlearningcited by 104
  35. 2023LLM360: Towards Fully Transparent Open-Source LLMscited by 100
  36. 2024Aligner: Efficient Alignment by Learning to Correctcited by 96
  37. 2024Representation Noising: A Defence Mechanism Against Harmful Finetuningcited by 93
  38. 2024Booster: Tackling Harmful Fine-tuning for Large Language Models via Attenuating Harmful Perturbationcited by 92

50 most cited of 828 All 828 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Safety Arithmetic: A Framework for Test-time Safety Alignment of Language Models by Steering Parameters and Activations
  2. 2026Cat-DPO: Category-Adaptive Safety Alignment
  3. 2025SafeTuneBed: A Toolkit for Benchmarking LLM Safety Alignment in Fine-Tuning
  4. 2026When Choices Become Risks: Safety Failures of Large Language Models under Multiple-Choice Constraints
  5. 2025Picky LLMs and Unreliable RMs: An Empirical Study on Safety Alignment after Instruction Tuning
  6. 2026Oyster-II: Reinforcement Learning for Constructive Safety Alignment in Large Language Models
  7. 2026Can Safety Emerge from Weak Supervision? A Systematic Analysis of Small Language Models
  8. 2023Red-Teaming Large Language Models using Chain of Utterances for Safety-Alignment