mlsanityData to 7 Oct 2026

Beyond the Safety Bundle: Auditing the Helpful and Harmless Dataset

arXiv 2411.08243 ↗First posted 12 Nov 2024cs.CL cs.CYcited by 8

Not on the map. Why?

Abstract (arXiv)

In an effort to mitigate the harms of large language models (LLMs), learning from human feedback (LHF) has been used to steer LLMs towards outputs that are intended to be both less harmful and more helpful. Despite the widespread adoption of LHF in practice, the quality of this feedback and its effectiveness as a safety mitigation technique remain unclear. This study addresses these issues by auditing the widely-used Helpful and Harmless (HH) dataset by Anthropic. Our work includes: (1) a thorough investigation of the dataset's content through both manual and automated evaluation; (2) experiments demonstrating the dataset's impact on models' safety; and (3) an analysis of the 100 most influential papers citing this dataset. Through our audit, we showcase how conceptualization failures and quality issues identified in the HH dataset can create additional harms by leading to disparate safety behaviors across demographic groups. Our findings highlight the need for more nuanced, context-sensitive approaches to safety mitigation in LLMs.

Key facts. Cited by 8 papers in the map (data through 7 Oct 2026). Builds on 89 of its 126 references that are in the map. First posted on arXiv 12 Nov 2024.

Builds on

89 of 126 references.

References · 89

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  3. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  4. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  5. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  6. 2017Deep reinforcement learning from human preferencescited by 3,846
  7. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  8. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  9. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  10. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,317
  11. 2023Qwen Technical Reportcited by 3,273
  12. 2019The Curious Case of Neural Text Degenerationcited by 2,881
38 more
  1. 2020Learning to summarize from human feedbackcited by 2,562
  2. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,734
  3. 2023Jailbroken: How Does LLM Safety Training Fail?cited by 1,565
  4. 2023The Rise and Potential of Large Language Model Based Agents: A Surveycited by 1,422
  5. 2023CAMEL: Communicative Agents for "Mind" Exploration of Large Language Model Societycited by 1,348
  6. 2023Not what you've signed up for: Compromising Real-World LLM-Integrated Applications with Indirect Prompt Injectioncited by 1,339
  7. 2023FActScore: Fine-grained Atomic Evaluation of Factual Precision in Long Form Text Generationcited by 1,259
  8. 2023Inference-Time Intervention: Eliciting Truthful Answers from a Language Modelcited by 1,259
  9. 2020RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Modelscited by 1,235
  10. 2023LIMA: Less Is More for Alignmentcited by 1,080
  11. 2022Scaling Laws for Reward Model Overoptimizationcited by 1,056
  12. 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,023
  13. 2023Towards Understanding Sycophancy in Language Modelscited by 988
  14. 2022Red Teaming Language Models with Language Modelscited by 983
  15. 2023BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Datasetcited by 828
  16. 2022Discovering Language Model Behaviors with Model-Written Evaluationscited by 805
  17. 2022BERTopic: Neural topic modeling with a class-based TF-IDF procedurecited by 779
  18. 2023CRITIC: Large Language Models Can Self-Correct with Tool-Interactive Critiquingcited by 751
  19. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 743
  20. 2023The Flan Collection: Designing Data and Methods for Effective Instruction Tuningcited by 734
  21. 2023Can Large Language Models Be an Alternative to Human Evaluations?cited by 729
  22. 2023Aligning Large Multimodal Models with Factually Augmented RLHFcited by 703
  23. 2023RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignmentcited by 672
  24. 2023Capabilities of GPT-4 on Medical Challenge Problemscited by 630
  25. 2023Safe RLHF: Safe Reinforcement Learning from Human Feedbackcited by 630
  26. 2023Eliciting Latent Predictions from Transformers with the Tuned Lenscited by 579
  27. 2023GPTFUZZER: Red Teaming Large Language Models with Auto-Generated Jailbreak Promptscited by 520
  28. 2023Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervisioncited by 445
  29. 2023Benchmarking Large Language Models for News Summarizationcited by 444
  30. 2023Catastrophic Jailbreak of Open-source LLMs via Exploiting Generationcited by 439
  31. 2022ToxiGen: A Large-Scale Machine-Generated Dataset for Adversarial and Implicit Hate Speech Detectioncited by 419
  32. 2023Scalable Extraction of Training Data from (Production) Language Modelscited by 415
  33. 2023GPT-4 Is Too Smart To Be Safe: Stealthy Chat with LLMs via Ciphercited by 401
  34. 2023Fine-Grained Human Feedback Gives Better Rewards for Language Model Trainingcited by 399
  35. 2023Trustworthy LLMs: a Survey and Guideline for Evaluating Large Language Models' Alignmentcited by 394
  36. 2023Safety-Tuned LLaMAs: Lessons From Improving the Safety of Large Language Models that Follow Instructionscited by 377
  37. 2022Self-critiquing models for assisting human evaluatorscited by 372
  38. 2023Character-LLM: A Trainable Agent for Role-Playingcited by 335

50 most cited of 89 All 89 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 6; 2026: 2.

8 papers.

  1. 2025Neither Valid nor Reliable? Investigating the Use of LLMs as Judgescited by 37
  2. 2025Temporal Sparse Autoencoders: Leveraging the Sequential Nature of Language for Interpretabilitycited by 16
  3. 2025Conversational User-AI Intervention: A Study on Prompt Rewriting for Improved LLM Response Generationcited by 10
  4. 2025Scalable Valuation of Human Feedback through Provably Robust Model Alignmentcited by 4
  5. 2025Diagnosing the Performance Trade-off in Moral Alignment: A Case Study on Gender Stereotypescited by 2
  6. 2025The Problem with Safety Classification is not just the Modelscited by 1
  7. 2026The Refusal--Compliance Tradeoff: A Large-Scale Safety Behavior Audit of Large Language Modelscited by 0
  8. 2026LLJ Cards: Best practices for the Use of LLMs as Judgescited by 0