mlsanityData to 7 Oct 2026

HelpSteer: Multi-attribute Helpfulness Dataset for SteerLM

arXiv 2311.09528 ↗First posted 16 Nov 2023cs.CL cs.AI cs.LGcited by 140

Not on the map. Why?

Cited by 101 in LLM alignment / RLHF / preference optimisation, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Existing open-source helpfulness preference datasets do not specify what makes some responses more helpful and others less so. Models trained on these datasets can incidentally learn to model dataset artifacts (e.g. preferring longer but unhelpful responses only due to their length). To alleviate this problem, we collect HelpSteer, a multi-attribute helpfulness dataset annotated for the various aspects that make responses helpful. Specifically, our 37k-sample dataset has annotations for correctness, coherence, complexity, and verbosity in addition to overall helpfulness of responses. Training Llama 2 70B using the HelpSteer dataset with SteerLM technique produces a model that scores 7.54 on MT Bench, which is currently the highest score for open models that do not require training data from more powerful models (e.g. GPT4). We release this dataset with CC-BY-4.0 license at https://huggingface.co/datasets/nvidia/HelpSteer

Key facts. Cited by 140 papers in the map (data through 7 Oct 2026). Builds on 19 of its 53 references that are in the map. First posted on arXiv 16 Nov 2023.

Builds on

19 of 53 references.

References · 19

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  4. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  5. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
  6. 2020Learning to summarize from human feedbackcited by 2,562
  7. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,722
  8. 2023BeaverTails: Towards Improved Safety Alignment of LLM via a Human-Preference Datasetcited by 828
  9. 2023AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedbackcited by 738
  10. 2023OpenAssistant Conversations -- Democratizing Large Language Model Alignmentcited by 712
  11. 2023Fine-Grained Human Feedback Gives Better Rewards for Language Model Trainingcited by 399
  12. 2023NeMo Guardrails: A Toolkit for Controllable and Safe LLM Applications with Programmable Railscited by 347
7 more
  1. 2021Understanding Dataset Difficulty with V-Usable Informationcited by 288
  2. 2023A Long Way to Go: Investigating Length Correlations in RLHFcited by 285
  3. 2023SteerLM: Attribute Conditioned SFT as an (User-Steerable) Alternative to RLHFcited by 107
  4. 2019EASSE: Easier Automatic Sentence Simplification Evaluationcited by 50
  5. 2023The Alignment Ceiling: Objective Mismatch in Reinforcement Learning from Human Feedbackcited by 42
  6. 2021Rethinking Automatic Evaluation in Sentence Simplificationcited by 17
  7. 2021Can Transformer Models Measure Coherence In Text? Re-Thinking the Shuffle Testcited by 17

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 44; 2025: 64; 2026: 31.

140 papers.

Sort:

By area:

  1. 2024LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation Methodscited by 451
  2. 2024Skywork-Reward: Bag of Tricks for Reward Modeling in LLMscited by 322
  3. 2024RLHF Workflow: From Reward Modeling to Online RLHFcited by 265
  4. 2024Rewards-in-Context: Multi-objective Alignment of Foundation Models with Dynamic Preference Adjustmentcited by 171
  5. 2025RM-R1: Reward Modeling as Reasoningcited by 149
  6. 2024Arithmetic Control of LLMs for Diverse User Preferences: Directional Preference Alignment with Multi-Objective Rewardscited by 145
  7. 2024Nemotron-4 340B Technical Reportcited by 143
  8. 2024HelpSteer2-Preference: Complementing Ratings with Preferencescited by 129
  9. 2025LLM Post-Training: A Deep Dive into Reasoning Large Language Modelscited by 129
  10. 2025OpenRubrics: Towards Scalable Synthetic Rubric Generation for Reward Modeling and LLM Alignmentcited by 118
  11. 2024Critique-out-Loud Reward Modelscited by 105
  12. 2024Decoding-Time Language Model Alignment with Multiple Objectivescited by 90
38 more
  1. 2023Data Diversity Matters for Robust Instruction Tuningcited by 76
  2. 2024InfoRM: Mitigating Reward Hacking in RLHF via Information-Theoretic Reward Modelingcited by 76
  3. 2024Foundational Autoraters: Taming Large Language Models for Better Automatic Evaluationcited by 72
  4. 2025HelpSteer3-Preference: Open Human-Annotated Preference Data across Diverse Tasks and Languagescited by 71
  5. 2024Self-Generated Critiques Boost Reward Modeling for Language Modelscited by 69
  6. 2024Online Iterative Reinforcement Learning from Human Feedback with General Preference Modelcited by 52
  7. 2024Rethinking Bradley-Terry Models in Preference-Based Reward Modeling: Foundations, Theory, and Alternativescited by 48
  8. 2025Reward Reasoning Modelcited by 47
  9. 2025Multi-Attribute Steering of Language Models via Targeted Interventioncited by 44
  10. 2024The Perfect Blend: Redefining RLHF with Mixture of Judgescited by 43
  11. 2025A Survey on Personalized Alignment -- The Missing Piece for Large Language Models in Real-World Applicationscited by 40
  12. 2024Cascade Reward Sampling for Efficient Decoding-Time Alignmentcited by 38
  13. 2025A Survey of Direct Preference Optimizationcited by 35
  14. 2024Direct Judgement Preference Optimizationcited by 33
  15. 2024Sharp Analysis for KL-Regularized Contextual Bandits and RLHFcited by 32
  16. 2024PoisonBench: Assessing Large Language Model Vulnerability to Data Poisoningcited by 31
  17. 2024NativQA: Multilingual Culturally-Aligned Natural Query for LLMscited by 30
  18. 2026Reward Hacking in the Era of Large Models: Mechanisms, Emergent Misalignment, Challengescited by 30
  19. 2025Amulet: ReAlignment During Test Time for Personalized Preference Adaptation of LLMscited by 29
  20. 2024A Comprehensive Survey of Direct Preference Optimization: Datasets, Theories, Variants, and Applicationscited by 27
  21. 2024Multi-modal Preference Alignment Remedies Degradation of Visual Instruction Tuning on Language Modelscited by 26
  22. 2024Distributional Preference Alignment of LLMs via Optimal Transportcited by 26
  23. 2024Hybrid Preferences: Learning to Route Instances for Human vs. AI Feedbackcited by 25
  24. 2025Equilibrate RLHF: Towards Balancing Helpfulness-Safety Trade-off in Large Language Modelscited by 22
  25. 2026RubricBench: Aligning Model-Generated Rubrics with Human Standardscited by 21
  26. 2024Performance-Aligned LLMs for Generating Fast Codecited by 20
  27. 2024Magnetic Preference Optimization: Achieving Last-iterate Convergence for Language Model Alignmentcited by 20
  28. 2024Geometric-Averaged Preference Optimization for Soft Preference Labelscited by 18
  29. 2024Cross-lingual Transfer of Reward Models in Multilingual Alignmentcited by 17
  30. 2025Adaptive Multi-Subspace Representation Steering for Attribute Alignment in Large Language Modelscited by 16
  31. 2025The Good, the Bad and the Constructive: Automatically Measuring Peer Review's Utility for Authorscited by 16
  32. 2024CantTalkAboutThis: Aligning Language Models to Stay on Topic in Dialoguescited by 15
  33. 2025Self-Improvement Towards Pareto Optimality: Mitigating Preference Conflicts in Multi-Objective Alignmentcited by 15
  34. 2025Beyond the Surface: Enhancing LLM-as-a-Judge Alignment with Human via Internal Representationscited by 14
  35. 2024Suppressing Pink Elephants with Direct Principle Feedbackcited by 13
  36. 2025Bone Soups: A Seek-and-Soup Model Merging Approach for Controllable Multi-Objective Generationcited by 13
  37. 2025PersonaFeedback: A Large-scale Human-annotated Benchmark For Personalizationcited by 13
  38. 2025Ask a Strong LLM Judge when Your Reward Model is Uncertaincited by 13

50 most cited of 140 All 140 (JSON)