mlsanityData to 7 Oct 2026

HelpSteer2: Open-source dataset for training top-performing reward models

arXiv 2406.08673 ↗First posted 12 Jun 2024cs.CL cs.AI cs.LGcited by 232

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

High-quality preference datasets are essential for training reward models that can effectively guide large language models (LLMs) in generating high-quality responses aligned with human preferences. As LLMs become stronger and better aligned, permissively licensed preference datasets, such as Open Assistant, HH-RLHF, and HelpSteer need to be updated to remain effective for reward modeling. Methods that distil preference data from proprietary LLMs such as GPT-4 have restrictions on commercial usage imposed by model providers. To improve upon both generated responses and attribute labeling quality, we release HelpSteer2, a permissively licensed preference dataset (CC-BY-4.0). Using a powerful internal base model trained on HelpSteer2, we are able to achieve the SOTA score (92.0%) on Reward-Bench's primary dataset, outperforming currently listed open and proprietary models, as of June 12th, 2024. Notably, HelpSteer2 consists of only ten thousand response pairs, an order of magnitude fewer than existing preference datasets (e.g., HH-RLHF), which makes it highly efficient for training reward models. Our extensive experiments demonstrate that reward models trained with HelpSteer2 are effective in aligning LLMs. In particular, we propose SteerLM 2.0, a model alignment approach that can effectively make use of the rich multi-attribute score predicted by our reward models. HelpSteer2 is available at https://huggingface.co/datasets/nvidia/HelpSteer2 and code is available at https://github.com/NVIDIA/NeMo-Aligner

Key facts. Cited by 232 papers in the map (data through 7 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 26 of its 51 references that are in the map. First posted on arXiv 12 Jun 2024.

Builds on

26 of 51 references.

Closest ancestors · 7

Same direction, newest first

  1. 2024BRAIn: Bayesian Reward-conditioned Amortized Inference for natural language generation from feedback4 Feb
  2. 2023The Alignment Ceiling: Objective Mismatch in Reinforcement Learning from Human Feedback31 Oct
  3. 2023SteerLM: Attribute Conditioned SFT as an (User-Steerable) Alternative to RLHF9 Oct
  4. 2023A Long Way to Go: Investigating Length Correlations in RLHF5 Oct
  5. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May
  6. 2021WebGPT: Browser-assisted question-answering with human feedback17 Dec
  7. 2017Deep reinforcement learning from human preferences12 Jun

Other references · 19

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  3. 2023Let's Verify Step by Stepcited by 3,598
  4. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
  5. 2020Learning to summarize from human feedbackcited by 2,562
  6. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
  7. 2016Enriching Word Vectors with Subword Informationcited by 1,578
  8. 2024SimPO: Simple Preference Optimization with a Reference-Free Rewardcited by 951
  9. 2024Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluatorscited by 865
  10. 2023Enhancing Chat Language Models by Scaling High-quality Instructional Conversationscited by 825
  11. 2022BERTopic: Neural topic modeling with a class-based TF-IDF procedurecited by 779
  12. 2024Yi: Open Foundation Models by 01.AIcited by 706
7 more
  1. 2021FinQA: A Dataset of Numerical Reasoning over Financial Datacited by 552
  2. 2015Compositional Semantic Parsing on Semi-Structured Tablescited by 509
  3. 2023NeMo Guardrails: A Toolkit for Controllable and Safe LLM Applications with Programmable Railscited by 347
  4. 2021Understanding Dataset Difficulty with V-Usable Informationcited by 288
  5. 2023SciBench: Evaluating College-Level Scientific Problem-Solving Abilities of Large Language Modelscited by 246
  6. 2023#InsTag: Instruction Tagging for Analyzing Supervised Fine-tuning of Large Language Modelscited by 108
  7. 2023ARB: Advanced Reasoning Benchmark for Large Language Modelscited by 60

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 50; 2025: 111; 2026: 71.

232 papers.

Sort:

By area:

  1. 2025s1: Simple test-time scalingcited by 1,311
  2. 2024Tulu 3: Pushing Frontiers in Open Language Model Post-Trainingcited by 988
  3. 2025Scaling up Test-Time Compute with Latent Reasoning: A Recurrent Depth Approachcited by 372
  4. 2024Skywork-Reward: Bag of Tricks for Reward Modeling in LLMscited by 322
  5. 2024JudgeBench: A Benchmark for Evaluating LLM-based Judgescited by 294
  6. 2024OLMoE: Open Mixture-of-Experts Language Modelscited by 291
  7. 2025Improving Video Generation with Human Feedbackcited by 267
  8. 2025Skywork-Reward-V2: Scaling Preference Data Curation via Human-AI Synergycited by 204
  9. 2024Meta-Rewarding Language Models: Self-Improving Alignment with LLM-as-a-Meta-Judgecited by 202
  10. 2025Llama-Nemotron: Efficient Reasoning Modelscited by 151
  11. 2025RewardBench 2: Advancing Reward Model Evaluationcited by 136
  12. 2024HelpSteer2-Preference: Complementing Ratings with Preferencescited by 129
38 more
  1. 2025OpenRubrics: Towards Scalable Synthetic Rubric Generation for Reward Modeling and LLM Alignmentcited by 118
  2. 2024Critique-out-Loud Reward Modelscited by 105
  3. 2024Taming Overconfidence in LLMs: Reward Calibration in RLHFcited by 105
  4. 2025Nemotron-H: A Family of Accurate and Efficient Hybrid Mamba-Transformer Modelscited by 88
  5. 2025Learning to Plan & Reason for Evaluation with Thinking-LLM-as-a-Judgecited by 75
  6. 2024Foundational Autoraters: Taming Large Language Models for Better Automatic Evaluationcited by 72
  7. 2024Uncertainty-aware Reward Model: Teaching Reward Models to Know What is Unknowncited by 71
  8. 2025HelpSteer3-Preference: Open Human-Annotated Preference Data across Diverse Tasks and Languagescited by 71
  9. 2024Self-Taught Evaluatorscited by 70
  10. 2024Self-Generated Critiques Boost Reward Modeling for Language Modelscited by 69
  11. 2024Reinforcement Learning Enhanced LLMs: A Surveycited by 69
  12. 2025R1-Reward: Training Multimodal Reward Model Through Stable Reinforcement Learningcited by 65
  13. 2024RMB: Comprehensively Benchmarking Reward Models in LLM Alignmentcited by 60
  14. 2025KVLink: Accelerating Large Language Models via Efficient KV Cache Reusecited by 60
  15. 2024PAD: Personalized Alignment of LLMs at Decoding-Timecited by 58
  16. 2024DailyDilemmas: Revealing Value Preferences of LLMs with Quandaries of Daily Lifecited by 57
  17. 2024FuseChat: Knowledge Fusion of Chat Modelscited by 53
  18. 2025A Comprehensive Survey of Reward Models: Taxonomy, Applications, Challenges, and Futurecited by 53
  19. 2025Reinforcement Learning from Human Feedbackcited by 53
  20. 2025No Free Labels: Limitations of LLM-as-a-Judge Without Human Groundingcited by 50
  21. 2025Agentic Reward Modeling: Integrating Human Preferences with Verifiable Correctness Signals for Reliable Reward Systemscited by 49
  22. 2024Immune: Improving Safety Against Jailbreaks in Multi-modal LLMs via Inference-Time Alignmentcited by 42
  23. 2024A Survey on Data Synthesis and Augmentation for Large Language Modelscited by 40
  24. 2025A Survey on Personalized Alignment -- The Missing Piece for Large Language Models in Real-World Applicationscited by 40
  25. 2025Tower+: Bridging Generality and Translation Specialization in Multilingual LLMscited by 40
  26. 2025Improving LLM-as-a-Judge Inference with the Judgment Distributioncited by 36
  27. 2025Improve LLM-as-a-Judge Ability as a General Abilitycited by 35
  28. 2024Direct Judgement Preference Optimizationcited by 33
  29. 2024Sharp Analysis for KL-Regularized Contextual Bandits and RLHFcited by 32
  30. 2024AI Safety in Generative AI Large Language Models: A Surveycited by 31
  31. 2025Skywork-VL Reward: An Effective Reward Model for Multimodal Understanding and Reasoningcited by 30
  32. 2026Reward Hacking in the Era of Large Models: Mechanisms, Emergent Misalignment, Challengescited by 30
  33. 2025From 1,000,000 Users to Every User: Scaling Up Personalized Preference for User-level Alignmentcited by 28
  34. 2024A Comprehensive Survey of Direct Preference Optimization: Datasets, Theories, Variants, and Applicationscited by 27
  35. 2025RewardAnything: Generalizable Principle-Following Reward Modelscited by 27
  36. 2026Open Rubric System: Scaling Reinforcement Learning with Pairwise Adaptive Rubriccited by 26
  37. 2024Hybrid Preferences: Learning to Route Instances for Human vs. AI Feedbackcited by 25
  38. 2025reWordBench: Benchmarking and Improving the Robustness of Reward Models with Transformed Inputscited by 25

50 most cited of 232 All 232 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Boosting Reward Model with Preference-Conditional Multi-Aspect Synthetic Data Generation
  2. 2024RewardBench: Evaluating Reward Models for Language Modeling
  3. 2026DT2IT-MRM: Debiased Preference Construction and Iterative Training for Multimodal Reward Modeling
  4. 2024Reward Difference Optimization For Sample Reweighting In Offline RLHF
  5. 2026BiasReducer: Adaptive Bias Mitigation for Reward Models
  6. 2024Secrets of RLHF in Large Language Models Part II: Reward Modeling
  7. 2025EVALUESTEER: Measuring Reward Model Steerability Towards Values and Preferences
  8. 2024Towards Comprehensive Preference Data Collection for Reward Modeling