mlsanityData to 14 Oct 2026

HelpSteer3-Preference: Open Human-Annotated Preference Data across Diverse Tasks and Languages

arXiv 2505.11475 ↗First posted 16 May 2025cs.CL cs.AI cs.LGcited by 75

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

Preference datasets are essential for training general-domain, instruction-following language models with Reinforcement Learning from Human Feedback (RLHF). Each subsequent data release raises expectations for future data collection, meaning there is a constant need to advance the quality and diversity of openly available preference data. To address this need, we introduce HelpSteer3-Preference, a permissively licensed (CC-BY-4.0), high-quality, human-annotated preference dataset comprising of over 40,000 samples. These samples span diverse real-world applications of large language models (LLMs), including tasks relating to STEM, coding and multilingual scenarios. Using HelpSteer3-Preference, we train Reward Models (RMs) that achieve top performance on RM-Bench (82.4%) and JudgeBench (73.7%). This represents a substantial improvement (~10% absolute) over the previously best-reported results from existing RMs. We demonstrate HelpSteer3-Preference can also be applied to train Generative RMs and how policy models can be aligned with RLHF using our RMs. Dataset (CC-BY-4.0): https://huggingface.co/datasets/nvidia/HelpSteer3#preference Models (NVIDIA Open Model): https://huggingface.co/collections/nvidia/reward-models-68377c5955575f71fcc7a2a3

Key facts. Cited by 75 papers in the map (data through 14 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 35 of its 64 references that are in the map. First posted on arXiv 16 May 2025.

Builds on

35 of 64 references.

Closest ancestors · 10

Same direction, newest first

  1. 2025Inference-Time Scaling for Generalist Reward Modeling3 Apr
  2. 2025reWordBench: Benchmarking and Improving the Robustness of Reward Models with Transformed Inputs14 Mar
  3. 2024Self-Generated Critiques Boost Reward Modeling for Language Models25 Nov
  4. 2024RM-Bench: Benchmarking Reward Models of Language Models with Subtlety and Style21 Oct
  5. 2024HelpSteer2-Preference: Complementing Ratings with Preferences2 Oct
  6. 2024Critique-out-Loud Reward Models21 Aug
  7. 2024HelpSteer2: Open-source dataset for training top-performing reward models12 Jun
  8. 2024Improving Reward Models with Synthetic Critiques31 May
  9. 2024Iterative Data Smoothing: Mitigating Reward Overfitting and Overoptimization in RLHF29 Jan
  10. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May

Other references · 25

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  3. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  4. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  5. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  6. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  7. 2021Program Synthesis with Large Language Modelscited by 3,637
  8. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,975
  9. 2020Learning to summarize from human feedbackcited by 2,588
  10. 2023CAMEL: Communicative Agents for "Mind" Exploration of Large Language Model Societycited by 1,395
  11. 2024SimPO: Simple Preference Optimization with a Reference-Free Rewardcited by 963
  12. 2024Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluatorscited by 897
13 more
  1. 2023Enhancing Chat Language Models by Scaling High-quality Instructional Conversationscited by 839
  2. 2024WildGuard: Open One-Stop Moderation Tools for Safety Risks, Jailbreaks, and Refusals of LLMscited by 496
  3. 2024Generative Verifiers: Reward Modeling as Next-Token Predictioncited by 480
  4. 2023LMSYS-Chat-1M: A Large-Scale Real-World LLM Conversation Datasetcited by 461
  5. 2024Magpie: Alignment Data Synthesis from Scratch by Prompting Aligned LLMs with Nothingcited by 346
  6. 2024JudgeBench: A Benchmark for Evaluating LLM-based Judgescited by 318
  7. 2021Understanding Dataset Difficulty with V-Usable Informationcited by 291
  8. 2024WildBench: Benchmarking LLMs with Challenging Tasks from Real Users in the Wildcited by 218
  9. 2024Aya Dataset: An Open-Access Collection for Multilingual Instruction Tuningcited by 177
  10. 2023HelpSteer: Multi-attribute Helpfulness Dataset for SteerLMcited by 142
  11. 2024Granite Code Models: A Family of Open Foundation Models for Code Intelligencecited by 117
  12. 2024OffsetBias: Leveraging Debiased Data for Tuning Evaluatorscited by 94
  13. 2025OpenCodeInstruct: A Large-scale Instruction Tuning Dataset for Code LLMscited by 60

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 27; 2026: 48.

75 papers.

Sort:

By area:

  1. 2025OpenRubrics: Towards Scalable Synthetic Rubric Generation for Reward Modeling and LLM Alignmentcited by 132
  2. 2025NVIDIA Nemotron Nano 2: An Accurate and Efficient Hybrid Mamba-Transformer Reasoning Modelcited by 108
  3. 2025Nemotron 3 Nano: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoningcited by 104
  4. 2025Nemotron-Cascade: Scaling Cascaded Reinforcement Learning for General-Purpose Reasoning Modelscited by 48
  5. 2026Nemotron 3 Ultra: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoningcited by 42
  6. 2026Nemotron 3 Super: Open, Efficient Mixture-of-Experts Hybrid Mamba-Transformer Model for Agentic Reasoningcited by 38
  7. 2025Auto-Rubric: Learning From Implicit Weights to Explicit Rubrics for Reward Modelingcited by 30
  8. 2025What's In My Human Feedback? Learning Interpretable Descriptions of Preference Datacited by 26
  9. 2025R3: Robust Rubric-Agnostic Reward Modelscited by 23
  10. 2025Incentivizing Agentic Reasoning in LLM Judges via Tool-Integrated Reinforcement Learningcited by 17
  11. 2026CDRRM: Contrast-Driven Rubric Generation for Reliable and Interpretable Reward Modelingcited by 12
  12. 2026Reward Modeling from Natural Language Human Feedbackcited by 11
38 more
  1. 2026Outcome Accuracy is Not Enough: Aligning the Reasoning Process of Reward Modelscited by 11
  2. 2026Co-Evolving LLM Evaluators and Policies via DynamicRubriccited by 10
  3. 2026Toward Robust LLM-Based Judges: Taxonomic Bias Evaluation and Debiasing Optimizationcited by 9
  4. 2025RLBFF: Binary Flexible Feedback to bridge between Human Feedback & Verifiable Rewardscited by 8
  5. 2025Who's Your Judge? On the Detectability of LLM-Generated Judgmentscited by 8
  6. 2025mR3: Multilingual Rubric-Agnostic Reward Reasoning Modelscited by 8
  7. 2025Enhancing Large Language Model Reasoning with Reward Models: An Analytical Surveycited by 8
  8. 2025GRAM-R2: Self-Training Generative Foundation Reward Models for Reward Reasoningcited by 7
  9. 2025Think Twice: Branch-and-Rethink Reasoning Reward Modelcited by 7
  10. 2025Eliminating Inductive Bias in Reward Models with Information-Theoretic Guidancecited by 7
  11. 2026The Model Organism Lottery: Model Organism Interpretability Strongly Depends on Training Methodologycited by 6
  12. 2025URPO: A Unified Reward & Policy Optimization Framework for Large Language Modelscited by 5
  13. 2025S2J: Bridging the Gap Between Solving and Judging Ability in Generative Reward Modelscited by 5
  14. 2025Foundational Automatic Evaluators: Scaling Multi-Task Generative Evaluator Training for Reasoning-Centric Domainscited by 5
  15. 2026MSRL: Scaling Generative Multimodal Reward Modeling via Multi-Stage Reinforcement Learningcited by 5
  16. 2026RUBRIC-ARROW: Alternating Pointwise Rubric Reward Modeling for LLM Post-training in Non-verifiable Domainscited by 5
  17. 2025Aligning Multilingual Reasoning with Verifiable Semantics from a High-Resource Expert Modelcited by 4
  18. 2025Reward Model Routing in Alignmentcited by 4
  19. 2026ZAYA1-8B Technical Reportcited by 4
  20. 2026PReMISE: Policy Rubrics as Measurement Specifications for LLM Judgescited by 4
  21. 2025Decoding Alignment: A Critical Survey of LLM Development Initiatives through Value-setting and Data-centric Lenscited by 3
  22. 2025Compass-v3: Scaling Domain-Specific LLMs for Multilingual E-Commerce in Southeast Asiacited by 3
  23. 2025From <Answer> to <Think>: Multidimensional Supervision of Reasoning Process for LLM Optimizationcited by 3
  24. 2026R-Align: Enhancing Generative Reward Models through Rationale-Centric Meta-Judgingcited by 3
  25. 2026WildReward: Learning Reward Models from In-the-Wild Human Interactionscited by 3
  26. 2026CAMEL: Confidence-Gated Reflection for Reward Modelingcited by 3
  27. 2025Evaluating and Improving Cultural Awareness of Reward Models for LLM Alignmentcited by 2
  28. 2025T-pro 2.0: An Efficient Russian Hybrid-Reasoning Model and Playgroundcited by 2
  29. 2026ClinAlign: Scaling Healthcare Alignment from Clinician Preferencecited by 2
  30. 2026DynaCF: Mitigating Shortcut Learning in Reward Models via Dynamic Counterfactual Sensitivitycited by 2
  31. 2025When Data is the Algorithm: A Systematic Study and Curation of Preference Optimization Datasetscited by 1
  32. 2026IRPM: Intergroup Relative Preference Modeling for Pointwise Generative Reward Modelscited by 1
  33. 2026Controlling Multimodal Conversational Agents with Coverage-Enhanced Latent Actionscited by 1
  34. 2026AdaJudge: Adaptive Multi-Perspective Judging for Reward Modelingcited by 1
  35. 2026Owen-Shapley Policy Optimization: A Principled RL Algorithm for Generative Search LLMscited by 1
  36. 2026Judging What We Cannot Solve: A Consequence-Based Approach for Oracle-Free Evaluation of Research-Level Mathcited by 1
  37. 2026Beyond Length Scaling: Synergizing Breadth and Depth for Generative Reward Modelscited by 1
  38. 2026EstLLM: Enhancing Estonian Capabilities in Multilingual LLMs via Continued Pretraining and Post-Trainingcited by 1

50 most cited of 75 All 75 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Towards Data-Centric RLHF: Simple Metrics for Preference Dataset Comparison
  2. 2024Data-Centric Human Preference with Rationales for Direct Preference Alignment
  3. 2024West-of-N: Synthetic Preferences for Self-Improving Reward Models
  4. 2023Safer-Instruct: Aligning Language Models with Automated Preference Data
  5. 2024A Systematic Examination of Preference Learning through the Lens of Instruction-Following
  6. 2024Towards Comprehensive Preference Data Collection for Reward Modeling
  7. 2024Preference Tuning with Human Feedback on Language, Speech, and Vision Tasks: A Survey
  8. 2025Skywork-Reward-V2: Scaling Preference Data Curation via Human-AI Synergy