mlsanityData to 7 Oct 2026

HelpSteer2-Preference: Complementing Ratings with Preferences

arXiv 2410.01257 ↗First posted 2 Oct 2024cs.LG cs.AI cs.CLcited by 129

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

Reward models are critical for aligning models to follow instructions, and are typically trained following one of two popular paradigms: Bradley-Terry style or Regression style. However, there is a lack of evidence that either approach is better than the other, when adequately matched for data. This is primarily because these approaches require data collected in different (but incompatible) formats, meaning that adequately matched data is not available in existing public datasets. To tackle this problem, we release preference annotations (designed for Bradley-Terry training) to complement existing ratings (designed for Regression style training) in the HelpSteer2 dataset. To improve data interpretability, preference annotations are accompanied with human-written justifications. Using this data, we conduct the first head-to-head comparison of Bradley-Terry and Regression models when adequately matched for data. Based on insights derived from such a comparison, we propose a novel approach to combine Bradley-Terry and Regression reward modeling. A Llama-3.1-70B-Instruct model tuned with this approach scores 94.1 on RewardBench, emerging top of more than 140 reward models as of 1 Oct 2024. This reward model can then be used with REINFORCE algorithm (RLHF) to align an Instruct model to reach 85.0 on Arena Hard, which is No. 1 as of 1 Oct 2024. We open-source this dataset (CC-BY-4.0 license) at https://huggingface.co/datasets/nvidia/HelpSteer2#preferences-new -- 1-oct-2024 and openly release the trained Reward and Instruct models at https://huggingface.co/nvidia/Llama-3.1-Nemotron-70B-Reward and https://huggingface.co/nvidia/Llama-3.1-Nemotron-70B-Instruct

Key facts. Cited by 129 papers in the map (data through 7 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 15 of its 34 references that are in the map. First posted on arXiv 2 Oct 2024.

Builds on

15 of 34 references.

Closest ancestors · 5

Same direction, newest first

  1. 2024Towards Data-Centric RLHF: Simple Metrics for Preference Dataset Comparison15 Sep
  2. 2024Interpretable Preferences via Multi-Objective Reward Modeling and Mixture-of-Experts18 Jun
  3. 2024HelpSteer2: Open-source dataset for training top-performing reward models12 Jun
  4. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May
  5. 2021WebGPT: Browser-assisted question-answering with human feedback17 Dec

Other references · 10

  1. 2017Proximal Policy Optimization Algorithmscited by 14,441
  2. 2024SimPO: Simple Preference Optimization with a Reference-Free Rewardcited by 951
  3. 2024Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluatorscited by 865
  4. 2023Evaluating Large Language Models at Evaluating Instruction Followingcited by 290
  5. 2021Understanding Dataset Difficulty with V-Usable Informationcited by 288
  6. 2024Arithmetic Control of LLMs for Diverse User Preferences: Directional Preference Alignment with Multi-Objective Rewardscited by 145
  7. 2023HelpSteer: Multi-attribute Helpfulness Dataset for SteerLMcited by 140
  8. 2024OffsetBias: Leveraging Debiased Data for Tuning Evaluatorscited by 94
  9. 2020DisCor: Corrective Feedback in Reinforcement Learning via Distribution Correctioncited by 71
  10. 2024Direct Judgement Preference Optimizationcited by 33

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 15; 2025: 84; 2026: 30.

129 papers.

Sort:

By area:

  1. 2024Qwen2.5 Technical Reportcited by 4,713
  2. 2024Tulu 3: Pushing Frontiers in Open Language Model Post-Trainingcited by 988
  3. 2024LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation Methodscited by 451
  4. 2024LongBench v2: Towards Deeper Understanding and Reasoning on Realistic Long-context Multitaskscited by 367
  5. 2025Improving Video Generation with Human Feedbackcited by 267
  6. 2025Skywork-Reward-V2: Scaling Preference Data Curation via Human-AI Synergycited by 204
  7. 2025RewardBench 2: Advancing Reward Model Evaluationcited by 136
  8. 2025Checklists Are Better Than Reward Models For Aligning Language Modelscited by 115
  9. 2025Multi-Agent Collaboration via Evolving Orchestrationcited by 102
  10. 2025TRAIL: Trace Reasoning and Agentic Issue Localizationcited by 98
  11. 2024VL-RewardBench: A Challenging Benchmark for Vision-Language Generative Reward Modelscited by 85
  12. 2024AceMath: Advancing Frontier Math Reasoning with Post-Training and Reward Modelingcited by 76
38 more
  1. 2025Learning to Plan & Reason for Evaluation with Thinking-LLM-as-a-Judgecited by 75
  2. 2025InternLM-XComposer2.5-Reward: A Simple Yet Effective Multi-Modal Reward Modelcited by 73
  3. 2025HelpSteer3-Preference: Open Human-Annotated Preference Data across Diverse Tasks and Languagescited by 71
  4. 2024Self-Generated Critiques Boost Reward Modeling for Language Modelscited by 69
  5. 2025NoveltyBench: Evaluating Language Models for Humanlike Diversitycited by 64
  6. 2024DailyDilemmas: Revealing Value Preferences of LLMs with Quandaries of Daily Lifecited by 57
  7. 2025No Free Labels: Limitations of LLM-as-a-Judge Without Human Groundingcited by 50
  8. 2025Rethinking External Slow-Thinking: From Snowball Errors to Probability of Correct Reasoningcited by 36
  9. 2025Improving LLM-as-a-Judge Inference with the Judgment Distributioncited by 36
  10. 2025Chasing Moving Targets with Online Self-Play Reinforcement Learning for Safer Language Modelscited by 33
  11. 2025MicroVQA: A Multimodal Reasoning Benchmark for Microscopy-Based Scientific Researchcited by 30
  12. 2026Open Rubric System: Scaling Reinforcement Learning with Pairwise Adaptive Rubriccited by 26
  13. 2024Hybrid Preferences: Learning to Route Instances for Human vs. AI Feedbackcited by 25
  14. 2024U-MATH: A University-Level Benchmark for Evaluating Mathematical Skills in LLMscited by 24
  15. 2025OpenCodeReasoning-II: A Simple Test Time Scaling Approach via Self-Critiquecited by 24
  16. 2024RAG-RewardBench: Benchmarking Reward Models in Retrieval Augmented Generation for Preference Alignmentcited by 22
  17. 2025R3: Robust Rubric-Agnostic Reward Modelscited by 22
  18. 2025LEMMA: Learning from Errors for MatheMatical Advancement in LLMscited by 20
  19. 2025Think-J: Learning to Think for Generative LLM-as-a-Judgecited by 20
  20. 2025VLRMBench: A Comprehensive and Challenging Benchmark for Vision-Language Reward Modelscited by 19
  21. 2024Puzzle: Distillation-Based NAS for Inference-Optimized LLMscited by 17
  22. 2025Adversarial Training of Reward Modelscited by 17
  23. 2025HelpSteer3: Human-Annotated Feedback and Edit Data to Empower Inference-Time Scaling in Open-Ended General-Domain Taskscited by 16
  24. 2025Think-RM: Enabling Long-Horizon Reasoning in Generative Reward Modelscited by 15
  25. 2025Alignment and Safety in Large Language Models: Safety Mechanisms, Training Paradigms, and Emerging Challengescited by 14
  26. 2025Beyond the Surface: Enhancing LLM-as-a-Judge Alignment with Human via Internal Representationscited by 14
  27. 2025Krikri: Advancing Open Large Language Models for Greekcited by 13
  28. 2025Ask a Strong LLM Judge when Your Reward Model is Uncertaincited by 13
  29. 2025ARM-Thinker: Reinforcing Multimodal Generative Reward Models with Agentic Tool Use and Visual Reasoningcited by 13
  30. 2024CoPS: Empowering LLM Agents with Provable Cross-Task Experience Sharingcited by 12
  31. 2025Can LLMs Predict Their Own Failures? Self-Awareness via Internal Circuitscited by 12
  32. 2025Persona-judge: Personalized Alignment of Large Language Models via Token-level Self-judgmentcited by 11
  33. 2025Reward-aware Preference Optimization: A Unified Mathematical Framework for Model Alignmentcited by 10
  34. 2025FusionFactory: Fusing LLM Capabilities with Multi-LLM Log Datacited by 10
  35. 2026GraphPlanner: Graph Memory-Augmented Agentic Routing for Multi-Agent LLMscited by 10
  36. 2025Preference Learning Unlocks LLMs' Psycho-Counseling Skillscited by 9
  37. 2025Towards Reward Fairness in RLHF: From a Resource Allocation Perspectivecited by 9
  38. 2025Can Reasoning Help Large Language Models Capture Human Annotator Disagreement?cited by 9

50 most cited of 129 All 129 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Reward Learning From Preference With Ties
  2. 2024Rethinking Bradley-Terry Models in Preference-Based Reward Modeling: Foundations, Theory, and Alternatives
  3. 2024Diverging Preferences: When do Annotators Disagree and do Models Know?
  4. 2024Dr. SoW: Density Ratio of Strong-over-weak LLMs for Reducing the Cost of Human Annotation in Preference Tuning
  5. 2026What Does Preference Learning Recover from Pairwise Comparison Data?
  6. 2024Beyond Bradley-Terry Models: A General Preference Model for Language Model Alignment
  7. 2025Debiasing Reward Models by Representation Learning with Guarantees
  8. 2026Beyond Binary Preferences: A Principled Framework for Reward Modeling with Ordinal Feedback