mlsanityData to 14 Oct 2026

A Systematic Analysis of Base Model Choice for Reward Modeling

arXiv 2505.10775 ↗First posted 16 May 2025cs.CL cs.AIcited by 0

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

Reinforcement learning from human feedback (RLHF) and, at its core, reward modeling have become a crucial part of training powerful large language models (LLMs). One commonly overlooked factor in training high-quality reward models (RMs) is the effect of the base model, which is becoming more challenging to choose given the rapidly growing pool of LLMs. In this work, we present a systematic analysis of the effect of base model selection on reward modeling performance. Our results show that the performance can be improved by up to 14% compared to the most common (i.e., default) choice. Moreover, we showcase the strong statistical relation between some existing benchmarks and downstream performances. We also demonstrate that the results from a small set of benchmarks could be combined to boost the model selection (+18% on average in the top 5-10). Lastly, we illustrate the impact of different post-training steps on the final performance and explore using estimated data distributions to reduce performance prediction error.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 54 of its 82 references that are in the map. First posted on arXiv 16 May 2025.

Builds on

54 of 82 references.

Closest ancestors · 11

Same direction, newest first

  1. 2024Skywork-Reward: Bag of Tricks for Reward Modeling in LLMs24 Oct
  2. 2024RM-Bench: Benchmarking Reward Models of Language Models with Subtlety and Style21 Oct
  3. 2024M-RewardBench: Evaluating Reward Models in Multilingual Settings20 Oct
  4. 2024HelpSteer2-Preference: Complementing Ratings with Preferences2 Oct
  5. 2024Uncertainty-aware Reward Model: Teaching Reward Models to Know What is Unknown1 Oct
  6. 2024Quantile Regression for Distributional Reward Models in RLHF16 Sep
  7. 2024Interpretable Preferences via Multi-Objective Reward Modeling and Mixture-of-Experts18 Jun
  8. 2024HelpSteer2: Open-source dataset for training top-performing reward models12 Jun
  9. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May
  10. 2022Scaling Laws for Reward Model Overoptimization19 Oct
  11. 2019Fine-Tuning Language Models from Human Preferences18 Sep

Other references · 43

  1. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  2. 2022Training language models to follow instructions with human feedbackcited by 15,971
  3. 2017Proximal Policy Optimization Algorithmscited by 15,215
  4. 2024The Llama 3 Herd of Modelscited by 14,715
  5. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  6. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  7. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,886
  8. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,100
  9. 2021Program Synthesis with Large Language Modelscited by 3,637
  10. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,455
  11. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 3,050
  12. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,975
31 more
  1. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,955
  2. 2019PIQA: Reasoning about Physical Commonsense in Natural Languagecited by 2,883
  3. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,879
  4. 2020Learning to summarize from human feedbackcited by 2,588
  5. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,183
  6. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,164
  7. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,917
  8. 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,848
  9. 2018CommonsenseQA: A Question Answering Challenge Targeting Commonsense Knowledgecited by 1,822
  10. 2022Challenging BIG-Bench Tasks and Whether Chain-of-Thought Can Solve Themcited by 1,807
  11. 2018Know What You Don't Know: Unanswerable Questions for SQuADcited by 1,554
  12. 2023Is Your Code Generated by ChatGPT Really Correct? Rigorous Evaluation of Large Language Models for Code Generationcited by 1,391
  13. 2020What Disease does this Patient Have? A Large-scale Open Domain Question Answering Dataset from Medical Examscited by 1,356
  14. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,346
  15. 2019DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphscited by 903
  16. 2023AGIEval: A Human-Centric Benchmark for Evaluating Foundation Modelscited by 716
  17. 2020CrowS-Pairs: A Challenge Dataset for Measuring Social Biases in Masked Language Modelscited by 710
  18. 2021A Dataset of Information-Seeking Questions and Answers Anchored in Research Paperscited by 486
  19. 2022ToxiGen: A Large-Scale Machine-Generated Dataset for Adversarial and Implicit Hate Speech Detectioncited by 422
  20. 2023UltraFeedback: Boosting Language Models with Scaled AI Feedbackcited by 406
  21. 2024Magpie: Alignment Data Synthesis from Scratch by Prompting Aligned LLMs with Nothingcited by 346
  22. 2023HelpSteer: Multi-attribute Helpfulness Dataset for SteerLMcited by 142
  23. 2022RL with KL penalties is better viewed as Bayesian inferencecited by 135
  24. 2023LLM360: Towards Fully Transparent Open-Source LLMscited by 104
  25. 2024Pretraining Data Detection for Large Language Models: A Divergence-based Calibration Methodcited by 55
  26. 2024MARS: Meaning-Aware Response Scoring for Uncertainty Estimation in Generative LLMscited by 46
  27. 2020ANLIzing the Adversarial Natural Language Inference Datasetcited by 36
  28. 2024Sloth: scaling laws for LLM skills to predict multi-benchmark performance across familiescited by 24
  29. 2024Detecting Training Data of Large Language Models via Expectation Maximizationcited by 13
  30. 2024Adaptive Pre-training Data Detection for Large Language Models via Surprising Tokenscited by 11
  31. 2023Learning Multilingual Sentence Representations with Cross-lingual Consistency Regularizationcited by 7

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024Secrets of RLHF in Large Language Models Part II: Reward Modeling
  2. 2024Towards Understanding the Influence of Reward Margin on Preference Model Performance
  3. 2024How to Evaluate Reward Models for RLHF
  4. 2024RLHF Deciphered: A Critical Analysis of Reinforcement Learning from Human Feedback for LLMs
  5. 2026Reinforcement Learning from Human Feedback: A Statistical Perspective
  6. 2024Does RLHF Scale? Exploring the Impacts From Data, Model, and Method
  7. 2025Rethinking Reward Model Evaluation Through the Lens of Reward Overoptimization
  8. 2024Improving Reinforcement Learning from Human Feedback Using Contrastive Rewards