mlsanityData to 14 Oct 2026

Why is Your Language Model a Poor Implicit Reward Model?

arXiv 2507.07981 ↗First posted 10 Jul 2025cs.CL cs.AI cs.LGcited by 6

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

Reward models are key to language model post-training and inference pipelines. Conveniently, recent work showed that every language model defines an implicit reward model (IM-RM), without requiring any architectural changes. However, such IM-RMs tend to generalize worse, especially out-of-distribution, compared to explicit reward models (EX-RMs) that apply a dedicated linear head over the hidden representations of a language model. The existence of a generalization gap is puzzling, as EX-RMs and IM-RMs are nearly identical. They can be trained using the same data, loss function, and language model, and differ only in how the reward is computed. Toward a fundamental understanding of the implicit biases underlying different reward model types, we investigate the root cause of this gap. Our main finding, backed by theory and experiments, is that IM-RMs rely more heavily on superficial token-level cues. Consequently, they often generalize worse than EX-RMs under token-level distribution shifts, as well as in-distribution. Furthermore, we provide evidence against alternative hypotheses for the generalization gap. Most notably, we challenge the claim that IM-RMs struggle in tasks where generation is harder than verification because they can operate both as a verifier and a generator. Overall, our results highlight that seemingly minor design choices can substantially impact the generalization behavior of reward models.

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 84 of its 96 references that are in the map. First posted on arXiv 10 Jul 2025.

Builds on

84 of 96 references.

Closest ancestors · 31

Same direction, newest first

  1. 2025RewardBench 2: Advancing Reward Model Evaluation2 Jun
  2. 2025Understanding the Performance Gap in Preference Learning: A Dichotomy of RLHF and DPO26 May
  3. 2025On the Robustness of Reward Models for Language Model Alignment12 May
  4. 2025What Makes a Reward Model a Good Teacher? An Optimization Perspective19 Mar
  5. 2025reWordBench: Benchmarking and Improving the Robustness of Reward Models with Transformed Inputs14 Mar
  6. 2025Can RLHF be More Efficient with Imperfect Reward Models? A Policy Coverage Perspective26 Feb
  7. 2024Skywork-Reward: Bag of Tricks for Reward Modeling in LLMs24 Oct
  8. 2024Optimal Design for Reward Modeling in RLHF22 Oct
  9. 2024RM-Bench: Benchmarking Reward Models of Language Models with Subtlety and Style21 Oct
  10. 2024How to Evaluate Reward Models for RLHF18 Oct
  11. 2024RMB: Comprehensively Benchmarking Reward Models in LLM Alignment13 Oct
  12. 2024The Accuracy Paradox in RLHF: When Better Reward Models Don't Yield Better Language Models9 Oct
19 more
  1. 2024Rethinking Reward Model Evaluation: Are We Barking up the Wrong Tree?8 Oct
  2. 2024Interpretable Preferences via Multi-Objective Reward Modeling and Mixture-of-Experts18 Jun
  3. 2024RLHF Workflow: From Reward Modeling to Online RLHF13 May
  4. 2024RewardBench: Evaluating Reward Models for Language Modeling20 Mar
  5. 2024Active Preference Optimization for Sample Efficient RLHF16 Feb
  6. 2024Exploration-Driven Policy Optimization in RLHF: Theoretical Insights on Efficient Data Utilization15 Feb
  7. 2024Reward Generalization in RLHF: A Topological Perspective15 Feb
  8. 2024Reinforcement Learning from Human Feedback with Active Queries14 Feb
  9. 2024Transforming and Combining Rewards for Aligning Large Language Models1 Feb
  10. 2023Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraint18 Dec
  11. 2023Making RL with Preference-based Feedback Efficient via Randomization23 Oct
  12. 2023Reinforced Self-Training (ReST) for Language Modeling17 Aug
  13. 2023Provable Benefits of Policy Learning from Human Preferences in Contextual Bandit Problems24 Jul
  14. 2023Is RLHF More Difficult than Standard RL?25 Jun
  15. 2023Provable Reward-Agnostic Preference-Based Reinforcement Learning29 May
  16. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May
  17. 2023RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignment13 Apr
  18. 2023Principled Reinforcement Learning with Human Feedback from Pairwise or K-wise Comparisons26 Jan
  19. 2019Fine-Tuning Language Models from Human Preferences18 Sep

Other references · 53

  1. 2017Attention Is All You Needcited by 60,361
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2022Training language models to follow instructions with human feedbackcited by 15,971
  4. 2024The Llama 3 Herd of Modelscited by 14,715
  5. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  6. 2024Qwen2.5 Technical Reportcited by 4,942
  7. 2023Let's Verify Step by Stepcited by 3,789
  8. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,218
  9. 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,998
  10. 2024Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameterscited by 1,986
  11. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,786
  12. 2023Representation Engineering: A Top-Down Approach to AI Transparencycited by 1,336
38 more
  1. 2024SimPO: Simple Preference Optimization with a Reference-Free Rewardcited by 963
  2. 2022Solving math word problems with process- and outcome-based feedbackcited by 901
  3. 2022Discovering Language Model Behaviors with Model-Written Evaluationscited by 836
  4. 2013Exact solutions to the nonlinear dynamics of learning in deep linear neural networkscited by 763
  5. 2023The Linear Representation Hypothesis and the Geometry of Large Language Modelscited by 716
  6. 2017The Implicit Bias of Gradient Descent on Separable Datacited by 655
  7. 2020Estimating Training Data Influence by Tracing Gradient Descentcited by 580
  8. 2024LESS: Selecting Influential Data for Targeted Instruction Tuningcited by 532
  9. 2024Generative Verifiers: Reward Modeling as Next-Token Predictioncited by 480
  10. 2019Implicit Regularization in Deep Matrix Factorizationcited by 405
  11. 2017Implicit Regularization in Matrix Factorizationcited by 322
  12. 2018Characterizing Implicit Bias in Terms of Optimization Geometrycited by 266
  13. 2024A Survey on Data Selection for Language Modelscited by 236
  14. 2024Nemotron-4 340B Technical Reportcited by 144
  15. 2019Implicit Regularization of Discrete Gradient Dynamics in Linear Neural Networkscited by 124
  16. 2020Implicit Regularization in Deep Learning May Not Be Explainable by Normscited by 119
  17. 2019Dynamics of stochastic gradient descent for two-layer neural networks in the teacher-student setupcited by 112
  18. 2024Learning Dynamics of LLM Finetuningcited by 95
  19. 2024Inference-Aware Fine-Tuning for Best-of-N Sampling in Large Language Modelscited by 89
  20. 2022On the Implicit Bias in Deep-Learning Algorithmscited by 86
  21. 2018An analytic theory of generalization dynamics and transfer learning in deep linear networkscited by 85
  22. 2024Unintentional Unalignment: Likelihood Displacement in Direct Preference Optimizationcited by 67
  23. 2025All Roads Lead to Likelihood: The Value of Reinforcement Learning in Fine-Tuningcited by 59
  24. 2023Provable Offline Preference-Based Reinforcement Learningcited by 55
  25. 2021Implicit Regularization in Tensor Factorizationcited by 44
  26. 2020Gradient Descent for Deep Matrix Factorization: Dynamics and Implicit Bias towards Low Rankcited by 42
  27. 2024The Implicit Bias of Adam on Separable Datacited by 36
  28. 2024InfAlign: Inference-aware language model alignmentcited by 35
  29. 2022Incremental Learning in Diagonal Linear Networkscited by 28
  30. 2022Implicit Regularization in Hierarchical Tensor Factorization and Deep Convolutional Neural Networkscited by 27
  31. 2023Vanishing Gradients in Reinforcement Finetuning of Language Modelscited by 27
  32. 2024On the Limited Generalization Capability of the Implicit Reward Model Induced by Direct Preference Optimizationcited by 27
  33. 2024Evaluating Robustness of Reward Models for Mathematical Reasoningcited by 22
  34. 2023How to prepare your task head for finetuningcited by 19
  35. 2025The Rich and the Simple: On the Implicit Bias of Adam and SGDcited by 13
  36. 2024D2PO: Discriminator-Guided DPO with Response Evaluation Modelscited by 9
  37. 2022Learning Low Dimensional State Spaces with Overparameterized Recurrent Neural Netscited by 7
  38. 2024Can DPO Learn Diverse Human Values? A Theoretical Scaling Lawcited by 7

50 most cited of 53 All 53 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 5.

6 papers.

  1. 2025SuperFlow: Training Flow Matching Models with RL on the Flycited by 10
  2. 2026Implicit Turn-Wise Policy Optimization for Proactive User-LLM Interactioncited by 4
  3. 2026Learning Dynamics in RL Post-Training for Language Modelscited by 2
  4. 2026When Good Verifiers Go Bad: Silent Negative Transfer in Verifier-Guided VLM Trainingcited by 0
  5. 2026Towards Bridging the Gap Between Offline and Iterative Alignment via Preference Distillationcited by 0
  6. 2026Improving LLM Collaboration via Multi-Agent Preference Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Reward Models Inherit Value Biases from Pretraining
  2. 2026Automatically Finding Reward Model Biases
  3. 2024Interpreting Language Reward Models via Contrastive Explanations
  4. 2024M-RewardBench: Evaluating Reward Models in Multilingual Settings
  5. 2025Reward Model Interpretability via Optimal and Pessimal Tokens
  6. 2024Improving Reward Models with Synthetic Critiques
  7. 2025RewardAnything: Generalizable Principle-Following Reward Models
  8. 2026One Bias After Another: Mechanistic Reward Shaping and Persistent Biases in Language Reward Models