Data to 7 Oct 2026

What Makes a Reward Model a Good Teacher? An Optimization Perspective

arXiv 2503.15477 ↗First posted 19 Mar 2025cs.LG cs.AI cs.CLcited by 70

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

The success of Reinforcement Learning from Human Feedback (RLHF) critically depends on the quality of the reward model. However, while this quality is primarily evaluated through accuracy, it remains unclear whether accuracy fully captures what makes a reward model an effective teacher. We address this question from an optimization perspective. First, we prove that regardless of how accurate a reward model is, if it induces low reward variance, then the RLHF objective suffers from a flat landscape. Consequently, even a perfectly accurate reward model can lead to extremely slow optimization, underperforming less accurate models that induce higher reward variance. We additionally show that a reward model that works well for one language model can induce low reward variance, and thus a flat objective landscape, for another. These results establish a fundamental limitation of evaluating reward models solely based on accuracy or independently of the language model they guide. Experiments using models of up to 8B parameters corroborate our theory, demonstrating the interplay between reward variance, accuracy, and reward maximization rate. Overall, our findings highlight that beyond accuracy, a reward model needs to induce sufficient variance for efficient optimization.

Key facts. Cited by 70 papers in the map (data through 7 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 35 of its 95 references that are in the map. First posted on arXiv 19 Mar 2025.

Builds on

35 of 95 references.

Closest ancestors · 17

Same direction, newest first

5 more

Other references · 18

2017Proximal Policy Optimization Algorithmscited by 14,441
2024The Llama 3 Herd of Modelscited by 14,192
2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,146
2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,201
2016Concrete Problems in AI Safetycited by 1,582
2024Tulu 3: Pushing Frontiers in Open Language Model Post-Trainingcited by 988
2025SFT Memorizes, RL Generalizes: A Comparative Study of Foundation Model Post-trainingcited by 655
2023Zephyr: Direct Distillation of LM Alignmentcited by 498
2024Is DPO Superior to PPO for LLM Alignment? A Comprehensive Studycited by 262
2019Neural Policy Gradient Methods: Global Optimality and Rates of Convergencecited by 157
2024Understanding the performance gap between online and offline alignment algorithmscited by 110

6 more

2023Provable Offline Preference-Based Reinforcement Learningcited by 55
2019Ray Interference: a Source of Plateaus in Deep Reinforcement Learningcited by 40
2024InfAlign: Inference-aware language model alignmentcited by 34
2024Stronger Models are NOT Stronger Teachers for Instruction Tuningcited by 16
2019Neural Replicator Dynamicscited by 13
2024On The Global Convergence Of Online RLHF With Neural Parametrizationcited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 43; 2026: 27.

70 papers.

Sort:

By area:

2025Reinforcement Learning for Reasoning in Large Language Models with One Training Examplecited by 288
2025TTRL: Test-Time Reinforcement Learningcited by 253
2025RewardBench 2: Advancing Reward Model Evaluationcited by 136
2025Checklists Are Better Than Reward Models For Aligning Language Modelscited by 115
2025Not All Rollouts are Useful: Down-Sampling Rollouts in LLM Reinforcement Learningcited by 84
2025Retaining by Doing: The Role of On-Policy Data in Mitigating Forgettingcited by 82
2025RewardDance: Reward Scaling in Visual Generationcited by 63
2025EditScore: Unlocking Online RL for Image Editing via High-Fidelity Reward Modelingcited by 61
2025Consistent Paths Lead to Truth: Self-Rewarding Reinforcement Learning for LLM Reasoningcited by 43
2025RLVE: Scaling Up Reinforcement Learning for Language Models with Adaptive Verifiable Environmentscited by 43
2025MMR1: Enhancing Multimodal Reasoning with Variance-Aware Sampling and Open Resourcescited by 37
2025RePO: Replay-Enhanced Policy Optimizationcited by 36

38 more

2025ETTRL: Balancing Exploration and Exploitation in LLM Test-Time Reinforcement Learning Via Entropy Mechanismcited by 35
2025Cultivating Pluralism In Algorithmic Monoculture: The Community Alignment Datasetcited by 32
2025100 Days After DeepSeek-R1: A Survey on Replication Studies and More Directions for Reasoning Language Modelscited by 29
2025RewardAnything: Generalizable Principle-Following Reward Modelscited by 27
2025VL-Cogito: Progressive Curriculum Reinforcement Learning for Advanced Multimodal Reasoningcited by 26
2025The Best of N Worlds: Aligning Reinforcement Learning with Best-of-N Sampling via max@k Optimisationcited by 16
2025Discriminative Policy Optimization for Token-Level Reward Modelscited by 15
2025Sailing by the Stars: A Survey on Reward Models and Learning Strategies for Learning from Rewardscited by 14
2025Pre-Trained Policy Discriminators are General Reward Modelscited by 14
2025Improving Generalization in Intent Detection: GRPO with Reward-Based Curriculum Samplingcited by 12
2025Reinforcement Learning with Inverse Rewards for World Model Post-trainingcited by 12
2025Reshaping Reasoning in LLMs: A Theoretical Analysis of RL Training Dynamics through Pattern Selectioncited by 11
2026Small Generalizable Prompt Predictive Models Can Steer Efficient RL Post-Training of Large Reasoning Modelscited by 11
2025Understanding the Performance Gap in Preference Learning: A Dichotomy of RLHF and DPOcited by 10
2025Bradley-Terry and Multi-Objective Reward Modeling Are Complementarycited by 10
2025SuperFlow: Training Flow Matching Models with RL on the Flycited by 10
2025Behavior Injection: Preparing Language Models for Reinforcement Learningcited by 8
2025LearnAlign: Data Selection for LLM Reinforcement Learning with Improved Gradient Alignmentcited by 8
2025ReDit: Reward Dithering for Improved LLM Policy Optimizationcited by 8
2025Enhancing Large Language Model Reasoning with Reward Models: An Analytical Surveycited by 7
2026SCALER:Synthetic Scalable Adaptive Learning Environment for Reasoningcited by 7
2026Learning as Reasoning Unfolds: Progressive Rollout Allocation for Efficient Reinforcement Learningcited by 7
2025Why is Your Language Model a Poor Implicit Reward Model?cited by 6
2026On the Non-decoupling of Supervised Fine-tuning and Reinforcement Learning in Post-trainingcited by 6
2025Temporal Self-Rewarding Language Models: Decoupling Chosen-Rejected via Past-Futurecited by 5
2026Reward Modeling for Reinforcement Learning-Based LLM Reasoning: Design, Challenges, and Evaluationcited by 5
2025Collision- and Reachability-Aware Multi-Robot Control with Grounded LLM Plannerscited by 4
2026Aletheia: What Makes RLVR For Code Verifiers Tick?cited by 4
2026Alternating Reinforcement Learning with Contextual Rubric Rewards: Beyond the Scalarization Strategycited by 4
2026Localizing Credit at the Divergence: Path-Conditioned Self-Distillation for LLM Reasoningcited by 4
2025ToolSample: Dual Dynamic Sampling Methods with Curriculum Learning for RL-based Tool Learningcited by 3
2025On the Role of Preference Variance in Preference Optimizationcited by 3
2025In Good GRACEs: Principled Teacher Selection for Knowledge Distillationcited by 3
2025Beyond Correctness: Confidence-Aware Reward Modeling for Enhancing Large Language Model Reasoningcited by 3
2026Explaining and Preventing Alignment Collapse in Iterative RLHFcited by 3
2026Discretizing Reward Modelscited by 3
2025Accelerating RLHF Training with Reward Variance Increasecited by 2
2025SDA: Steering-Driven Distribution Alignment for Open LLMs without Fine-Tuningcited by 2

50 most cited of 70 All 70 (JSON)

Similar, not cited

Close in wording, no citation either way