The Flip Side of RLHF: On-Policy Feedback for Reward Model Self-Supervised Improvement
On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF
Abstract (arXiv)
Building strong reward models (RMs) for language model alignment is bottlenecked by the cost and difficulty of acquiring diverse and reliable preference data from human annotation or judge models. It is dramatically worse as the policy evolves beyond the static RM training. Therefore, we propose SAVE (Self-supervised reward model improvement via Value-Anchored On-policy feedback), a framework that grades on-policy responses as feedback by using the value function for on-policy RM training. SAVE naturally converts the reward-graded on-policy responses into supervision with a prompt-specific value head as an adaptive anchor. It computes RM advantages and filters ambiguous samples to update the RM via a contrastive objective. The effectiveness of SAVE for enhancing RM training is strongly validated through rigorous empirical evaluation across six diverse benchmarks. It achieves outperforming results across all datasets while maintaining consistent improvements across three RL algorithms (GRPO, RLOO, GSPO) and different policy backbones.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 24 of its 38 references that are in the map. First posted on arXiv 29 May 2026.
Builds on
24 of 38 references.
Closest ancestors · 11
Same direction, newest first
- 2026Reward Models Are Secretly Value Functions: Temporally Coherent Reward Modeling24 Apr
- 2026Real-Time Aligned Reward Model beyond Semantics30 Jan
- 2025Skywork-Reward-V2: Scaling Preference Data Curation via Human-AI Synergy2 Jul
- 2025RewardBench 2: Advancing Reward Model Evaluation2 Jun
- 2024RM-Bench: Benchmarking Reward Models of Language Models with Subtlety and Style21 Oct
- 2024How to Evaluate Reward Models for RLHF18 Oct
- 2024Interpretable Preferences via Multi-Objective Reward Modeling and Mixture-of-Experts18 Jun
- 2024Regularizing Hidden States Enables Learning Generalizable Reward Model for LLMs14 Jun
- 2023Reward Model Ensembles Help Mitigate Overoptimization4 Oct
- 2023RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedback1 Sep
- 2022Scaling Laws for Reward Model Overoptimization19 Oct
Other references · 13
- 2017Proximal Policy Optimization Algorithmscited by 15,215
- 2024The Llama 3 Herd of Modelscited by 14,715
- 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
- 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
- 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scalecited by 2,810
- 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 2,026
- 2024Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluatorscited by 897
- 2024From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipelinecited by 500
- 2023UltraFeedback: Boosting Language Models with Scaled AI Feedbackcited by 406
- 2024JudgeBench: A Benchmark for Evaluating LLM-based Judgescited by 318
- 2015Coordinate Descent Algorithmscited by 219
- 2025Agentic Reinforcement Learning with Implicit Step Rewardscited by 47
1 more
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2024West-of-N: Synthetic Preferences for Self-Improving Reward Models
- 2024Aligning Large Language Models by On-Policy Self-Judgment
- 2024Fine-Tuning Language Models with Reward Learning on Policy
- 2024Sequence to Sequence Reward Modeling: Improving RLHF by Language Feedback
- 2024Solving the Inverse Alignment Problem for Efficient RLHF
- 2025Self-Rewarding PPO: Aligning Large Language Models with Demonstrations Only
- 2026Reinforcement Learning from Human Feedback: A Statistical Perspective
- 2025A First-Order Logic-Based Alternative to Reward Models in RLHF