mlsanityData to 13 Oct 2026

Ask a Strong LLM Judge when Your Reward Model is Uncertain

arXiv 2510.20369 ↗First posted 23 Oct 2025cs.LGcited by 13

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

Reward model (RM) plays a pivotal role in reinforcement learning with human feedback (RLHF) for aligning large language models (LLMs). However, classical RMs trained on human preferences are vulnerable to reward hacking and generalize poorly to out-of-distribution (OOD) inputs. By contrast, strong LLM judges equipped with reasoning capabilities demonstrate superior generalization, even without additional training, but incur significantly higher inference costs, limiting their applicability in online RLHF. In this work, we propose an uncertainty-based routing framework that efficiently complements a fast RM with a strong but costly LLM judge. Our approach formulates advantage estimation in policy gradient (PG) methods as pairwise preference classification, enabling principled uncertainty quantification to guide routing. Uncertain pairs are forwarded to the LLM judge, while confident ones are evaluated by the RM. Experiments on RM benchmarks demonstrate that our uncertainty-based routing strategy significantly outperforms random judge calling at the same cost, and downstream alignment results showcase its effectiveness in improving online RLHF.

Key facts. Cited by 13 papers in the map (data through 13 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 51 of its 65 references that are in the map. First posted on arXiv 23 Oct 2025.

Builds on

51 of 65 references.

Closest ancestors · 21

Same direction, newest first

  1. 2025A Unified Pairwise Framework for RLHF: Bridging Generative Reward Modeling and Policy Optimization7 Apr
  2. 2024Self-Generated Critiques Boost Reward Modeling for Language Models25 Nov
  3. 2024Skywork-Reward: Bag of Tricks for Reward Modeling in LLMs24 Oct
  4. 2024RM-Bench: Benchmarking Reward Models of Language Models with Subtlety and Style21 Oct
  5. 2024Beyond Bradley-Terry Models: A General Preference Model for Language Model Alignment3 Oct
  6. 2024Generative Reward Models2 Oct
  7. 2024HelpSteer2-Preference: Complementing Ratings with Preferences2 Oct
  8. 2024Uncertainty-aware Reward Model: Teaching Reward Models to Know What is Unknown1 Oct
  9. 2024Quantile Regression for Distributional Reward Models in RLHF16 Sep
  10. 2024Interpretable Preferences via Multi-Objective Reward Modeling and Mixture-of-Experts18 Jun
  11. 2024HelpSteer2: Open-source dataset for training top-performing reward models12 Jun
  12. 2024Multi-turn Reinforcement Learning from Preference Human Feedback23 May
9 more
  1. 2024Self-Play Preference Optimization for Language Model Alignment1 May
  2. 2024RewardBench: Evaluating Reward Models for Language Modeling20 Mar
  3. 2024A Minimaximalist Approach to Reinforcement Learning from Human Feedback8 Jan
  4. 2023Uncertainty-Penalized Reinforcement Learning from Human Feedback with Diverse Reward LoRA Ensembles30 Dec
  5. 2023Nash Learning from Human Feedback1 Dec
  6. 2023SteerLM: Attribute Conditioned SFT as an (User-Steerable) Alternative to RLHF9 Oct
  7. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback12 Apr
  8. 2019Fine-Tuning Language Models from Human Preferences18 Sep
  9. 2017Deep reinforcement learning from human preferences12 Jun

Other references · 30

  1. 2022Training language models to follow instructions with human feedbackcited by 15,856
  2. 2017Proximal Policy Optimization Algorithmscited by 14,592
  3. 2024The Llama 3 Herd of Modelscited by 14,486
  4. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,376
  5. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,091
  6. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,492
  7. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,247
  8. 2015Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learningcited by 4,430
  9. 2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,902
  10. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 3,011
  11. 2020Learning to summarize from human feedbackcited by 2,575
  12. 2023G-Eval: NLG Evaluation using GPT-4 with Better Human Alignmentcited by 2,228
18 more
  1. 2020Shortcut Learning in Deep Neural Networkscited by 1,944
  2. 2018Spectral Normalization for Generative Adversarial Networkscited by 1,621
  3. 2016Concrete Problems in AI Safetycited by 1,614
  4. 2024Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluatorscited by 877
  5. 2023ChatGPT Outperforms Crowd-Workers for Text-Annotation Taskscited by 857
  6. 2017Deep Bayesian Active Learning with Image Datacited by 816
  7. 2023OpenAssistant Conversations -- Democratizing Large Language Model Alignmentcited by 717
  8. 2023LLM-Blender: Ensembling Large Language Models with Pairwise Ranking and Generative Fusioncited by 553
  9. 2024From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipelinecited by 491
  10. 2023UltraFeedback: Boosting Language Models with Scaled AI Feedbackcited by 398
  11. 2020Simple and Principled Uncertainty Estimation with Deterministic Deep Learning via Distance Awarenesscited by 389
  12. 2024Aligning with Human Judgement: The Role of Pairwise Preference in Large Language Model Evaluatorscited by 147
  13. 2023HelpSteer: Multi-attribute Helpfulness Dataset for SteerLMcited by 141
  14. 2024Unpacking DPO and PPO: Disentangling Best Practices for Learning from Preference Feedbackcited by 107
  15. 2024Human Alignment of Large Language Models through Online Preference Optimisationcited by 90
  16. 2021Uncertainty Baselines: Benchmarks for Uncertainty & Robustness in Deep Learningcited by 82
  17. 2022A Simple Approach to Improve Single-Model Deep Uncertainty via Distance-Awarenesscited by 49
  18. 2025What is the Alignment Objective of GRPO?cited by 23

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2026: 13.

13 papers.

Sort:

By area:

  1. 2026JEV-as-a-Judge: Accept When Confident, Escalate When Unsurecited by 21
  2. 2026Diagnosing the Reliability of LLM-as-a-Judge via Item Response Theorycited by 15
  3. 2026When LLM Judge Scores Look Good but Best-of-N Decisions Failcited by 8
  4. 2026LLMs Exhibit Significantly Lower Uncertainty in Creative Writing Than Professional Writerscited by 4
  5. 2026WildReward: Learning Reward Models from In-the-Wild Human Interactionscited by 3
  6. 2026Rubrics as an Attack Surface: Stealthy Preference Drift in LLM Judgescited by 3
  7. 2026AdaJudge: Adaptive Multi-Perspective Judging for Reward Modelingcited by 1
  8. 2026RankJudge: A Multi-Turn LLM-as-a-Judge Synthetic Benchmark Generatorcited by 1
  9. 2026A Unifying Lens on Reward Uncertainty in RLHFcited by 1
  10. 2026C2T: Captioning-Structure and LLM-Aligned Common-Sense Reward Learning for Traffic--Vehicle Coordinationcited by 0
  11. 2026Share the Judge, Learn the Deferral: Where Specialization Helps LLM Evaluationcited by 0
  12. 2026DCRL: Decoupling and Coupling Reinforcement Learning via Policy-Reward Manifold Alignmentcited by 0
1 more
  1. 2026Auditing and Repairing LLM-as-Judge Failures in a Production Text-to-SQL Pipelinecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Uncertainty Quantification for Large Language Model Reward Learning under Heterogeneous Human Feedback
  2. 2024Reward-Robust RLHF in LLMs
  3. 2024Towards Reliable Alignment: Uncertainty-aware RLHF
  4. 2026Uncertainty-Aware Reward Modeling for Stable RLHF
  5. 2026RewardUQ: A Unified Framework for Uncertainty-Aware Reward Models
  6. 2024Bayesian Reward Models for LLM Alignment
  7. 2025Beyond Correctness: Confidence-Aware Reward Modeling for Enhancing Large Language Model Reasoning
  8. 2025Reward Model Routing in Alignment