mlsanityData to 7 Oct 2026

Expected Reward Prediction, with Applications to Model Routing

arXiv 2603.20217 ↗First posted 3 Mar 2026cs.CL cs.LGcited by 0

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

Reward models are a standard tool to score responses from LLMs. Reward models are built to rank responses to a fixed prompt sampled from a single model, for example to choose the best of n sampled responses. In this paper, we study whether scores from response-level reward models lifted to score a model's suitability for a prompt, prior to seeing responses from that model. Specifically, we show that it is straightforward to predict the expected reward that an LLM would earn from the reward model under repeated sampling. Further, we show that these expected reward predictions are precise and discriminative enough to support an application to a model routing protocol that routes prompts to models at inference time to maximize reward while controlling computational cost. We demonstrate the performance of this routing procedure on the open-perfectblend dataset, using a model pool composed of Llama3.1-Instruct 8B/70B, Gemma2-IT 9B/27B, and Gemma1-IT 7B models. Our simple expected reward prediction--based routing (ERP) outperforms baselines that route prompts to models with the best average performance within each prompt's category, and explains the success of more complex routing protocols that implicitly estimate an expected reward. Our approach has the added advantage of being trivially extensible as new models are added to the pool.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 21 of its 23 references that are in the map. First posted on arXiv 3 Mar 2026.

Builds on

21 of 23 references.

Closest ancestors · 8

Same direction, newest first

  1. 2024The Perfect Blend: Redefining RLHF with Mixture of Judges30 Sep
  2. 2024Regularizing Hidden States Enables Learning Generalizable Reward Model for LLMs14 Jun
  3. 2024RewardBench: Evaluating Reward Models for Language Modeling20 Mar
  4. 2024Transforming and Combining Rewards for Aligning Large Language Models1 Feb
  5. 2024Secrets of RLHF in Large Language Models Part II: Reward Modeling11 Jan
  6. 2022Scaling Laws for Reward Model Overoptimization19 Oct
  7. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback12 Apr
  8. 2017Deep reinforcement learning from human preferences12 Jun

Other references · 13

  1. 2024The Llama 3 Herd of Modelscited by 14,192
  2. 2020Learning to summarize from human feedbackcited by 2,562
  3. 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,928
  4. 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,758
  5. 2024Gemma: Open Models Based on Gemini Research and Technologycited by 889
  6. 2023FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performancecited by 791
  7. 2024RouteLLM: Learning to Route LLMs with Preference Datacited by 633
  8. 2022MTEB: Massive Text Embedding Benchmarkcited by 623
  9. 2024Semantic Entropy Probes: Robust and Cheap Hallucination Detection in LLMscited by 246
  10. 2023Routing to the Expert: Efficient Reward-guided Ensemble of Large Language Modelscited by 141
  11. 2023AutoMix: Automatically Mixing Language Modelscited by 116
  12. 2024MetaLLM: A High-performant and Cost-efficient Dynamic Framework for Wrapping LLMscited by 39
1 more
  1. 2024OptLLM: Optimal Assignment of Queries to Large Language Modelscited by 14

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2024Exploring Domain Robust Lightweight Reward Models based on Router Mechanism
  2. 2025Sailing by the Stars: A Survey on Reward Models and Learning Strategies for Learning from Rewards
  3. 2026ReCal: Reward Calibration for RL-based LLM Routing
  4. 2025RewardBench 2: Advancing Reward Model Evaluation
  5. 2025Learning to Route LLMs from Bandit Feedback: One Policy, Many Trade-offs
  6. 2026Reward Models Are Secretly Value Functions: Temporally Coherent Reward Modeling
  7. 2024Skywork-Reward: Bag of Tricks for Reward Modeling in LLMs
  8. 2025Probing Preference Representations: A Multi-Dimensional Evaluation and Analysis Method for Reward Models