mlsanityData to 7 Oct 2026

Evaluator Ensembles Under Reward Hacking: Covariance Geometry and Finite-Search Guarantees

arXiv 2608.08002 ↗First posted 8 Aug 2026cs.LGcited by 0

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

Language-model judges and reward models enable scalable supervision, but finite optimization can exploit evaluator errors rather than improve response quality. We characterize this failure through the covariance geometry of evaluator ensembles. For calibrated judges, the ensemble mean retains common-mode error along the all-ones direction, whereas cross-judge disagreement captures only orthogonal error. Consequently, disagreement can be high despite robust aggregation, or low while shared response-dependent errors persist. We prove that common-mode error is not identifiable from internal judge scores alone. Under a joint sub-Gaussian model, we bound best-of-K selection overstatement and target-quality regret, extending the guarantees to predictably adaptive search under conditional calibration. The resulting search terms scale as the square root of log K and are asymptotically tight for Gaussian projected errors. We further show that noisy quality proxies introduce artificial rank-one covariance without changing disagreement, and propose a bounded two-anchor Bernstein certificate for finite-search error and regret. Fixed-seed Gaussian stress tests over 120 (J, rho, K) configurations and real-model audits validate the theory while revealing the limits of disagreement-based diagnostics under increasing search pressure.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 23 of its 33 references that are in the map. First posted on arXiv 8 Aug 2026.

Builds on

23 of 33 references.

Closest ancestors · 8

Same direction, newest first

  1. 2024Scaling Laws for Reward Model Overoptimization in Direct Alignment Algorithms5 Jun
  2. 2024WARM: On the Benefits of Weight Averaged Reward Models22 Jan
  3. 2023Helping or Herding? Reward Model Ensembles Mitigate but do not Eliminate Reward Hacking14 Dec
  4. 2023Reward Model Ensembles Help Mitigate Overoptimization4 Oct
  5. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May
  6. 2022Scaling Laws for Reward Model Overoptimization19 Oct
  7. 2022Uncertainty Estimation for Language Reward Models14 Mar
  8. 2017Deep reinforcement learning from human preferences12 Jun

Other references · 15

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2020Learning to summarize from human feedbackcited by 2,562
  3. 2016Concrete Problems in AI Safetycited by 1,582
  4. 2017Deep Bayesian Active Learning with Image Datacited by 812
  5. 2017Selective Classification for Deep Neural Networkscited by 799
  6. 2024LLM Evaluators Recognize and Favor Their Own Generationscited by 651
  7. 2024Replacing Judges with Juries: Evaluating LLM Generations with a Panel of Diverse Modelscited by 301
  8. 2024JudgeBench: A Benchmark for Evaluating LLM-based Judgescited by 294
  9. 2025Is Best-of-N the Best of Them? Coverage, Scaling, and Optimality in Inference-Time Alignmentcited by 91
  10. 2025Great Models Think Alike and this Undermines AI Oversightcited by 53
  11. 2024Limits to scalable evaluation at the frontier: LLM as Judge won't beat twice the datacited by 46
  12. 2026Nine Judges, Two Effective Votes: Correlated Errors Undermine LLM Evaluation Panelscited by 21
3 more
  1. 2025Detecting Proxy Gaming in RL and LLM Alignment via Evaluator Stress Testscited by 19
  2. 2026When Does Combining Language Models Help? A Co-Failure Ceiling on Routing, Voting, and Mixture-of-Agents Across 67 Frontier Modelscited by 10
  3. 2026When LLM Judge Scores Look Good but Best-of-N Decisions Failcited by 8

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Calibrate, Don't Curate: Label-Efficient Estimation from Noisy LLM Judges
  2. 2026From Curiosity to Caution: Mitigating Reward Hacking for Best-of-N with Pessimism
  3. 2026Co-Evolving LLM Evaluators and Policies via DynamicRubric
  4. 2026Consolidating Rewarded Perturbations for LLM Post-Training
  5. 2026On Cost-Effective LLM-as-a-Judge Improvement Techniques
  6. 2026Optimizing the Score, Losing Sight of the Task: Reward Hacking Across Weights, Selection, and Prompts
  7. 2025Evaluating Judges as Evaluators: The JETTS Benchmark of LLM-as-Judges as Test-Time Scaling Evaluators
  8. 2026Automatically Finding Reward Model Biases