mlsanityData to 14 Oct 2026

EvalStop: Using World Feedback to Detect and Correct Reward Overoptimization in Multi-Tenant RLHF Platforms

arXiv 2606.04145 ↗First posted 2 Jun 2026cs.LG cs.AI cs.DCcited by 1

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

Cloud LLM fine-tuning platforms increasingly serve RLHF workloads, where a learned reward model is optimized as a proxy for human quality. As Gao et al. (2023) showed, this proxy diverges from world feedback (downstream eval metrics) under sustained optimization pressure, a phenomenon known as reward overoptimization. Existing platform schedulers ignore this divergence: non-clairvoyant schedulers optimize JCT without any quality signal, SLAQ-style quality-aware schedulers use training loss (a weaker proxy that drops monotonically through hacking), and classical per-job early stopping requires human monitoring and does not free shared GPUs. We propose EvalStop, a composable scheduling primitive that terminates jobs on k consecutive eval-score declines, releases GPUs, preserves the best checkpoint, and delegates to any base scheduler. We frame scheduler-level early stopping as a detection problem and evaluate it in a discrete-event simulator whose RLHF workload mixes reward-hacking and structurally healthy runs, with ground-truth labels hidden from schedulers. On RLHF-heavy workloads (80% RLHF, 64 GPUs), EvalStop achieves precision 98% / recall 99% / FPR 1.5% while improving JCT by 9% and cutting wasted compute by 22% over SRTF-Est (p<0.05). Trivial fixed-progress and loss-plateau competitors either incur 65% FPR on healthy RLHF or miss over half of true hacking cases. Gains compose across every base scheduler tested (9-25% JCT) and detection quality stays stable under eval noise (precision at least 91% at noise std <= 0.05) and hacking base rate (precision at least 89% across 20-80% hacking fractions).

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 18 of its 27 references that are in the map. First posted on arXiv 2 Jun 2026.

Builds on

18 of 27 references.

Closest ancestors · 4

Same direction, newest first

  1. 2024Scaling Laws for Reward Model Overoptimization in Direct Alignment Algorithms5 Jun
  2. 2023Confronting Reward Model Overoptimization with Constrained RLHF6 Oct
  3. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May
  4. 2022Scaling Laws for Reward Model Overoptimization19 Oct

Other references · 14

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2020Learning to summarize from human feedbackcited by 2,588
  3. 2022The Effects of Reward Misspecification: Mapping and Mitigating Misaligned Modelscited by 328
  4. 2022Defining and Characterizing Reward Hackingcited by 203
  5. 2023S-LoRA: Serving Thousands of Concurrent LoRA Adapterscited by 152
  6. 2020Pollux: Co-adaptive Cluster Scheduling for Goodput-Optimized Deep Learningcited by 72
  7. 2020Heterogeneity-Aware Cluster Scheduling Policies for Deep Learning Workloadscited by 56
  8. 2024Parcae: Proactive, Liveput-Optimized DNN Training on Preemptible Instancescited by 17
  9. 2022Shockwave: Fair and Efficient Cluster Scheduling for Dynamic Adaptation in Machine Learningcited by 12
  10. 2025KIS-S: A GPU-Aware Kubernetes Inference Simulator with RL-Based Auto-Scalingcited by 9
  11. 2025AAPA: An Archetype-Aware Predictive Autoscaler with Uncertainty Quantification for Serverless Workloads on Kubernetescited by 3
  12. 2026When Does Deep RL Beat Calibrated Baselines? A Benchmark Study on Adaptive Resource Controlcited by 3
2 more
  1. 2025Deadline-Aware Online Scheduling for LLM Fine-Tuning with Spot Market Predictionscited by 1
  2. 2026MuxTune: Efficient Multi-Task LLM Fine-Tuning in Multi-Tenant Datacenters via Spatial-Temporal Backbone Multiplexingcited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Winning by Peeking: Unenforced Budgets and Test-Set Selection Inflate Short-Budget AutoML Comparisonscited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2026When RLHF Fails: A Mechanistic Taxonomy of Reward Hacking, Collapse, and Evaluator Gaming
  2. 2026Retroactive Advantage Correction: Closed-Form V-Trace Bias Correction for Delay-Aware RLHF
  3. 2026Reward Models Are Secretly Value Functions: Temporally Coherent Reward Modeling
  4. 2025Reward Model Overoptimisation in Iterated RLHF
  5. 2026Predicting Closed-Loop Performance of Latent World Models: Offline Checkpoint Selection for MPC and Model-Based RL Under Non-Markovian Rewards in LunarLander
  6. 2026RHyVE: Competence-Aware Verification and Phase-Aware Deployment for LLM-Generated Reward Hypotheses
  7. 2026Countdown-Code: A Testbed for Studying The Emergence and Generalization of Reward Hacking in RLVR
  8. 2025Scaling Test-Time Compute Without Verification or RL is Suboptimal