mlsanityData to 14 Oct 2026

Who Sets the Pace of Judge Hacking? Judge Capability versus the Policy-Judge Gap

arXiv 2610.15778 ↗First posted 11 Oct 2026cs.LG cs.CLcited by 0

New in LLM reasoning; no direction yet. LLM reasoning

Abstract (arXiv)

Under GRPO against a frozen, reference-free LLM judge, RL exploits the judge's false positives. Does the policy-judge capability gap or the judge itself set how fast and how far? We train 0.5B and 1.5B Qwen2.5 policies on GSM8K against four judges at least as large, and the 1.5B policy also against one smaller judge. In a post-hoc analysis, judge acceptance rose, but no cell of 12 gained more than 0.04 in seed-mean held-out greedy accuracy. A gold-reward control with the same recipe raised the 1.5B policy's accuracy by +0.054 (for the 0.5B policy, gold reward gave no detectable gain either). The depth of judge hacking tracks the judge: the plateau correlates with judge AUC (Spearman -0.69 over 12 cells), and within a family the larger judge is fooled less deeply. For the pace, two pre-specified pairs contradict the gap account: the cell with the larger gap was fooled 3.5x sooner at the pre-specified threshold. With 12 cells, however, we cannot formally separate judge AUC from the gap for either endpoint. A blind test of step-0 predictors on three held-out judges was inconclusive. At 7-8B, judges from three families plateaued between 0.37 and 0.85.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 33 of its 34 references that are in the map. First posted on arXiv 11 Oct 2026.

Builds on

33 of 34 references.

References · 33

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  3. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  4. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  5. 2024Qwen2.5 Technical Reportcited by 4,942
  6. 2023Let's Verify Step by Stepcited by 3,789
  7. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scalecited by 2,810
  8. 2022Scaling Laws for Reward Model Overoptimizationcited by 1,095
  9. 2024RewardBench: Evaluating Reward Models for Language Modelingcited by 499
  10. 2024Generative Verifiers: Reward Modeling as Next-Token Predictioncited by 480
  11. 2023Weak-to-Strong Generalization: Eliciting Strong Capabilities With Weak Supervisioncited by 467
  12. 20242 OLMo 2 Furiouscited by 340
21 more
  1. 2022The Effects of Reward Misspecification: Mapping and Mitigating Misaligned Modelscited by 328
  2. 2024Debating with More Persuasive LLMs Leads to More Truthful Answerscited by 266
  3. 2023Reward Model Ensembles Help Mitigate Overoptimizationcited by 231
  4. 2022Measuring Progress on Scalable Oversight for Large Language Modelscited by 225
  5. 2024Scaling Laws for Reward Model Overoptimization in Direct Alignment Algorithmscited by 143
  6. 2025General-Reasoner: Advancing LLM Reasoning Across All Domainscited by 135
  7. 2024Language Models Learn to Mislead Humans via RLHFcited by 124
  8. 2024On scalable oversight with weak LLMs judging strong LLMscited by 99
  9. 2024How to Evaluate Reward Models for RLHFcited by 88
  10. 2025What Makes a Reward Model a Good Teacher? An Optimization Perspectivecited by 73
  11. 2025One Token to Fool LLM-as-a-Judgecited by 60
  12. 2024Rethinking Reward Model Evaluation: Are We Barking up the Wrong Tree?cited by 31
  13. 2025Scaling Laws For Scalable Oversightcited by 23
  14. 2024Evaluating Robustness of Reward Models for Mathematical Reasoningcited by 22
  15. 2025From Accuracy to Robustness: A Study of Rule- and Model-based Verifiers in Mathematical Reasoningcited by 22
  16. 2026Examining Reasoning LLMs-as-Judges in Non-Verifiable LLM Post-Trainingcited by 13
  17. 2025Rethinking Reward Model Evaluation Through the Lens of Reward Overoptimizationcited by 11
  18. 2026More Convincing, Not More Correct: Self-Play Reward Hacking of Reference-Free LLM Judgescited by 10
  19. 2025AdvJudge-Zero: Binary Decision Flips in LLM-as-a-Judge via Adversarial Control Tokenscited by 4
  20. 2026When the Reward Suite Is Leaky: A Preregistered Causal Contrast of Natural Verifier False Positives in RLVRcited by 4
  21. 2026Debate Training Reduces Reward Hacking in RLAIFcited by 4

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026The Coin Flip Judge? Reliability and Bias in LLM-as-a-Judge Evaluation
  2. 2026Reproducing, Analyzing, and Detecting Reward Hacking in Rubric-Based Reinforcement Learning
  3. 2026Rubric Dropout: A Simple Way to Mitigate Reward Hacking in Rubric-as-Reward RL
  4. 2025Know Thy Judge: On the Robustness Meta-Evaluation of LLM Safety Judges
  5. 2026Calibrate, Don't Curate: Label-Efficient Estimation from Noisy LLM Judges
  6. 2026Jagged Judges: Epistemic Stability Under Perturbation, Pressure, and Persistence
  7. 2026Will the Judge Flip? Predicting Position-Sensitive LLM Judgments from Residual Stream Activations
  8. 2026A Near-Zero Monitor Readout Is Not Evidence of Behavioral Control