mlsanityData to 14 Oct 2026

Reproducing, Analyzing, and Detecting Reward Hacking in Rubric-Based Reinforcement Learning

arXiv 2606.04923 ↗First posted 3 Jun 2026cs.LG cs.AI cs.CLcited by 16

On the map: LLM alignment / RLHF / preference optimisation → Reward hacking and misspecification

also in 1 more
Abstract (arXiv)

Rubric-based reinforcement learning (RL) uses an LLM-as-a-Judge (LaaJ) to score model outputs according to rubrics as rewards. However, policy models may exploit latent biases in the judge, leading to reward hacking and ineffective or unsafe training outcomes. In real-world rubric-based RL, such hacking behaviors are often subtle and entangled with multiple judge biases, making them difficult to analyze, detect, and mitigate. In this paper, we introduce CHERRL, a Controllable Hacking Environment for Rubric-based RL. By injecting known biases into LaaJ, CHERRL enables stable reproduction of reward hacking, explicit observation of reward divergence, and identification of hacking onset. This provides a clean experimental testbed for studying the mechanisms and mitigations of reward hacking in rubric-based RL. To demonstrate its utility, we analyze different judge biases from the perspectives of discoverability and exploitability, and explore an agent for automatically detecting reward hacking onset from training logs. The code and environment are publicly available at https://github.com/THUAIS-Lab/CHERRL.

Key facts. Cited by 16 papers in the map (data through 14 Oct 2026). Placed in Reward hacking and misspecification, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 48 of its 57 references that are in the map. First posted on arXiv 3 Jun 2026.

Builds on

48 of 57 references.

Closest ancestors · 6

Same direction, newest first

  1. 2026Reward Hacking in Rubric-Based Reinforcement Learning12 May
  2. 2026Reward Hacking in the Era of Large Models: Mechanisms, Emergent Misalignment, Challenges15 Apr
  3. 2026Rubrics to Tokens: Bridging Response-level Rubrics and Token-level Rewards in Instruction Following Tasks3 Apr
  4. 2026Countdown-Code: A Testbed for Studying The Emergence and Generalization of Reward Hacking in RLVR7 Mar
  5. 2026Learning Query-Specific Rubrics from Human Preferences for DeepResearch Report Generation3 Feb
  6. 2025Detecting Proxy Gaming in RL and LLM Alignment via Evaluator Stress Tests8 Jul

Other references · 42

  1. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  2. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  3. 2023Large Language Models are not Fair Evaluatorscited by 1,085
  4. 2023Towards Understanding Sycophancy in Language Modelscited by 1,040
  5. 2024LLM Evaluators Recognize and Favor Their Own Generationscited by 686
  6. 2025Process Reinforcement through Implicit Rewardscited by 504
  7. 2024From Crowdsourced Data to High-Quality Benchmarks: Arena-Hard and BenchBuilder Pipelinecited by 500
  8. 2024From Generation to Judgment: Opportunities and Challenges of LLM-as-a-judgecited by 494
  9. 2025HealthBench: Evaluating Large Language Models Towards Improved Human Healthcited by 429
  10. 2024Justice or Prejudice? Quantifying Biases in LLM-as-a-Judgecited by 368
  11. 2025Rubrics as Rewards: Reinforcement Learning Beyond Verifiable Domainscited by 366
  12. 2024Humans or LLMs as the Judge? A Study on Judgement Biasescited by 323
30 more
  1. 2023Helping or Herding? Reward Model Ensembles Mitigate but do not Eliminate Reward Hackingcited by 200
  2. 2025Generalizing Verifiable Instruction Followingcited by 194
  3. 2025DR Tulu: Reinforcement Learning with Evolving Rubrics for Deep Researchcited by 114
  4. 2025Reinforcement Learning with Rubric Anchorscited by 113
  5. 2025WritingBench: A Comprehensive Benchmark for Generative Writingcited by 106
  6. 2025Reward Shaping to Mitigate Reward Hacking in RLHFcited by 88
  7. 2023Human Feedback is not Gold Standardcited by 83
  8. 2025Breaking the Exploration Bottleneck: Rubric-Scaffolded Reinforcement Learning for General LLM Reasoningcited by 63
  9. 2025One Token to Fool LLM-as-a-Judgecited by 60
  10. 2025AdvancedIF: Rubric-Based Benchmarking and Reinforcement Learning for Advancing LLM Instruction Followingcited by 51
  11. 2025Writing-Zero: Bridge the Gap Between Non-verifiable Tasks and Verifiable Rewardscited by 50
  12. 2025Online Rubrics Elicitation from Pairwise Comparisonscited by 46
  13. 2025VerIF: Verification Engineering for Reinforcement Learning in Instruction Followingcited by 44
  14. 2025RL Tango: Reinforcing Generator and Verifier Together for Language Reasoningcited by 40
  15. 2025Training AI Co-Scientists Using Rubric Rewardscited by 29
  16. 2025Is It Thinking or Cheating? Detecting Implicit Reward Hacking by Measuring Reasoning Effortcited by 27
  17. 2025Sparks of Science: Hypothesis Generation Using Structured Paper Datacited by 25
  18. 2026Chaining the Evidence: Robust Reinforcement Learning for Deep Search Agents with Citation-Aware Rubric Rewardscited by 14
  19. 2025Generative RLHF-V: Learning Principles from Multi-modal Human Preferencecited by 13
  20. 2025InfiMed-ORBIT: Aligning LLMs on Open-Ended Complex Tasks via Rubric-Based Incremental Trainingcited by 12
  21. 2026HeurekaBench: A Benchmarking Framework for AI Co-scientistcited by 12
  22. 2025RLMR: Reinforcement Learning with Mixed Rewards for Creative Writingcited by 11
  23. 2025Self-Rewarding Rubric-Based Reinforcement Learning for Open-Ended Reasoningcited by 11
  24. 2025AutoRubric: Rubric-Based Generative Rewards for Faithful Multimodal Reasoningcited by 11
  25. 2026Toward Robust LLM-Based Judges: Taxonomic Bias Evaluation and Debiasing Optimizationcited by 9
  26. 2026Health-SCORE: Towards Scalable Rubrics for Improving Health-LLMscited by 8
  27. 2025IFDECORATOR: Wrapping Instruction Following Reinforcement Learning with Verifiable Rewardscited by 7
  28. 2025An Efficient Rubric-based Generative Verifier for Search-Augmented LLMscited by 4
  29. 2026R2-Write: Reflection and Revision for Open-Ended Writing with Deep Reasoningcited by 3
  30. 2026HealthCraft: A Reinforcement Learning Safety Environment for Emergency Medicinecited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 16.

16 papers.

Sort:

By area:

  1. 2026Rubric Dropout: A Simple Way to Mitigate Reward Hacking in Rubric-as-Reward RLcited by 9
  2. 2026Scaling Large Reasoning Models beyond Human Supervision: A Path toward Superintelligencecited by 5
  3. 2026ImpossibleRubrics: Stress-Testing Generated Rubrics as Reward Signalscited by 1
  4. 2026RubricArmor: Adversarial Evolution Improves LLM-Based Rubric Generationcited by 1
  5. 2026Distill Where the Student Goes: Teacher-Regularized RL for English-Evidence Cross-Lingual RAGcited by 0
  6. 2026AgentOmnia: Scaling Agentic Models for Full-Scenario Applicationscited by 0
  7. 2026From Migration to Calibration: Aligning Agents with Deployment Requirementscited by 0
  8. 2026Verifier Errors in RLVR: Reward Hacking, Limits of Feedback, and Selective Controlcited by 0
  9. 2026Binarization Flattens the Score Spacecited by 0
  10. 2026Scoring Higher, Answering Worse: Mitigating Reward Hacking in Rubric-Based RL via Protocol-Level Rubricscited by 0
  11. 2026IdeaScientist: Orchestrating Agents for Grounded Scientific Ideationcited by 0
  12. 2026Self-Evaluating Recursive Agentscited by 0
4 more
  1. 2026How to post-train on a surrogate: Envelope sampling mitigates reward hackingcited by 0
  2. 2026Reflective Prompt Optimization of a Deep-Research Agent: A Topic-Disjoint, Multi-Seed Protocol for Measuring Transferable Gains and Prompt-Level Overfittingcited by 0
  3. 2026SFT Leaves a Reproducible Signature in Activation Geometry: Calibrating Post-Training Interpretability Against Run-to-Run Noisecited by 0
  4. 2026Checking the Checkers: A Synthesis of Verifier Noise and Robustness in RLVRcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026CATCH: A Controllable Analysis Testbed for Reward Hacking in Coding RL
  2. 2026Detecting and Suppressing Reward Hacking with Gradient Fingerprints
  3. 2026Adversarial Reward Auditing for Active Detection and Mitigation of Reward Hacking
  4. 2026Reward Hacking Benchmark: Measuring Exploits in LLM Agents with Tool Use
  5. 2026Gradient Routing Localizes and Removes Unintended Behaviors in RL
  6. 2026Not Every Rubric Teaches Equally: Policy-Aware Rubric Rewards for RLVR
  7. 2026Open Rubric System: Scaling Reinforcement Learning with Pairwise Adaptive Rubric
  8. 2026Alternating Reinforcement Learning for Rubric-Based Reward Modeling in Non-Verifiable LLM Post-Training