mlsanityData to 7 Oct 2026

Reward hacking and misspecification

94 papers2 this weekMedium confidence in this grouping: mostly one topic, with blurry edgesMedium confidence in this grouping: mostly one topic, with blurry edges · About

Keywords: hacking, reward, reward hacking, rubric, rubrics

Key facts. Reward hacking and misspecification: 94 papers on arXiv since 2018 (data through 7 Oct 2026). 21 of its 94 mapped papers (22%) cite “Monitoring Reasoning Models for Misbehavior and the Risks of Promoting Obfuscation” (2025, arXiv 2503.11926). 20 of its 94 mapped papers (21%) cite “The Effects of Reward Misspecification: Mapping and Mitigating Misaligned Models” (2022, arXiv 2201.03544). 20 new papers 17 Sep–7 Oct.

What’s new

17 Sep – 7 Oct 2026

20 new papers, 17 Sep – 7 Oct; 2 this week.

  1. This week · 2
  2. RH-Detect: A Unified Benchmark for Reward Hacking Detection
  3. Better Call Reward: Reward Hacking as Strategic Abstention in Legal Reasoning Models
  4. Earlier · 18
  5. RubricArmor: Adversarial Evolution Improves LLM-Based Rubric Generation
  6. Pressure, Context, and Machine Self-Control: A Criminological Test of Reward Hacking in Generative AI Models
  7. Reward-DAgger: Robot-Gated Interactive Imitation Learning with General-Purpose Progress-Based Reward Models also fits: Reward modeling and RLHF
  8. A Near-Zero Monitor Readout Is Not Evidence of Behavioral Control
  9. Gains and Collapse in On-Policy Distillation:A Reinforcement Learning Perspective also fits: Reward modeling and RLHF
  10. hacktrace: behavior-supervised detection of reward hacking during code generation
  11. CriticHack: Evaluating Visual Rewards Under Robot Policy Optimization also fits: Reward modeling and RLHF
  12. CATCH: A Controllable Analysis Testbed for Reward Hacking in Coding RL
All 20
  1. Alignment via Training Against Probes Without Losing Monitorability also fits: Safety alignment and jailbreak attacks
  2. Scoring Higher, Answering Worse: Mitigating Reward Hacking in Rubric-Based RL via Protocol-Level Rubrics
  3. Revisiting scaling laws for reward optimization also fits: Reward modeling and RLHF
  4. STAR-GRPO: Canonical Anchoring and Reliability-First Advantages against Representation-Dependent Reward Hacking
  5. Counterfactual Rollout Replay: Forkable Environments as Free Process Rewards for Software Engineering Agents also fits: Process reward models for reasoning
  6. Verifier Errors in RLVR: Reward Hacking, Limits of Feedback, and Selective Control
  7. Reward Hacking Challenges Oversight of Autonomous Research Agents
  8. ProCredit: From Outcome Rewards to Progress Credit in Agentic Reinforcement Learning
  9. Optimizing the Score, Losing Sight of the Task: Reward Hacking Across Weights, Selection, and Prompts
  10. RewardVerse: Rubric-Guided Policy Optimization for Video Reward Modeling also fits: Reward modeling and RLHF

Foundations

Most cited by papers here

  1. Monitoring Reasoning Models for Misbehavior and the Risks of Promoting Obfuscation2025cited by 21 here
  2. The Effects of Reward Misspecification: Mapping and Mitigating Misaligned Models2022cited by 20 here
  3. School of Reward Hacks: Hacking harmless tasks generalizes to misaligned behavior in LLMs2025cited by 14 here
  4. Natural Emergent Misalignment from Reward Hacking in Production RL2025cited by 12 here
  5. OpenRubrics: Towards Scalable Synthetic Rubric Generation for Reward Modeling and LLM Alignment2025cited by 11 here

Lineage

The papers this direction builds on, oldest first.

Each paper on the line cites the one before.

  1. 2017Deep reinforcement learning from human preferencescited by 20 here
  2. 2018Scalable agent alignment via reward modeling: a research directioncited by 10 here
  3. 2020Learning to summarize from human feedbackcited by 18 here
  4. 2022Training language models to follow instructions with human feedbackcited by 32 here
  5. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 10 here
  6. 2022Scaling Laws for Reward Model Overoptimizationcited by 27 here
  7. 2023Reward Model Ensembles Help Mitigate Overoptimizationcited by 10 here
  8. 2023Helping or Herding? Reward Model Ensembles Mitigate but do not Eliminate Reward Hackingcited by 9 here
  9. 2024ODIN: Disentangled Reward Mitigates Hacking in RLHFcited by 9 here
  10. 2024Interpretable Preferences via Multi-Objective Reward Modeling and Mixture-of-Expertscited by 9 here
  11. 2025RewardBench 2: Advancing Reward Model Evaluationcited by 9 here
  12. 2025Checklists Are Better Than Reward Models For Aligning Language Modelscited by 13 here
  13. 2025Chasing the Tail: Effective Rubric-based Reward Modeling for Large Language Model Post-Trainingcited by 10 here
  14. 2025OpenRubrics: Towards Scalable Synthetic Rubric Generation for Reward Modeling and LLM Alignmentcited by 11 here
10 more it builds on, not in this chain
  1. 2022The Effects of Reward Misspecification: Mapping and Mitigating Misaligned Modelscited by 20 here
  2. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 16 here
  3. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 18 here
  4. 2023A Long Way to Go: Investigating Length Correlations in RLHFcited by 9 here
  5. 2024Correlated Proxies: A New Definition and Improved Mitigation for Reward Hackingcited by 10 here
  6. 2024Language Models Learn to Mislead Humans via RLHFcited by 10 here
  7. 2025Monitoring Reasoning Models for Misbehavior and the Risks of Promoting Obfuscationcited by 21 here
  8. 2025RM-R1: Reward Modeling as Reasoningcited by 10 here
  9. 2025School of Reward Hacks: Hacking harmless tasks generalizes to misaligned behavior in LLMscited by 14 here
  10. 2025Natural Emergent Misalignment from Reward Hacking in Production RLcited by 12 here

The 24 most cited of 304.

Browse

30 of 94, ranked 6–35

  1. 6Chasing the Tail: Effective Rubric-based Reward Modeling for Large Language Model Post-Training202510
  2. 7Correlated Proxies: A New Definition and Improved Mitigation for Reward Hacking202410
  3. 8Alternating Reinforcement Learning for Rubric-Based Reward Modeling in Non-Verifiable LLM Post-Training20267
  4. 9EvilGenie: A Reward Hacking Benchmark20256
  5. 10Defining and Characterizing Reward Hacking20226
  6. 11Rethinking Rubric Generation for Improving LLM Judge and Reward Modeling for Open-ended Tasks20265
  7. 12Reward Hacking in the Era of Large Models: Mechanisms, Emergent Misalignment, Challenges20265
  8. 13Teaching Models to Verbalize Reward Hacking in Chain-of-Thought Reasoning20255
  9. 14Benchmarking Reward Hack Detection in Code Environments via Contrastive Analysis20264
  10. 15Countdown-Code: A Testbed for Studying The Emergence and Generalization of Reward Hacking in RLVR20264
All 30
  1. 16Inoculation Prompting: Instructing LLMs to misbehave at train-time improves test-time alignment20254
  2. 17Auto-Rubric: Learning From Implicit Weights to Explicit Rubrics for Reward Modeling20254
  3. 18Feedback Loops With Language Models Drive In-Context Reward Hacking20244
  4. 19Open Rubric System: Scaling Reinforcement Learning with Pairwise Adaptive Rubric20263
  5. 20IR3: Contrastive Inverse Reinforcement Learning for Interpretable Detection and Mitigation of Reward Hacking20263
  6. 21Monitoring Emergent Reward Hacking During Generation via Internal Activations20263
  7. 22Reward Hacking Benchmark: Measuring Exploits in LLM Agents with Tool Use20263
  8. 23Adversarial Reward Auditing for Active Detection and Mitigation of Reward Hacking20262
  9. 24Learning Query-Specific Rubrics from Human Preferences for DeepResearch Report Generation20262
  10. 25Robust Optimization for Mitigating Reward Hacking with Correlated Proxies20262
  11. 26Reward Hacking in Rubric-Based Reinforcement Learning20262
  12. 27MONA: Myopic Optimization with Non-myopic Approval Can Mitigate Multi-step Reward Hacking20252
  13. 28Adversarial Training of Reward Models20252
  14. 29Detecting Proxy Gaming in RL and LLM Alignment via Evaluator Stress Tests20252
  15. 30Spontaneous Reward Hacking in Iterative Self-Refinement20242
  16. 31Reward Under Attack: Analyzing the Robustness and Hackability of Process Reward Models20261
  17. 32Reward Hacking as Equilibrium under Finite Evaluation20261
  18. 33Rubrics to Tokens: Bridging Response-level Rubrics and Token-level Rewards in Instruction Following Tasks20261
  19. 34C2: Scalable Rubric-Augmented Reward Modeling from Binary Preferences20261
  20. 35Hack-Verifiable Environments: Towards Evaluating Reward Hacking at Scale20261