mlsanityData to 14 Oct 2026

Inference-Time Reward Hacking in Large Language Models

arXiv 2506.19248 ↗First posted 24 Jun 2025cs.LGcited by 34

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

A common paradigm to improve the performance of large language models is optimizing for a reward model. Reward models assign a numerical score to an LLM's output that indicates, for example, how likely it is to align with user preferences or safety goals. However, reward models are never perfect. They inevitably function as proxies for complex desiderata such as correctness, helpfulness, and safety. By overoptimizing for a misspecified reward, we can subvert intended alignment goals and reduce overall performance, a phenomenon commonly referred to as reward hacking. In this work, we characterize reward hacking in inference-time alignment and demonstrate when and how we can mitigate it by hedging on the proxy reward. We study this phenomenon under Best-of-n (BoN) and Soft Best-of-n (SBoN), and we introduce Best-of-Poisson (BoP) that provides an efficient, near-exact approximation of the optimal reward-KL divergence policy at inference time. We show that the characteristic pattern of hacking as observed in practice (where the true reward first increases before declining) is an inevitable property of a broad class of inference-time mechanisms, including BoN and BoP. To counter this effect, we introduce HedgeTune, an efficient algorithm to find the optimal inference-time parameter. We demonstrate that hedging mitigates reward hacking and achieves superior reward-distortion tradeoffs on math, reasoning, and human-preference setups.

Key facts. Cited by 34 papers in the map (data through 14 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 70 of its 94 references that are in the map. First posted on arXiv 24 Jun 2025.

Builds on

70 of 94 references.

Closest ancestors · 13

Same direction, newest first

  1. 2025Soft Best-of-n Sampling for Model Alignment6 May
  2. 2025Is Best-of-N the Best of Them? Coverage, Scaling, and Optimality in Inference-Time Alignment27 Mar
  3. 2025Evaluation of Best-of-N Sampling Strategies for Language Model Alignment18 Feb
  4. 2025Robust LLM Alignment via Distributionally Robust Direct Preference Optimization4 Feb
  5. 2024InfAlign: Inference-aware language model alignment27 Dec
  6. 2024Faster WIND: Accelerating Iterative Best-of-N Distillation for LLM Alignment28 Oct
  7. 2024TreeBoN: Enhancing Inference-Time Alignment with Speculative Tree-Search and Best-of-N Sampling18 Oct
  8. 2024Correcting the Mythos of KL-Regularization: Direct Alignment without Overoptimization via Chi-Squared Preference Optimization18 Jul
  9. 2024Variational Best-of-N Alignment8 Jul
  10. 2024Information Theoretic Guarantees For Policy Alignment In Large Language Models9 Jun
  11. 2024BoNBoN Alignment for Large Language Models and the Sweetness of Best-of-n Sampling2 Jun
  12. 2024Asymptotics of Language Model Alignment2 Apr
1 more
  1. 2024InternLM2 Technical Report26 Mar

Other references · 57

  1. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  2. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  3. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  4. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  5. 2017Deep reinforcement learning from human preferencescited by 3,912
  6. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  7. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,879
  8. 2020Learning to summarize from human feedbackcited by 2,588
  9. 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,848
  10. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,786
  11. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,745
  12. 2016Concrete Problems in AI Safetycited by 1,642
38 more
  1. 2022Scaling Laws for Reward Model Overoptimizationcited by 1,095
  2. 2023Towards Understanding Sycophancy in Language Modelscited by 1,040
  3. 2023AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedbackcited by 748
  4. 2024RewardBench: Evaluating Reward Models for Language Modelingcited by 499
  5. 2025Monitoring Reasoning Models for Misbehavior and the Risks of Promoting Obfuscationcited by 384
  6. 2022The Effects of Reward Misspecification: Mapping and Mitigating Misaligned Modelscited by 328
  7. 2024Skywork-Reward: Bag of Tricks for Reward Modeling in LLMscited by 327
  8. 2023Evaluating Large Language Models at Evaluating Instruction Followingcited by 302
  9. 2017Inverse Reward Designcited by 249
  10. 2023Reward Model Ensembles Help Mitigate Overoptimizationcited by 231
  11. 2023Helping or Herding? Reward Model Ensembles Mitigate but do not Eliminate Reward Hackingcited by 200
  12. 2024Sycophancy to Subterfuge: Investigating Reward-Tampering in Large Language Modelscited by 178
  13. 2024WARM: On the Benefits of Weight Averaged Reward Modelscited by 147
  14. 2024Regularizing Hidden States Enables Learning Generalizable Reward Model for LLMscited by 142
  15. 2024From Decoding to Meta-Generation: Inference-time Algorithms for Large Language Modelscited by 137
  16. 2024ODIN: Disentangled Reward Mitigates Hacking in RLHFcited by 136
  17. 2023Controlled Decoding from Language Modelscited by 135
  18. 2022Goal Misgeneralization: Why Correct Specifications Aren't Enough For Correct Goalscited by 126
  19. 2024Theoretical guarantees on the best-of-n alignment policycited by 124
  20. 2024Provably Mitigating Overoptimization in RLHF: Your SFT Loss is Implicitly an Adversarial Regularizercited by 109
  21. 2024OffsetBias: Leveraging Debiased Data for Tuning Evaluatorscited by 94
  22. 2024Inference-Aware Fine-Tuning for Best-of-N Sampling in Large Language Modelscited by 89
  23. 2024How to Evaluate Reward Models for RLHFcited by 88
  24. 2025Reward Shaping to Mitigate Reward Hacking in RLHFcited by 88
  25. 2024InfoRM: Mitigating Reward Hacking in RLHF via Information-Theoretic Reward Modelingcited by 79
  26. 2024Feedback Loops With Language Models Drive In-Context Reward Hackingcited by 77
  27. 2024BOND: Aligning LLMs with Best-of-N Distillationcited by 76
  28. 2024RRM: Robust Reward Model Training Mitigates Reward Hackingcited by 76
  29. 2024Iterative Data Smoothing: Mitigating Reward Overfitting and Overoptimization in RLHFcited by 57
  30. 2025Demonstrating specification gaming in reasoning modelscited by 56
  31. 2024Correlated Proxies: A New Definition and Improved Mitigation for Reward Hackingcited by 50
  32. 2025Beyond Reward Hacking: Causal Rewards for Large Language Model Alignmentcited by 42
  33. 2023Goodhart's Law in Reinforcement Learningcited by 41
  34. 2024Overcoming Reward Overoptimization via Adversarial Policy Optimization with Lightweight Uncertainty Estimationcited by 33
  35. 2023The Trickle-down Impact of Reward (In-)consistency on RLHFcited by 29
  36. 2024Spontaneous Reward Hacking in Iterative Self-Refinementcited by 26
  37. 2024Catastrophic Goodhart: regularizing RLHF with KL divergence does not mitigate heavy-tailed reward misspecificationcited by 23
  38. 2025AI Alignment at Your Discretioncited by 20

50 most cited of 57 All 57 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 33.

34 papers.

Sort:

By area:

  1. 2026Reward Hacking in the Era of Large Models: Mechanisms, Emergent Misalignment, Challengescited by 33
  2. 2025A Survey on Agentic Multimodal Large Language Modelscited by 23
  3. 2026From Rebound to Remedy: Understanding and Mitigating Reward Hacking via Representation Engineeringcited by 7
  4. 2026Test-Time Scaling in Reasoning LLMs: Inference Regimes, Evaluation, and Reproducibilitycited by 6
  5. 2026Can Revealed Preferences Clarify LLM Alignment and Steering?cited by 5
  6. 2026Improving Vision-language Models with Perception-centric Process Reward Modelscited by 4
  7. 2026Reward Hacking in Language Model Agents: Revisiting AI Safety Gridworldscited by 4
  8. 2026Revisiting the (Sub)Optimality of Best-of-N for Inference-Time Alignmentcited by 3
  9. 2026Best-of-Tails: Bridging Optimism and Pessimism in Inference-Time Alignmentcited by 3
  10. 2026From Curiosity to Caution: Mitigating Reward Hacking for Best-of-N with Pessimismcited by 3
  11. 2026Modularized Reinforcement Learning on LLMs: From MDP Creation to Exploration and Learningcited by 2
  12. 2026Verify, Repair, Repeat, or Stop? Robust Stopping for Noisy Verify-Repair Loops in LLM Agentscited by 2
22 more
  1. 2026Reward Hacking Challenges Oversight of Autonomous Research Agentscited by 2
  2. 2026Inference-Aware Meta-Alignment of LLMs via Non-Linear GRPOcited by 1
  3. 2026Disentangling Deception and Hallucination Failures in LLMscited by 0
  4. 2026AI Space Physics: Constitutive boundary semantics for open AI institutionscited by 0
  5. 2026DARC: Disagreement-Aware Alignment via Risk-Constrained Decodingcited by 0
  6. 2026Agentic AI, Retrieval-Augmented Generation, and the Institutional Turn: Legal Architectures and Financial Governance in the Age of Distributional AGIcited by 0
  7. 2026PAC-CF: Calibrating Irreversible Frontier Pruning in LLM-Guided Searchcited by 0
  8. 2026Likelihood scoring for continuations of mathematical text: a self-supervised benchmark with tests for shortcut vulnerabilitiescited by 0
  9. 2026Epistemic Uncertainty for Test-Time Discoverycited by 0
  10. 2026Temper and Tilt Lead to SLOP: Reward Hacking Mitigation with Inference-Time Alignmentcited by 0
  11. 2026Gradient-Guided Reward Optimization for Inference-time Alignmentcited by 0
  12. 2026Best-of-Better-N: Generating Pre-Aligned Responses with In-Context Learningcited by 0
  13. 2026Truncate Bad, Upweight Good: BoN-Style Distillation via Rank-Based Classificationcited by 0
  14. 2026Privacy Without Regret: Differentially Private Inference-Time Alignmentcited by 0
  15. 2026Mitigating Over-Optimization in PRM-Guided Search in Mathematical Reasoning by Optimizing the Guidecited by 0
  16. 2026Attention Sensitivity Is Not Enough: Dissociating Attention-Level and Behavioural In-Context Learning under Fine-Tuningcited by 0
  17. 2026Optimizing the Score, Losing Sight of the Task: Reward Hacking Across Weights, Selection, and Promptscited by 0
  18. 2026Scaling Properties of Same-Family On-Policy Distillationcited by 0
  19. 2026Verifier Errors in RLVR: Reward Hacking, Limits of Feedback, and Selective Controlcited by 0
  20. 2026When Updating Stops Being Learning: Rethinking LLM Self-Evolution via learnable information gaincited by 0
  21. 2026Holdout Best-of-N: Unbiased Evaluation and Its Costcited by 0
  22. 2026Tail Competition Explains Scaling in Best-of-N Sampling for Verifiable Problemscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Small Reward Models via Backward Inference
  2. 2026Theoretical Limits of Language Model Alignment
  3. 2026HARVE: Hacking-Aware Reward-Head Vector Editing for Robust Reward Models
  4. 2025reWordBench: Benchmarking and Improving the Robustness of Reward Models with Transformed Inputs
  5. 2025Sailing by the Stars: A Survey on Reward Models and Learning Strategies for Learning from Rewards
  6. 2024Rethinking the Role of Proxy Rewards in Language Model Alignment
  7. 2024Elephant in the Room: Unveiling the Impact of Reward Model Quality in Alignment
  8. 2026On the Power of (Approximate) Reward Models for Inference-Time Scaling