mlsanityData to 14 Oct 2026

Temper and Tilt Lead to SLOP: Reward Hacking Mitigation with Inference-Time Alignment

arXiv 2605.13537 ↗First posted 13 May 2026cs.LG cs.AI cs.CLcited by 0

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

Inference-time alignment techniques offer a lightweight alternative or complement to costly reinforcement learning, while enabling continual adaptation as alignment objectives and reward targets evolve. Existing theoretical analyses justify these methods as approximations to sampling from distributions optimally tilted toward a given reward model. We extend these techniques by introducing reference-model temperature adjustment, which leads to further generalization of inference-time alignment to ensembles of generative reward models combined as a sharpened logarithmic opinion pool (SLOP). To mitigate reward hacking, we propose an algorithm for calibrating SLOP weight parameters and experimentally demonstrate that it improves robustness while preserving alignment performance.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 30 of its 41 references that are in the map. First posted on arXiv 13 May 2026.

Builds on

30 of 41 references.

Closest ancestors · 7

Same direction, newest first

  1. 2023Reward-Augmented Decoding: Efficient Controlled Text Generation With a Unidirectional Reward Model14 Oct
  2. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May
  3. 2022Scaling Laws for Reward Model Overoptimization19 Oct
  4. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback12 Apr
  5. 2021WebGPT: Browser-assisted question-answering with human feedback17 Dec
  6. 2019Fine-Tuning Language Models from Human Preferences18 Sep
  7. 2017Deep reinforcement learning from human preferences12 Jun

Other references · 23

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  3. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  4. 2025Qwen3 Technical Reportcited by 8,820
  5. 2023Visual Instruction Tuningcited by 7,869
  6. 2023Improved Baselines with Visual Instruction Tuningcited by 4,418
  7. 2025Qwen3-VL Technical Reportcited by 2,793
  8. 2020Learning to summarize from human feedbackcited by 2,588
  9. 2024Qwen2 Technical Reportcited by 2,120
  10. 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,919
  11. 2025Gemma 3 Technical Reportcited by 1,824
  12. 2023AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedbackcited by 748
11 more
  1. 2022The Effects of Reward Misspecification: Mapping and Mitigating Misaligned Modelscited by 328
  2. 2022RL with KL penalties is better viewed as Bayesian inferencecited by 135
  3. 2025Is Best-of-N the Best of Them? Coverage, Scaling, and Optimality in Inference-Time Alignmentcited by 97
  4. 2022On Reinforcement Learning and Distribution Matching for Fine-Tuning Language Models with no Catastrophic Forgettingcited by 82
  5. 2024Asymptotics of Language Model Alignmentcited by 47
  6. 2025Inference-Time Reward Hacking in Large Language Modelscited by 34
  7. 2025Soft Best-of-n Sampling for Model Alignmentcited by 30
  8. 2024Regularized Best-of-N Sampling with Minimum Bayes Risk Objective for Language Model Alignmentcited by 25
  9. 2024Information Theoretic Guarantees For Policy Alignment In Large Language Modelscited by 23
  10. 2025Best-of-N through the Smoothing Lens: KL Divergence and Regret Analysiscited by 17
  11. 2025Rethinking Approximate Gaussian Inference in Classificationcited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Reward Shaping for (Inference-Time) Alignment: A Stackelberg Game Perspective
  2. 2026Diamond Maps: Efficient Reward Alignment via Stochastic Flow Maps
  3. 2024Solving the Inverse Alignment Problem for Efficient RLHF
  4. 2023Helping or Herding? Reward Model Ensembles Mitigate but do not Eliminate Reward Hacking
  5. 2026Revisiting the (Sub)Optimality of Best-of-N for Inference-Time Alignment
  6. 2026Towards a theory of inference-time alignment with unknown rewards
  7. 2026Countdown-Code: A Testbed for Studying The Emergence and Generalization of Reward Hacking in RLVR
  8. 2026Uncertainty-Aware Reward Modeling for Stable RLHF