mlsanityData to 7 Oct 2026

Reward Shaping for (Inference-Time) Alignment: A Stackelberg Game Perspective

arXiv 2602.02572 ↗First posted 31 Jan 2026cs.LG cs.AIcited by 2

On the map: LLM alignment / RLHF / preference optimisation → Reward modeling and RLHF

Abstract (arXiv)

Existing alignment methods directly use the reward model learned from user preference data to optimize an LLM policy, subject to KL regularization with respect to the base policy. This practice is suboptimal for maximizing user's utility because the KL regularization may cause the LLM to inherit the bias in the base policy that conflicts with user preferences. While amplifying rewards for preferred outputs can mitigate this bias, it also increases the risk of reward hacking. This tradeoff motivates the problem of optimally designing reward models under KL regularization. We formalize this reward model optimization problem as a Stackelberg game, and show that a simple reward shaping scheme can effectively approximate the optimal reward model. We empirically evaluate our method in inference-time alignment settings and demonstrate that it integrates seamlessly into existing alignment methods with minimal overhead. Our method consistently improves average reward and achieves win-tie rates exceeding 66% against all baselines, averaged across evaluation settings.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in Reward modeling and RLHF, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 49 of its 61 references that are in the map. First posted on arXiv 31 Jan 2026.

Builds on

49 of 61 references.

Closest ancestors · 14

Same direction, newest first

  1. 2025Skywork-Reward-V2: Scaling Preference Data Curation via Human-AI Synergy2 Jul
  2. 2025Strategyproof Reinforcement Learning from Human Feedback12 Mar
  3. 2025Reward Shaping to Mitigate Reward Hacking in RLHF26 Feb
  4. 2024RED: Unleashing Token-Level Rewards from Holistic Feedback via Reward Redistribution13 Nov
  5. 2024Skywork-Reward: Bag of Tricks for Reward Modeling in LLMs24 Oct
  6. 2024Iterative Nash Policy Optimization: Aligning LLMs with General Preferences via No-Regret Learning30 Jun
  7. 2024Direct Nash Optimization: Teaching Language Models to Self-Improve with General Preferences4 Apr
  8. 2024Improving Reinforcement Learning from Human Feedback Using Contrastive Rewards12 Mar
  9. 2024Transforming and Combining Rewards for Aligning Large Language Models1 Feb
  10. 2024A Minimaximalist Approach to Reinforcement Learning from Human Feedback8 Jan
  11. 2023Nash Learning from Human Feedback1 Dec
  12. 2023PARL: A Unified Framework for Policy Alignment in Reinforcement Learning from Human Feedback3 Aug
2 more
  1. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Model29 May
  2. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedback12 Apr

Other references · 35

  1. 2022Training language models to follow instructions with human feedbackcited by 15,712
  2. 2024The Llama 3 Herd of Modelscited by 14,192
  3. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  4. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  5. 2025Qwen3 Technical Reportcited by 8,195
  6. 2020Learning to summarize from human feedbackcited by 2,562
  7. 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,928
  8. 2024Self-Play Fine-Tuning Converts Weak Language Models to Strong Language Modelscited by 552
  9. 2023UltraFeedback: Boosting Language Models with Scaled AI Feedbackcited by 394
  10. 2021Persistent Anti-Muslim Bias in Large Language Modelscited by 376
  11. 2021Understanding Dataset Difficulty with V-Usable Informationcited by 288
  12. 2022A Contrastive Framework for Neural Text Generationcited by 224
23 more
  1. 2023Red-Teaming Large Language Models using Chain of Utterances for Safety-Alignmentcited by 213
  2. 2023Gender bias and stereotypes in Large Language Modelscited by 185
  3. 2023Controlled Decoding from Language Modelscited by 133
  4. 2024ARGS: Alignment as Reward-Guided Searchcited by 99
  5. 2023Verbosity Bias in Preference Labeling by Large Language Modelscited by 89
  6. 2024Aligning Large Language Models with Representation Editing: A Control Perspectivecited by 62
  7. 2018Incomplete Contracting and AI Alignmentcited by 60
  8. 2024DeAL: Decoding-time Alignment for Large Language Modelscited by 55
  9. 2024CULTURE-GEN: Revealing Global Cultural Perception in Language Models through Natural Language Promptingcited by 55
  10. 2024GenARM: Reward Guided Generation with Autoregressive Reward Model for Test-time Alignmentcited by 54
  11. 2024Transfer Q Star: Principled Decoding for LLM Alignmentcited by 46
  12. 2018Simple versus Optimal Contractscited by 33
  13. 2024Self-Play with Adversarial Critic: Provable and Scalable Offline Alignment for Language Modelscited by 16
  14. 2021Combinatorial Contractscited by 12
  15. 2023Principal-Agent Reward Shaping in MDPscited by 12
  16. 2023Deep Contract Design via Discontinuous Networkscited by 10
  17. 2024Incentivizing Quality Text Generation via Statistical Contractscited by 10
  18. 2023Of Models and Tin Men: A Behavioural Economics Study of Principal-Agent Problems in AI Alignment using Large-Language Modelscited by 9
  19. 2025Stackelberg Self-Annotation: A Robust Approach to Data-Efficient LLM Alignmentcited by 9
  20. 2024Principal-Agent Reinforcement Learning: Orchestrating AI Agents with Contractscited by 8
  21. 2025Robust Multi-Objective Controlled Decoding of Large Language Modelscited by 8
  22. 2025AI Testing Should Account for Sophisticated Strategic Behaviourcited by 6
  23. 2025Fair-PP: A Synthetic Dataset for Aligning LLM with Personalized Preferences of Social Equitycited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026Reward Hacking in the Era of Large Models: Mechanisms, Emergent Misalignment, Challengescited by 30
  2. 2026Explaining and Preventing Alignment Collapse in Iterative RLHFcited by 3

Similar, not cited

Close in wording, no citation either way

  1. 2025Direct Alignment with Heterogeneous Preferences
  2. 2024Elephant in the Room: Unveiling the Impact of Reward Model Quality in Alignment
  3. 2024Solving the Inverse Alignment Problem for Efficient RLHF
  4. 2025Towards Reliable, Uncertainty-Aware Alignment
  5. 2025The Reward Model Selection Crisis in Personalized Alignment
  6. 2025Probing Preference Representations: A Multi-Dimensional Evaluation and Analysis Method for Reward Models
  7. 2025A Survey on Progress in LLM Alignment from the Perspective of Reward Design
  8. 2025Inference-Time Reward Hacking in Large Language Models