mlsanityData to 7 Oct 2026

Gradient-Guided Reward Optimization for Inference-time Alignment

arXiv 2606.09635 ↗First posted 8 Jun 2026cs.CL cs.LGcited by 0

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Ensuring the reliability of Large Language Models (LLMs) under distribution drift requires inference-time adaptation. While inference-time alignment methods such as Best-of-N and rejection sampling are widely used, they frame the task as a sampling-intensive, reward-guided search, leading to two key limitations: their performance is bounded by the base model's generation quality, and their reliance on imperfect reward models makes them vulnerable to reward hacking. To address these challenges, we introduce Gradient-Guided Reward Optimization (GGRO), a lightweight inference-time method that performs targeted, minimal intervention during decoding via gradient guidance. Specifically, GGRO monitors token-level entropy to identify high-uncertainty regions indicative of drift or misalignment. Upon detection, it responds by injecting nudging tokens, generated using gradient signals from an off-the-shelf reward model, to steer the generation trajectory rather than merely re-ranking samples. Experiments show that GGRO consistently improves inference-time alignment across safety, helpfulness, and reasoning benchmarks. It also increases coverage of high-quality responses and robustness to reward hacking, with minimal computational overhead. Code is available at https://github.com/lhk2004/GGRO.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 41 of its 50 references that are in the map. First posted on arXiv 8 Jun 2026.

Builds on

41 of 50 references.

Closest ancestors · 8

Same direction, newest first

  1. 2025Inference-Time Reward Hacking in Large Language Models24 Jun
  2. 2025Inference-time Alignment in Continuous Space26 May
  3. 2025Sample, Don't Search: Rethinking Test-Time Alignment for Language Models4 Apr
  4. 2025Is Best-of-N the Best of Them? Coverage, Scaling, and Optimality in Inference-Time Alignment27 Mar
  5. 2025Evaluation of Best-of-N Sampling Strategies for Language Model Alignment18 Feb
  6. 2024Cascade Reward Sampling for Efficient Decoding-Time Alignment24 Jun
  7. 2024Weak-to-Strong Search: Align Large Language Models via Searching over Small Language Models29 May
  8. 2023Statistical Rejection Sampling Improves Preference Optimization13 Sep

Other references · 33

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2024The Llama 3 Herd of Modelscited by 14,192
  3. 2025Qwen3 Technical Reportcited by 8,195
  4. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  5. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  6. 2025Gemini 2.5: Pushing the Frontier with Advanced Reasoning, Multimodality, Long Context, and Next Generation Agentic Capabilitiescited by 3,655
  7. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  8. 2020Learning to summarize from human feedbackcited by 2,562
  9. 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,758
  10. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,722
  11. 2023Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!cited by 1,106
  12. 2023XSTest: A Test Suite for Identifying Exaggerated Safety Behaviours in Large Language Modelscited by 477
21 more
  1. 2024Jailbreaking Leading Safety-Aligned LLMs with Simple Adaptive Attackscited by 450
  2. 2024Safety Alignment Should Be Made More Than Just a Few Tokens Deepcited by 450
  3. 2025Skywork-Reward-V2: Scaling Preference Data Curation via Human-AI Synergycited by 204
  4. 2022COLD Decoding: Energy-based Constrained Text Generation with Langevin Dynamicscited by 159
  5. 2024Regularizing Hidden States Enables Learning Generalizable Reward Model for LLMscited by 142
  6. 2025RewardBench 2: Advancing Reward Model Evaluationcited by 136
  7. 2024Adaptive Activation Steering: A Tuning-Free LLM Truthfulness Improvement Method for Diverse Hallucinations Categoriescited by 74
  8. 2024SCANS: Mitigating the Exaggerated Safety for LLMs via Safety-Conscious Activation Steeringcited by 45
  9. 2025Multi-Attribute Steering of Language Models via Targeted Interventioncited by 44
  10. 2022A Langevin-like Sampler for Discrete Distributionscited by 42
  11. 2025To Steer or Not to Steer? Mechanistic Error Reduction with Abstention for Language Modelscited by 26
  12. 2023BOLT: Fast Energy-based Controlled Text Generation with Tunable Biasescited by 24
  13. 2024Nudging: Inference-time Alignment of LLMs via Guided Decodingcited by 22
  14. 2025Revisiting Uncertainty Estimation and Calibration of Large Language Modelscited by 22
  15. 2024Self-Consistency Boosts Calibration for Math Reasoningcited by 12
  16. 2024A Critical Look At Tokenwise Reward-Guided Text Generationcited by 12
  17. 2024Gradient-based Discrete Sampling with Automatic Cyclical Schedulingcited by 9
  18. 2025From Outcomes to Processes: Guiding PRM Learning from ORM for Inference-Time Alignmentcited by 9
  19. 2024mathbbUSCD: Improving Code Generation of LLMs by Uncertainty-Aware Selective Contrastive Decodingcited by 6
  20. 2025Controlled LLM Decoding via Discrete Auto-regressive Biasingcited by 5
  21. 2026Thinking by Subtraction: Confidence-Driven Contrastive Decoding for LLM Reasoningcited by 5

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Safe Inference-Time Alignment via Lagrangian Reward Augmentation
  2. 2026Revisiting the (Sub)Optimality of Best-of-N for Inference-Time Alignment
  3. 2024Inference Time Alignment with Reward-Guided Tree Search
  4. 2025AdaBoN: Adaptive Best-of-N Alignment
  5. 2025Reward-Shifted Speculative Sampling Is An Efficient Test-Time Weak-to-Strong Aligner
  6. 2026From Curiosity to Caution: Mitigating Reward Hacking for Best-of-N with Pessimism
  7. 2024InfAlign: Inference-aware language model alignment
  8. 2026Best-of-Tails: Bridging Optimism and Pessimism in Inference-Time Alignment