mlsanityData to 7 Oct 2026

InfAlign: Inference-aware language model alignment

arXiv 2412.19792 ↗First posted 27 Dec 2024cs.LG cs.CL cs.ITcited by 34

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Language model alignment is a critical step in training modern generative language models. Alignment targets to improve win rate of a sample from the aligned model against the base model. Today, we are increasingly using inference-time algorithms (e.g., Best-of-N, controlled decoding, tree search) to decode from language models rather than standard sampling. We show that this train/test mismatch makes standard RLHF framework sub-optimal in view of such inference-time methods. To this end, we propose a framework for inference-aware alignment (InfAlign), which aims to optimize inference-time win rate of the aligned policy against the base model. We prove that for any inference-time decoding procedure, the optimal aligned policy is the solution to the standard RLHF problem with a transformation of the reward. This motivates us to provide the calibrate-and-transform RL (InfAlign-CTRL) algorithm to solve this problem, which involves a reward calibration step and a KL-regularized reward maximization step with a transformation of the calibrated reward. For best-of-N sampling and best-of-N jailbreaking, we propose specific transformations offering up to 3-8% improvement on inference-time win rates. Finally, we also show that our proposed reward calibration method is a strong baseline for optimizing standard win rate.

Key facts. Cited by 34 papers in the map (data through 7 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 54 of its 68 references that are in the map. First posted on arXiv 27 Dec 2024.

Builds on

54 of 68 references.

Closest ancestors · 9

Same direction, newest first

  1. 2024TreeBoN: Enhancing Inference-Time Alignment with Speculative Tree-Search and Best-of-N Sampling18 Oct
  2. 2024Variational Best-of-N Alignment8 Jul
  3. 2024Information Theoretic Guarantees For Policy Alignment In Large Language Models9 Jun
  4. 2024BoNBoN Alignment for Large Language Models and the Sweetness of Best-of-n Sampling2 Jun
  5. 2024Transfer Q Star: Principled Decoding for LLM Alignment30 May
  6. 2024Robust Preference Optimization through Reward Model Distillation29 May
  7. 2024Asymptotics of Language Model Alignment2 Apr
  8. 2023A General Theoretical Paradigm to Understand Learning from Human Preferences18 Oct
  9. 2022Training language models to follow instructions with human feedback4 Mar

Other references · 45

  1. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  2. 2017Proximal Policy Optimization Algorithmscited by 14,441
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  4. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  5. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  6. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  7. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,601
  8. 2017Deep reinforcement learning from human preferencescited by 3,846
  9. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  10. 2023Self-Refine: Iterative Refinement with Self-Feedbackcited by 3,317
  11. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,155
  12. 2020Learning to summarize from human feedbackcited by 2,562
33 more
  1. 2024Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameterscited by 1,928
  2. 2016Concrete Problems in AI Safetycited by 1,582
  3. 2023Jailbreaking Black Box Large Language Models in Twenty Queriescited by 1,326
  4. 2023PaLM 2 Technical Reportcited by 1,154
  5. 2022Scaling Laws for Reward Model Overoptimizationcited by 1,056
  6. 2024Large Language Monkeys: Scaling Inference Compute with Repeated Samplingcited by 946
  7. 2024RewardBench: Evaluating Reward Models for Language Modelingcited by 480
  8. 2023Fine-Grained Human Feedback Gives Better Rewards for Language Model Trainingcited by 399
  9. 2024A StrongREJECT for Empty Jailbreakscited by 352
  10. 2023A Real-World WebAgent with Planning, Long Context Understanding, and Program Synthesiscited by 350
  11. 2022The Effects of Reward Misspecification: Mapping and Mitigating Misaligned Modelscited by 312
  12. 2023Reward Model Ensembles Help Mitigate Overoptimizationcited by 225
  13. 2023Helping or Herding? Reward Model Ensembles Mitigate but do not Eliminate Reward Hackingcited by 195
  14. 2024From Decoding to Meta-Generation: Inference-time Algorithms for Large Language Modelscited by 136
  15. 2023Controlled Decoding from Language Modelscited by 133
  16. 2022RL with KL penalties is better viewed as Bayesian inferencecited by 130
  17. 2022Calibrating Sequence likelihood Improves Conditional Language Generationcited by 121
  18. 2024Theoretical guarantees on the best-of-n alignment policycited by 116
  19. 2020Tilted Empirical Risk Minimizationcited by 103
  20. 2023Confronting Reward Model Overoptimization with Constrained RLHFcited by 96
  21. 2024Inference-Aware Fine-Tuning for Best-of-N Sampling in Large Language Modelscited by 86
  22. 2024Probabilistic Inference in Language Models via Twisted Sequential Monte Carlocited by 85
  23. 2023FLIRT: Feedback Loop In-context Red Teamingcited by 81
  24. 2024BOND: Aligning LLMs with Best-of-N Distillationcited by 73
  25. 2024Best-of-N Jailbreakingcited by 71
  26. 2021On Tilted Losses in Machine Learning: Theory and Applicationscited by 42
  27. 2021PPL-MCTS: Constrained Textual Generation Through Discriminator-Guided MCTS Decodingcited by 32
  28. 2022Reward Gaming in Conditional Text Generationcited by 29
  29. 2024Calibrating Reasoning in Language Models with Internal Consistencycited by 27
  30. 2024Transforming and Combining Rewards for Aligning Large Language Modelscited by 24
  31. 2024International Scientific Report on the Safety of Advanced AI (Interim Report)cited by 18
  32. 2021To Beam Or Not To Beam: That is a Question of Cooperation for Language GANscited by 14
  33. 2024Guaranteed Generation from Large Language Modelscited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 16; 2026: 18.

34 papers.

Sort:

By area:

  1. 2025A Survey of Frontiers in LLM Reasoning: Inference Scaling, Learning to Reason, and Agentic Systemscited by 108
  2. 2025Is Best-of-N the Best of Them? Coverage, Scaling, and Optimality in Inference-Time Alignmentcited by 91
  3. 2025Pass@K Policy Optimization: Solving Harder Reinforcement Learning Problemscited by 83
  4. 2025What Makes a Reward Model a Good Teacher? An Optimization Perspectivecited by 70
  5. 2025Optimizing Language Models for Inference Time Objectives using Reinforcement Learningcited by 49
  6. 2025Chasing the Tail: Effective Rubric-based Reward Modeling for Large Language Model Post-Trainingcited by 46
  7. 2026POPE: Learning to Reason on Hard Problems via Privileged On-Policy Explorationcited by 39
  8. 2025Inference-Time Reward Hacking in Large Language Modelscited by 32
  9. 2025Single-stream Policy Optimizationcited by 27
  10. 2025Representation-Based Exploration for Language Models: From Test-Time to Post-Trainingcited by 16
  11. 2026Reuse your FLOPs: Scaling RL on Hard Problems by Conditioning on Very Off-Policy Prefixescited by 16
  12. 2025REINFORCE Adversarial Attacks on Large Language Models: An Adaptive, Distributional, and Semantic Objectivecited by 15
22 more
  1. 2025Risk-Sensitive RL for Alleviating Exploration Dilemmas in Large Language Modelscited by 12
  2. 2025KL-Regularized RLHF with Multiple Reference Models: Exact Solutions and Sample Complexitycited by 11
  3. 2025Best-of-Majority: Minimax-Optimal Strategy for Pass@k Inference Scalingcited by 8
  4. 2026Vector Policy Optimization: Training for Diversity Improves Test-Time Searchcited by 8
  5. 2025Why is Your Language Model a Poor Implicit Reward Model?cited by 6
  6. 2026Why Pass@k Optimization Can Degrade Pass@1: Prompt Interference in LLM Post-trainingcited by 6
  7. 2026EvoIdeator: Evolving Scientific Ideas through Checklist-Grounded Reinforcement Learningcited by 4
  8. 2026Power Distribution Bridges Sampling, Self-Reward RL, and Self-Distillationcited by 3
  9. 2025PICACO: Pluralistic In-Context Value Alignment of LLMs via Total Correlation Optimizationcited by 2
  10. 2026Best-of-Tails: Bridging Optimism and Pessimism in Inference-Time Alignmentcited by 2
  11. 2026CLR-voyance: Reinforcing Open-Ended Reasoning for Inpatient Clinical Decision Support with Outcome-Aware Rubricscited by 2
  12. 2026Using Reward Uncertainty to Induce Diverse Behaviour in Reinforcement Learningcited by 2
  13. 2025Saffron-1: Safety Inference Scalingcited by 1
  14. 2026When Errors Can Be Beneficial: A Categorization of Imperfect Rewards for Policy Gradientcited by 1
  15. 2026Calibrating the Evaluator: Does Probability Calibration Mitigate Preference Coupling in LLM Agent Feedback Loops?cited by 1
  16. 2026Learnable Chernoff Baselines for Inference-Time Alignmentcited by 0
  17. 2026What should post-training optimize? A test-time scaling law perspectivecited by 0
  18. 2026ExpRL: Exploratory RL for LLM Mid-Trainingcited by 0
  19. 2026Best-of-Better-N: Generating Pre-Aligned Responses with In-Context Learningcited by 0
  20. 2026Inference-Time Policy Alignment for Fair Reinforcement Learningcited by 0
  21. 2026Truncate Bad, Upweight Good: BoN-Style Distillation via Rank-Based Classificationcited by 0
  22. 2026Direct Optimization of Generators for Search in Automated Theorem Provingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024DeAL: Decoding-time Alignment for Large Language Models
  2. 2025W2S-AlignTree: Weak-to-Strong Inference-Time Alignment for Large Language Models via Monte Carlo Tree Search
  3. 2026Gradient-Guided Reward Optimization for Inference-time Alignment
  4. 2025Inference-time Alignment in Continuous Space
  5. 2025On Almost Surely Safe Alignment of Large Language Models at Inference-Time
  6. 2024Inference-Time Language Model Alignment via Integrated Value Guidance
  7. 2025Alignment-Aware Decoding
  8. 2026Inference-Aware Meta-Alignment of LLMs via Non-Linear GRPO