mlsanityData to 14 Oct 2026

Towards a theory of inference-time alignment with unknown rewards

arXiv 2608.15402 ↗First posted 15 Aug 2026cs.LGcited by 0

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Generative model alignment has received broad interest, and significant progress has been made in supervised fine-tuning and inference-time computation. Yet, alignment has remained poorly understood from a statistical learning perspective. We formulate inference-time alignment as a weak-to-strong learning problem, where a reference policy (weak model) is assumed to be fairly good and the goal is to produce a strong model that predicts a good response at test time with arbitrarily high probability. Our problem is formulated as learning from scratch --- everything is learned from data rather than assuming access to a good reward estimate, and thus differs from the existing inference-time alignment theory. Our framework shares similarity to the recent work of Joshi et al., (arXiv:2510.15464), where for each prompt, there could be multiple good responses. Our definition of the alignment learnability follows the standard PAC learning principle. We introduce a novel combinatorial dimension of the reward class which we call the alignment dimension, and show that it completely characterizes the alignment learnability --- a reward class is alignment learnable if and only if its alignment dimension is finite. The core of our learning procedure works by learning a pairwise comparator and then running a tournament over candidate responses. We believe that our results might shed light toward establishing a complete theoretical understanding of alignment.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 39 of its 49 references that are in the map. First posted on arXiv 15 Aug 2026.

Builds on

39 of 49 references.

Closest ancestors · 9

Same direction, newest first

  1. 2026Revisiting the (Sub)Optimality of Best-of-N for Inference-Time Alignment5 Mar
  2. 2025Best-of-N through the Smoothing Lens: KL Divergence and Regret Analysis8 Jul
  3. 2025Is Best-of-N the Best of Them? Coverage, Scaling, and Optimality in Inference-Time Alignment27 Mar
  4. 2024GenARM: Reward Guided Generation with Autoregressive Reward Model for Test-time Alignment10 Oct
  5. 2024Correcting the Mythos of KL-Regularization: Direct Alignment without Overoptimization via Chi-Squared Preference Optimization18 Jul
  6. 2024SimPO: Simple Preference Optimization with a Reference-Free Reward23 May
  7. 2024Active Preference Learning for Large Language Models12 Feb
  8. 2023A General Theoretical Paradigm to Understand Learning from Human Preferences18 Oct
  9. 2022Training language models to follow instructions with human feedback4 Mar

Other references · 30

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  3. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  4. 2017Deep reinforcement learning from human preferencescited by 3,912
  5. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  6. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,211
  7. 2020Learning to summarize from human feedbackcited by 2,588
  8. 2019Fine-Tuning Language Models from Human Preferencescited by 2,150
  9. 2016Concrete Problems in AI Safetycited by 1,642
  10. 2024Tulu 3: Pushing Frontiers in Open Language Model Post-Trainingcited by 1,053
  11. 2024Self-Rewarding Language Modelscited by 681
  12. 2023RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedbackcited by 625
18 more
  1. 2020Is Pessimism Provably Efficient for Offline RL?cited by 364
  2. 2023Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraintcited by 350
  3. 2025Reinforcement Learning with Verifiable Rewards Implicitly Incentivizes Correct Reasoning in Base LLMscited by 289
  4. 2021Bellman-consistent Pessimism for Offline Reinforcement Learningcited by 269
  5. 2024Self-Play Preference Optimization for Language Model Alignmentcited by 232
  6. 2024Theoretical guarantees on the best-of-n alignment policycited by 124
  7. 2024Fast Best-of-N Decoding via Speculative Rejectioncited by 112
  8. 2024Provably Mitigating Overoptimization in RLHF: Your SFT Loss is Implicitly an Adversarial Regularizercited by 109
  9. 2024Exploratory Preference Optimization: Harnessing Implicit Q*-Approximation for Sample-Efficient RLHFcited by 106
  10. 2022A Characterization of Multiclass Learnabilitycited by 67
  11. 2014Optimal Learners for Multiclass Problemscited by 65
  12. 2024Active Preference Optimization for Sample Efficient RLHFcited by 62
  13. 2021A Theory of PAC Learnability of Partial Concept Classescited by 60
  14. 2023Provable Offline Preference-Based Reinforcement Learningcited by 55
  15. 2024Sharp Analysis for KL-Regularized Contextual Bandits and RLHFcited by 33
  16. 2024Reward Model Learning vs. Direct Policy Optimization: A Comparative Analysis of Learning from Human Preferencescited by 27
  17. 2025Learning to Answer from Correct Demonstrationscited by 7
  18. 2026Learning with Multiple Correct Answers -- Regret Bounds under Different Feedback Modelscited by 4

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Learnable Chernoff Baselines for Inference-Time Alignment
  2. 2025Inference-time Alignment in Continuous Space
  3. 2024Inference Time Alignment with Reward-Guided Tree Search
  4. 2026Best-of-Better-N: Generating Pre-Aligned Responses with In-Context Learning
  5. 2025On Almost Surely Safe Alignment of Large Language Models at Inference-Time
  6. 2025AdaBoN: Adaptive Best-of-N Alignment
  7. 2026Asymptotic Universal Alignment: A New Alignment Framework via Test-Time Scaling
  8. 2026Safe Inference-Time Alignment via Lagrangian Reward Augmentation