mlsanityData to 7 Oct 2026

Test-time reward-guided alignment of language models by importance sampling on pre-logit space

arXiv 2510.26219 ↗First posted 30 Oct 2025cs.LG cs.AIcited by 0

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Test-time alignment of large language models (LLMs) attracts attention because fine-tuning of LLMs requires high computational costs. In this paper, we propose a new test-time reward-guided alignment method called adaptive importance sampling on pre-logits (AISP) on the basis of the sampling-based model predictive control with the stochastic control input. AISP applies the Gaussian perturbation into pre-logits, which are outputs of the penultimate layer, so as to maximize expected rewards with respect to the mean of the perturbation. We demonstrate that the optimal mean is obtained by importance sampling with sampled rewards. AISP outperforms best-of-n sampling in terms of rewards over the number of used samples and achieves higher rewards than other reward-based test-time alignment methods.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 39 of its 53 references that are in the map. First posted on arXiv 30 Oct 2025.

Builds on

39 of 53 references.

Closest ancestors · 11

Same direction, newest first

  1. 2025Is Best-of-N the Best of Them? Coverage, Scaling, and Optimality in Inference-Time Alignment27 Mar
  2. 2025Evaluation of Best-of-N Sampling Strategies for Language Model Alignment18 Feb
  3. 2024GenARM: Reward Guided Generation with Autoregressive Reward Model for Test-time Alignment10 Oct
  4. 2024Cascade Reward Sampling for Efficient Decoding-Time Alignment24 Jun
  5. 2024Aligning Large Language Models with Representation Editing: A Control Perspective10 Jun
  6. 2024Transfer Q Star: Principled Decoding for LLM Alignment30 May
  7. 2024Value Augmented Sampling for Language Model Alignment and Personalization10 May
  8. 2024Asymptotics of Language Model Alignment2 Apr
  9. 2023RAIN: Your Language Models Can Align Themselves without Finetuning13 Sep
  10. 2023Aligning Language Models with Preferences through f-divergence Minimization16 Feb
  11. 2022Training language models to follow instructions with human feedback4 Mar

Other references · 28

  1. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  2. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  3. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  4. 2017Deep reinforcement learning from human preferencescited by 3,846
  5. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  6. 2023Let's Verify Step by Stepcited by 3,598
  7. 2021TruthfulQA: Measuring How Models Mimic Human Falsehoodscited by 2,919
  8. 2019Fine-Tuning Language Models from Human Preferencescited by 2,124
  9. 2024Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameterscited by 1,928
  10. 2018A Simple Unified Framework for Detecting Out-of-Distribution Samples and Adversarial Attackscited by 1,472
  11. 2024Large Language Monkeys: Scaling Inference Compute with Repeated Samplingcited by 946
  12. 2021Understanding Dataset Difficulty with V-Usable Informationcited by 288
16 more
  1. 2022A Contrastive Framework for Neural Text Generationcited by 224
  2. 2024Advancing LLM Reasoning Generalists with Preference Treescited by 191
  3. 2023Controlled Decoding from Language Modelscited by 133
  4. 2022RL with KL penalties is better viewed as Bayesian inferencecited by 130
  5. 2024Theoretical guarantees on the best-of-n alignment policycited by 116
  6. 2017Information Theoretic Model Predictive Control: Theory and Applications to Autonomous Drivingcited by 81
  7. 2024BOND: Aligning LLMs with Best-of-N Distillationcited by 73
  8. 2021Revisiting Mahalanobis Distance for Transformer-Based Out-of-Domain Detectioncited by 67
  9. 2022Critic-Guided Decoding for Controlled Text Generationcited by 45
  10. 2025Syntactic and Semantic Control of Large Language Models via Sequential Monte Carlocited by 45
  11. 2024TLCR: Token-Level Continuous Reward for Fine-grained Reinforcement Learning from Human Feedbackcited by 43
  12. 2020Stein Variational Model Predictive Controlcited by 29
  13. 2021How is BERT surprised? Layerwise detection of linguistic anomaliescited by 24
  14. 2022Variational Inference MPC using Normalizing Flows and Out-of-Distribution Projectioncited by 13
  15. 2025Soft Reasoning: Navigating Solution Spaces in Large Language Models through Controlled Embedding Explorationcited by 6
  16. 2024PID Control-Based Self-Healing to Improve the Robustness of Large Language Modelscited by 3

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Sample, Don't Search: Rethinking Test-Time Alignment for Language Models
  2. 2025AdaBoN: Adaptive Best-of-N Alignment
  3. 2025Leveraging Importance Sampling to Detach Alignment Modules from Large Language Models
  4. 2025Test-Time Alignment for Large Language Models via Textual Model Predictive Control
  5. 2025Reward-Shifted Speculative Sampling Is An Efficient Test-Time Weak-to-Strong Aligner
  6. 2024Dynamic Rewarding with Prompt Optimization Enables Tuning-free Self-Alignment of Language Models
  7. 2025Towards Reliable, Uncertainty-Aware Alignment
  8. 2026Revisiting the (Sub)Optimality of Best-of-N for Inference-Time Alignment