Data to 7 Oct 2026

Variational Best-of-N Alignment

arXiv 2407.06057 ↗First posted 8 Jul 2024cs.CL cs.AI cs.LGcited by 49

On the map: LLM alignment / RLHF / preference optimisation → LLM preference optimization methods

Abstract (arXiv)

Best-of-N (BoN) is a popular and effective algorithm for aligning language models to human preferences. The algorithm works as follows: at inference time, N samples are drawn from the language model, and the sample with the highest reward, as judged by a reward model, is returned as the output. Despite its effectiveness, BoN is computationally expensive; it reduces sampling throughput by a factor of N. To make BoN more efficient at inference time, one strategy is to fine-tune the language model to mimic what BoN does during inference. To achieve this, we derive the distribution induced by the BoN algorithm. We then propose to fine-tune the language model to minimize backward KL divergence to the BoN distribution. Our approach is analogous to mean-field variational inference and, thus, we term it variational BoN (vBoN). To the extent this fine-tuning is successful and we end up with a good approximation, we have reduced the inference cost by a factor of N. Our experiments on controlled generation and summarization tasks show that BoN is the most effective alignment method, and our variational approximation to BoN achieves the closest performance to BoN and surpasses models fine-tuned using the standard KL-constrained RL objective. In the controlled generation task, vBoN appears more frequently on the Pareto frontier of reward and KL divergence compared to other alignment methods. In the summarization task, vBoN achieves high reward values across various sampling temperatures.

Key facts. Cited by 49 papers in the map (data through 7 Oct 2026). Placed in LLM preference optimization methods, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 21 of its 34 references that are in the map. First posted on arXiv 8 Jul 2024.

Builds on

21 of 34 references.

Closest ancestors · 5

Same direction, newest first

Other references · 16

2023GPT-4 Technical Reportcited by 18,132
2017Proximal Policy Optimization Algorithmscited by 14,441
2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
2017Deep reinforcement learning from human preferencescited by 3,846
2020Learning to summarize from human feedbackcited by 2,562
2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,722
2022Scaling Laws for Reward Model Overoptimizationcited by 1,056
2017Deep Reinforcement Learning that Matterscited by 727
2023RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignmentcited by 672
2018Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Reviewcited by 466
2021FUDGE: Controlled Text Generation With Future Discriminatorscited by 374

4 more

2023Controlled Decoding from Language Modelscited by 133
2024Theoretical guarantees on the best-of-n alignment policycited by 116
2024The N+ Implementation Details of RLHF with PPO: A Case Study on TL;DR Summarizationcited by 49
2021Return-based Scaling: Yet Another Normalisation Trick for Deep RLcited by 22

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 12; 2025: 25; 2026: 12.

49 papers.

Sort:

By area:

2024LLaVA-CoT: Let Vision Language Models Reason Step-by-Stepcited by 446
2024Fast Best-of-N Decoding via Speculative Rejectioncited by 112
2024Self-Improvement in Language Models: The Sharpening Mechanismcited by 102
2025Is Best-of-N the Best of Them? Coverage, Scaling, and Optimality in Inference-Time Alignmentcited by 91
2025Pass@K Policy Optimization: Solving Harder Reinforcement Learning Problemscited by 83
2024BOND: Aligning LLMs with Best-of-N Distillationcited by 73
2024Systematic Evaluation of LLM-as-a-Judge in LLM Alignment Tasks: Explainable Metrics and Diverse Prompt Templatescited by 71
2025Revisiting the Test-Time Scaling of o1-like Models: Do they Truly Possess Test-Time Scaling Capabilities?cited by 69
2025Sampling-Efficient Test-Time Scaling: Self-Estimating the Best-of-N Sampling in Early Decodingcited by 62
2024TreeBoN: Enhancing Inference-Time Alignment with Speculative Tree-Search and Best-of-N Samplingcited by 61
2024RMB: Comprehensively Benchmarking Reward Models in LLM Alignmentcited by 60
2025Does Thinking More always Help? Mirage of Test-Time Scaling in Reasoning Modelscited by 52

37 more

2024Scaling of Search and Learning: A Roadmap to Reproduce o1 from Reinforcement Learning Perspectivecited by 49
2025Optimizing Language Models for Inference Time Objectives using Reinforcement Learningcited by 49
2025Syntactic and Semantic Control of Large Language Models via Sequential Monte Carlocited by 45
2025PonderLM: Pretraining Language Models to Ponder in Continuous Spacecited by 44
2025Don't Overthink It: A Survey of Efficient R1-style Large Reasoning Modelscited by 36
2024InfAlign: Inference-aware language model alignmentcited by 34
2025Inference-Time Reward Hacking in Large Language Modelscited by 32
2025Soft Best-of-n Sampling for Model Alignmentcited by 28
2024Policy Filtration for RLHF to Mitigate Noise in Reward Modelscited by 21
2025Learning from Peers in Reasoning Modelscited by 16
2025The Best of N Worlds: Aligning Reinforcement Learning with Best-of-N Sampling via max@k Optimisationcited by 16
2025PonderLM-2: Pretraining LLM with Latent Thoughts in Continuous Spacecited by 14
2025Risk-Sensitive RL for Alleviating Exploration Dilemmas in Large Language Modelscited by 12
2024Faster WIND: Accelerating Iterative Best-of-N Distillation for LLM Alignmentcited by 9
2025When Life Gives You Samples: The Benefits of Scaling up Inference Compute for Multilingual LLMscited by 9
2025PiCSAR: Probabilistic Confidence Selection And Ranking for Reasoning Chainscited by 9
2025Theoretical Modeling of Large Language Model Self-Improvement Training Dynamics Through Solver-Verifier Gapcited by 6
2025Quantile Reward Policy Optimization: Alignment with Pointwise Regression and Exact Partition Functionscited by 6
2026LaViDa-R1: Advancing Reasoning for Unified Multimodal Diffusion Language Modelscited by 6
2026Why Pass@k Optimization Can Degrade Pass@1: Prompt Interference in LLM Post-trainingcited by 6
2025Semantic Probabilistic Control of Language Modelscited by 4
2025Mitigating Premature Exploitation in Particle-based Monte Carlo for Inference-Time Scalingcited by 4
2024LIAR: Leveraging Inference Time Alignment (Best-of-N) to Jailbreak LLMs in Secondscited by 3
2025On the Role of Feedback in Test-Time Scaling of Agentic AI Workflowscited by 3
2025Fine-Tuning Diffusion Models via Intermediate Distribution Shapingcited by 3
2026Safety Recovery in Reasoning Models Is Only a Few Early Steering Steps Awaycited by 3
2026GIFT: Bootstrapping Image-to-CAD Program Synthesis via Geometric Feedbackcited by 3
2026Power Distribution Bridges Sampling, Self-Reward RL, and Self-Distillationcited by 3
2025Decoding Memories: An Efficient Pipeline for Self-Consistency Hallucination Detectioncited by 2
2026Best-of-Tails: Bridging Optimism and Pessimism in Inference-Time Alignmentcited by 2
2026From Curiosity to Caution: Mitigating Reward Hacking for Best-of-N with Pessimismcited by 2
2025From Perception to Reasoning: Deep Thinking Empowers Multimodal Large Language Modelscited by 1
2026Reward Learning from Best-of-N Preference Data: Targets, Tradeoffs, and Design Principlescited by 0
2026Don't Let Gains FADE: Breaking Down Policy Gradient Weights in RLcited by 0
2026Best-of-Better-N: Generating Pre-Aligned Responses with In-Context Learningcited by 0
2026Rank-Conditioned Sample Reuse for the Plackett--Luce Best-of-K Objectivecited by 0
2026The Asymptotics of Language Model Alignment with Memorycited by 0

Similar, not cited

Close in wording, no citation either way