mlsanityData to 14 Oct 2026

The Best of N Worlds: Aligning Reinforcement Learning with Best-of-N Sampling via max@k Optimisation

arXiv 2510.23393 ↗First posted 27 Oct 2025cs.LGcited by 16

Not on the map. Why?

Abstract (arXiv)

The application of Reinforcement Learning with Verifiable Rewards (RLVR) to mathematical and coding domains has demonstrated significant improvements in the reasoning and problem-solving abilities of Large Language Models. Despite its success in single generation problem solving, the reinforcement learning fine-tuning process may harm the model's exploration ability, as reflected in decreased diversity of generations and a resulting degradation of performance during Best-of-N sampling for large N values. In this work, we focus on optimizing the max@k metric, a continuous generalization of pass@k. We derive an unbiased on-policy gradient estimate for direct optimization of this metric. Furthermore, we extend our derivations to the off-policy updates, a common element in modern RLVR algorithms, that allows better sample efficiency. Empirically, we show that our objective effectively optimizes max@k metric in off-policy scenarios, aligning the model with the Best-of-N inference strategy.

Key facts. Cited by 16 papers in the map (data through 14 Oct 2026). Builds on 38 of its 48 references that are in the map. First posted on arXiv 27 Oct 2025.

Builds on

38 of 48 references.

References · 38

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2017Proximal Policy Optimization Algorithmscited by 15,215
  3. 2024The Llama 3 Herd of Modelscited by 14,715
  4. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  5. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  6. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  7. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  8. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,330
  9. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  10. 2017Deep reinforcement learning from human preferencescited by 3,912
  11. 2021Program Synthesis with Large Language Modelscited by 3,637
  12. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scalecited by 2,810
26 more
  1. 2020Learning to summarize from human feedbackcited by 2,588
  2. 2024LiveCodeBench: Holistic and Contamination Free Evaluation of Large Language Models for Codecited by 2,004
  3. 2025Gemma 3 Technical Reportcited by 1,824
  4. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,745
  5. 2022Competition-Level Code Generation with AlphaCodecited by 1,462
  6. 2024Qwen2.5-Coder Technical Reportcited by 1,203
  7. 2022Scaling Laws for Reward Model Overoptimizationcited by 1,095
  8. 2024Tulu 3: Pushing Frontiers in Open Language Model Post-Trainingcited by 1,053
  9. 2024Large Language Monkeys: Scaling Inference Compute with Repeated Samplingcited by 991
  10. 2025Does Reinforcement Learning Really Incentivize Reasoning Capacity in LLMs Beyond the Base Model?cited by 982
  11. 2023A General Theoretical Paradigm to Understand Learning from Human Preferencescited by 912
  12. 2025The Entropy Mechanism of Reinforcement Learning for Reasoning Language Modelscited by 485
  13. 2025The Surprising Effectiveness of Negative Reinforcement in LLM Reasoningcited by 162
  14. 2025Pass@k Training for Adaptively Balancing Exploration and Exploitation of Large Reasoning Modelscited by 138
  15. 2023Controlled Decoding from Language Modelscited by 135
  16. 2024BoNBoN Alignment for Large Language Models and the Sweetness of Best-of-n Samplingcited by 121
  17. 2024Inference-Aware Fine-Tuning for Best-of-N Sampling in Large Language Modelscited by 89
  18. 2025Pass@K Policy Optimization: Solving Harder Reinforcement Learning Problemscited by 87
  19. 2024BOND: Aligning LLMs with Best-of-N Distillationcited by 76
  20. 2025What Makes a Reward Model a Good Teacher? An Optimization Perspectivecited by 73
  21. 2025Co-Evolving LLM Coder and Unit Tester via Reinforcement Learningcited by 56
  22. 2025Beyond Pass@1: Self-Play with Variational Problem Synthesis Sustains RLVRcited by 54
  23. 2025Optimizing Language Models for Inference Time Objectives using Reinforcement Learningcited by 53
  24. 2024Variational Best-of-N Alignmentcited by 51
  25. 2024Asymptotics of Language Model Alignmentcited by 47
  26. 2025Training Long-Context, Multi-Turn Software Engineering Agents with Reinforcement Learningcited by 36

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 16.

16 papers.

Sort:

By area:

  1. 2026Vector Policy Optimization: Training for Diversity Improves Test-Time Searchcited by 8
  2. 2026Frontis-MA1: Training an AI4AI Model towards Recursive Self-Improvement in Machine Learning Engineeringcited by 8
  3. 2026Emergence of Exploration in Policy Gradient Reinforcement Learning via Retryingcited by 5
  4. 2026Regret Analysis of Retry-Based Banditscited by 4
  5. 2026HTAM: Hierarchical Transition-Attended Memory for Operator Optimizationcited by 2
  6. 2026On Advantage Estimates for Max@K Policy Gradientscited by 2
  7. 2026OrderGrad: Optimizing Beyond the Mean with Order-Statistic Policy Gradient Estimationcited by 2
  8. 2026Multi-Axis Max@K Reinforcement Learning for Representative Diversity in Text-to-Image Generationcited by 2
  9. 2026Inference-Aware Meta-Alignment of LLMs via Non-Linear GRPOcited by 1
  10. 2026Boosting LLM Reasoning via Human-Inspired Reward Shapingcited by 1
  11. 2026What should post-training optimize? A test-time scaling law perspectivecited by 0
  12. 2026Rank-Conditioned Sample Reuse for the Plackett--Luce Best-of-K Objectivecited by 0
4 more
  1. 2026Theoretical Foundations of max@k Reinforcement Learningcited by 0
  2. 2026Re3Cap: Retrieval-Guided Refinement for Image Captioning Enhancement via Reinforcement Learningcited by 0
  3. 2026The Statistical Benefits of Multiple Responses for Learning from Demonstrationscited by 0
  4. 2026SERA: Scale-Equalized Rollout Allocation for Maximum Likelihood Reinforcement Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025On the optimization dynamics of RLVR: Gradient gap and step size thresholds
  2. 2025Variance-Aware Baselines and Adaptive Learning Rates for Reinforcement Learning with Verifiable Rewards
  3. 2026Multi-Step Likelihood-Ratio Correction for Reinforcement Learning with Verifiable Rewards
  4. 2026Enabling Token-wise Importance Sampling in Off-Policy RLVR with Proximal KL Regularization
  5. 2026Revisiting Reinforcement Learning with Verifiable Rewards from a Contrastive Perspective
  6. 2026Non-vacuous Generalization Bounds for Reinforcement Learning with Verifiable Rewards
  7. 2025Unlocking Reasoning Capabilities in LLMs via Reinforcement Learning Exploration
  8. 2026How to Allocate, How to Learn? Dynamic Rollout Allocation and Advantage Modulation for Policy Optimization