Rank-Conditioned Sample Reuse for the Plackett--Luce Best-of-K Objective
Not on the map. Why?
Abstract (arXiv)
We study the coupled objective J_K^WOR = E_{S ~ PL-WOR_K}[max_{i in S} R_i]: the expected maximum reward of a size-K Plackett-Luce draw without replacement, the law of Gumbel-Top-K / Stochastic Beam Search decoding. This estimand differs from the conventional i.i.d. objective J_K^iid = E[max_{i<=K} R_i] targeted by existing sample-reuse Max@K estimators, and reusing their i.i.d. weights under the coupled sampler is provably biased (a closed-form three-item instance gives E[g_iid] = (4/5) grad J_K^WOR exactly; pass@K under the coupled sampler is the binary-reward special case). Generic joint-score REINFORCE is already unbiased for J_K^WOR; what it lacks is sample reuse. Our contribution is to instantiate standard rank-conditioned Horvitz-Thompson estimation for the J_K^WOR subset total: from one Gumbel-Top-n pool (n>K) and its observed priority threshold we build an estimator that reuses all C(n,K) embedded K-subsets, unbiased with an unbiased exact score-function surrogate gradient, plus a reward-sorted Max-specific dynamic program that collapses the C(n,K)-term subset sum (with K!-cost set probabilities) exactly to a one-dimensional integral. A fixed-Q quadrature evaluation costs O(n log n + nKQ) arithmetic and is numerically, not algebraically, exact; no epsilon-approximation rate is certified. Each nonzero degree-K Horvitz-Thompson term has finite second moment exactly when n >= 2K; under the same assumptions the full surrogate gradient has finite second moment whenever n >= 2K (sharpness there is open). At K=1 the construction recovers classical priority sampling. All quantities require only the values and differentiable computation graphs of the n+1 drawn items' probabilities, so finite structured sequence policies sampled by exact SBS are covered. A certified finite-Q quadrature bound and countably infinite support remain open. Validation code is included as ancillary files.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 23 of its 32 references that are in the map. First posted on arXiv 13 Jul 2026.
Builds on
23 of 32 references.
References · 23
- 2025Understanding R1-Zero-Like Training: A Critical Perspectivecited by 1,345
- 2020POMO: Policy Optimization with Multiple Optima for Reinforcement Learningcited by 348
- 2019Deep symbolic regression: Recovering mathematical expressions from data via risk-seeking policy gradientscited by 309
- 2019Stochastic Beams and Where to Find Them: The Gumbel-Top-k Trick for Sampling Sequences Without Replacementcited by 211
- 2025Pass@K Policy Optimization: Solving Harder Reinforcement Learning Problemscited by 87
- 2024BOND: Aligning LLMs with Best-of-N Distillationcited by 76
- 2019Reparameterizable Subset Sampling via Continuous Relaxationscited by 70
- 2020Gradient Estimation with Stochastic Softmax Trickscited by 67
- 2022Winner Takes It All: Training Performant RL Populations for Combinatorial Optimizationcited by 58
- 2024Variational Best-of-N Alignmentcited by 51
- 2021Computationally Efficient Optimization of Plackett-Luce Ranking Models for Relevance and Fairnesscited by 36
- 2024PolyNet: Learning Diverse Solution Strategies for Neural Combinatorial Optimizationcited by 35
11 more
- 2024Self-Improvement for Neural Combinatorial Optimization: Sample without Replacement, but Improvementcited by 34
- 2020Estimating Gradients for Discrete Random Variables by Sampling without Replacementcited by 31
- 2020Incremental Sampling Without Replacement for Sequence Modelscited by 23
- 2025The Best of N Worlds: Aligning Reinforcement Learning with Best-of-N Sampling via max@k Optimisationcited by 16
- 2019Low-variance Black-box Gradient Estimates for the Plackett-Luce Distributioncited by 11
- 2025Advantage Shaping as Surrogate Reward Maximization: Unifying Pass@K Policy Gradientscited by 11
- 2017On Sampling from Massive Graph Streamscited by 5
- 2024Leader Reward for POMO-Based Neural Combinatorial Optimizationcited by 5
- 2025RSPO: Risk-Seeking Policy Optimization for Pass@k and Max@k Metrics in Large Language Modelscited by 5
- 2026QuasiMoTTo: Quasi-Monte Carlo Test-Time Scalingcited by 4
- 2026Cast a Wider Net: Coordinated Pass@K Policy Optimization for Code Reasoningcited by 2
Built on by
Not cited yet.