mlsanityData to 14 Oct 2026

Truncate Bad, Upweight Good: BoN-Style Distillation via Rank-Based Classification

arXiv 2608.19748 ↗First posted 20 Aug 2026cs.LG cs.AI cs.CLcited by 0

On the map: LLM alignment / RLHF / preference optimisation → Process reward models for reasoning

Abstract (arXiv)

Inference-time selection methods, such as Best-of-N, improve generation by sampling a pool of candidates and selecting the top-ranked completion according to a reward model. Distillation seeks to amortize this procedure into a single policy by replacing raw rewards with in-pool ranks and learning a policy that upweights higher-ranked completions. However, existing rank-based policies typically use smooth full-support reweighting, so low-ranked completions receive less mass but remain in the target support. Although a sharper reweighting reduces lower-tail mass, it also increases reliance on brittle ranking at the top made by a single reward model. We propose TUP: a Truncate-bad, Upweight-good Policy that removes low-ranked completions from the support and reweights only the retained upper tail with a tunable sharpness. TUP admits a closed-form, prompt-independent normalization and can be trained fully offline via binary cross-entropy, using shifted-truncated win-rates as soft labels and distilled-to-reference log-likelihood ratios as logits. Theoretically, under certain assumptions, we show that for any unknown oracle reward, the best monotone rank-reweighting can be matched by a lower-tail truncation rule, providing formal support for removing the lower tail rather than merely downweighting it. Empirically, we show that TUP is competitive with strong offline alignment baselines.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Process reward models for reasoning, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 42 of its 45 references that are in the map. First posted on arXiv 20 Aug 2026.

Builds on

42 of 45 references.

References · 42

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2017Proximal Policy Optimization Algorithmscited by 15,215
  3. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  4. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  5. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  6. 2024GPT-4o System Cardcited by 4,588
  7. 2017Deep reinforcement learning from human preferencescited by 3,912
  8. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,720
  9. 2020Learning to summarize from human feedbackcited by 2,588
  10. 2019Fine-Tuning Language Models from Human Preferencescited by 2,150
  11. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,745
  12. 2022Scaling Laws for Reward Model Overoptimizationcited by 1,095
30 more
  1. 2024Tulu 3: Pushing Frontiers in Open Language Model Post-Trainingcited by 1,053
  2. 2024Large Language Monkeys: Scaling Inference Compute with Repeated Samplingcited by 991
  3. 2024SimPO: Simple Preference Optimization with a Reference-Free Rewardcited by 963
  4. 2024Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluatorscited by 897
  5. 2023RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignmentcited by 686
  6. 2024RewardBench: Evaluating Reward Models for Language Modelingcited by 499
  7. 2023RRHF: Rank Responses to Align Language Models with Human Feedback without tearscited by 463
  8. 2023SLiC-HF: Sequence Likelihood Calibration with Human Feedbackcited by 401
  9. 2024Interpretable Preferences via Multi-Objective Reward Modeling and Mixture-of-Expertscited by 376
  10. 2024Magpie: Alignment Data Synthesis from Scratch by Prompting Aligned LLMs with Nothingcited by 346
  11. 2025Skywork-Reward-V2: Scaling Preference Data Curation via Human-AI Synergycited by 214
  12. 2024Theoretical guarantees on the best-of-n alignment policycited by 124
  13. 2024BoNBoN Alignment for Large Language Models and the Sweetness of Best-of-n Samplingcited by 121
  14. 2024The Good, The Bad, and The Greedy: Evaluation of LLMs Should Not Ignore Non-Determinismcited by 116
  15. 2024Unpacking DPO and PPO: Disentangling Best Practices for Learning from Preference Feedbackcited by 107
  16. 2024Critique-out-Loud Reward Modelscited by 106
  17. 2025Is Best-of-N the Best of Them? Coverage, Scaling, and Optimality in Inference-Time Alignmentcited by 97
  18. 2024How to Evaluate Reward Models for RLHFcited by 88
  19. 2024BOND: Aligning LLMs with Best-of-N Distillationcited by 76
  20. 2024REBEL: Reinforcement Learning via Regressing Relative Rewardscited by 69
  21. 2022Causal Confusion and Reward Misidentification in Preference-Based Reward Learningcited by 53
  22. 2025Chasing the Tail: Effective Rubric-based Reward Modeling for Large Language Model Post-Trainingcited by 53
  23. 2020Understanding Learned Reward Functionscited by 37
  24. 2024InfAlign: Inference-aware language model alignmentcited by 35
  25. 2025Inference-Time Reward Hacking in Large Language Modelscited by 34
  26. 2025IPO: Your Language Model is Secretly a Preference Classifiercited by 17
  27. 2025Rethinking Reward Model Evaluation Through the Lens of Reward Overoptimizationcited by 11
  28. 2024Faster WIND: Accelerating Iterative Best-of-N Distillation for LLM Alignmentcited by 9
  29. 2026When LLM Judge Scores Look Good but Best-of-N Decisions Failcited by 9
  30. 2025Quantile Reward Policy Optimization: Alignment with Pointwise Regression and Exact Partition Functionscited by 6

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Best-of-N through the Smoothing Lens: KL Divergence and Regret Analysis
  2. 2026Scaling Reasoning Efficiently via Relaxed On-Policy Distillation
  3. 2026ResOPD: Tail Residualization for Sparse On-Policy Distillation
  4. 2026ReTaCo: Residual-Target Control for On-Policy Distillation
  5. 2026Token Utility for On-Policy Distillation via Local Trust-Region Optimization
  6. 2026On-Policy Distillation Meets Off-Policy GRPO: Training Compact Instruction-Following Rerankers
  7. 2026Tail-Aware Top-k On-Policy Distillation
  8. 2026Reward-Aligned Distillation