mlsanityData to 14 Oct 2026

Inference-Aware Fine-Tuning for Best-of-N Sampling in Large Language Models

arXiv 2412.15287 ↗First posted 18 Dec 2024cs.CL cs.AI cs.LGcited by 89

On the map: LLM reasoning → Test-time compute and efficient thinking

Abstract (arXiv)

Recent studies have indicated that effectively utilizing inference-time compute is crucial for attaining better performance from large language models (LLMs). In this work, we propose a novel inference-aware fine-tuning paradigm, in which the model is fine-tuned in a manner that directly optimizes the performance of the inference-time strategy. We study this paradigm using the simple yet effective Best-of-N (BoN) inference strategy, in which a verifier selects the best out of a set of LLM-generated responses. We devise the first imitation learning and reinforcement learning~(RL) methods for BoN-aware fine-tuning, overcoming the challenging, non-differentiable argmax operator within BoN. We empirically demonstrate that our BoN-aware models implicitly learn a meta-strategy that interleaves best responses with more diverse responses that might be better suited to a test-time input -- a process reminiscent of the exploration-exploitation trade-off in RL. Our experiments demonstrate the effectiveness of BoN-aware fine-tuning in terms of improved performance and inference-time compute. In particular, we show that our methods improve the Bo32 performance of Gemma 2B on Hendrycks MATH from 26.8% to 30.8%, and pass@32 from 60.0% to 67.0%, as well as the pass@16 on HumanEval from 61.6% to 67.1%.

Key facts. Cited by 89 papers in the map (data through 14 Oct 2026). Placed in Test-time compute and efficient thinking, one of 17 directions in LLM reasoning. Builds on 31 of its 49 references that are in the map. First posted on arXiv 18 Dec 2024.

Builds on

31 of 49 references.

Closest ancestors · 6

Same direction, newest first

  1. 2024Training Language Models to Self-Correct via Reinforcement Learning19 Sep
  2. 2024Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters6 Aug
  3. 2024Stream of Search (SoS): Learning to Search in Language1 Apr
  4. 2024V-STaR: Training Verifiers for Self-Taught Reasoners9 Feb
  5. 2024Theoretical guarantees on the best-of-n alignment policy3 Jan
  6. 2022STaR: Bootstrapping Reasoning With Reasoning28 Mar

Other references · 25

  1. 2017Proximal Policy Optimization Algorithmscited by 15,215
  2. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  3. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  4. 2021Evaluating Large Language Models Trained on Codecited by 8,320
  5. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  6. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  7. 2021Program Synthesis with Large Language Modelscited by 3,637
  8. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,224
  9. 2020Learning to summarize from human feedbackcited by 2,588
  10. 2015High-Dimensional Continuous Control Using Generalized Advantage Estimationcited by 2,026
  11. 2024Large Language Monkeys: Scaling Inference Compute with Repeated Samplingcited by 991
  12. 2023A General Theoretical Paradigm to Understand Learning from Human Preferencescited by 912
13 more
  1. 2024Generative Verifiers: Reward Modeling as Next-Token Predictioncited by 480
  2. 2023Beyond Human Data: Scaling Self-Training for Problem-Solving with Language Modelscited by 349
  3. 2024Preference Fine-Tuning of LLMs Should Leverage Suboptimal, On-Policy Datacited by 185
  4. 2024From Decoding to Meta-Generation: Inference-time Algorithms for Large Language Modelscited by 137
  5. 2024BoNBoN Alignment for Large Language Models and the Sweetness of Best-of-n Samplingcited by 121
  6. 2019Differentiable Ranks and Sorting using Optimal Transportcited by 113
  7. 2024Beyond A*: Better Planning with Transformers via Search Dynamics Bootstrappingcited by 90
  8. 2024Functional Benchmarks for Robust Evaluation of Reasoning Performance, and the Reasoning Gapcited by 79
  9. 2024BOND: Aligning LLMs with Best-of-N Distillationcited by 76
  10. 2024Value-Incentivized Preference Optimization: A Unified Approach to Online and Offline RLHFcited by 73
  11. 2024MathOdyssey: Benchmarking Mathematical Problem-Solving Skills in Large Language Models Using Odyssey Math Datacited by 52
  12. 2015Provable Bayesian Inference via Particle Mirror Descentcited by 23
  13. 2021Improve Agents without Retraining: Parallel Tree Search with Off-Policy Correctioncited by 10

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 47; 2026: 41.

89 papers.

Sort:

By area:

  1. 2025ReasoningBank: Scaling Agent Self-Evolving with Reasoning Memorycited by 284
  2. 2025RewardBench 2: Advancing Reward Model Evaluationcited by 150
  3. 2025Optimizing Test-Time Compute via Meta Reinforcement Fine-Tuningcited by 123
  4. 2025A Survey of Frontiers in LLM Reasoning: Inference Scaling, Learning to Reason, and Agentic Systemscited by 111
  5. 2025Is Best-of-N the Best of Them? Coverage, Scaling, and Optimality in Inference-Time Alignmentcited by 97
  6. 2025Pass@K Policy Optimization: Solving Harder Reinforcement Learning Problemscited by 87
  7. 2025Scaling Test-Time Compute Without Verification or RL is Suboptimalcited by 84
  8. 2025Rewarding the Unlikely: Lifting GRPO Beyond Distribution Sharpeningcited by 80
  9. 2025AgentGym-RL: Training LLM Agents for Long-Horizon Decision Making through Multi-Turn Reinforcement Learningcited by 72
  10. 2025Tapered Off-Policy REINFORCE: Stable and efficient reinforcement learning for LLMscited by 61
  11. 2025Autellix: An Efficient Serving Engine for LLM Agents as General Programscited by 58
  12. 2025Optimizing Language Models for Inference Time Objectives using Reinforcement Learningcited by 53
38 more
  1. 2025Exploring the Limit of Outcome Reward for Learning Mathematical Reasoningcited by 52
  2. 2025Sample, Scrutinize and Scale: Effective Inference-Time Search by Scaling Verificationcited by 50
  3. 2026POPE: Learning to Reason on Hard Problems via Privileged On-Policy Explorationcited by 42
  4. 2025Weight Ensembling Improves Reasoning in Language Modelscited by 38
  5. 2025Self-Improving Language Models for Evolutionary Program Synthesis: A Case Study on ARC-AGIcited by 36
  6. 2024InfAlign: Inference-aware language model alignmentcited by 35
  7. 2025Thinking vs. Doing: Agents that Reason by Scaling Test-Time Interactioncited by 34
  8. 2025Inference-Time Reward Hacking in Large Language Modelscited by 34
  9. 2025Reasoning on a Budget: A Survey of Adaptive and Controllable Test-Time Compute in LLMscited by 31
  10. 2026Scalable Power Sampling: Unlocking Efficient, Training-Free Reasoning for LLMs via Distribution Sharpeningcited by 31
  11. 2025Roll the dice & look before you leap: Going beyond the creative limits of next-token predictioncited by 29
  12. 2025Reinforce LLM Reasoning through Multi-Agent Reflectioncited by 27
  13. 2025AI-Slop to AI-Polish? Aligning Language Models through Edit-Based Writing Rewards and Test-time Computationcited by 25
  14. 2025Tina: Tiny Reasoning Models via LoRAcited by 25
  15. 2025Differential Smoothing Mitigates Sharpening and Improves LLM Reasoningcited by 22
  16. 2025SofT-GRPO: Surpassing Discrete-Token LLM Reinforcement Learning via Gumbel-Reparameterized Soft-Thinking Policy Optimizationcited by 21
  17. 2025Representation-Based Exploration for Language Models: From Test-Time to Post-Trainingcited by 16
  18. 2025The Best of N Worlds: Aligning Reinforcement Learning with Best-of-N Sampling via max@k Optimisationcited by 16
  19. 2026Reuse your FLOPs: Scaling RL on Hard Problems by Conditioning on Very Off-Policy Prefixescited by 16
  20. 2025Generative AI Act II: Test Time Scaling Drives Cognition Engineeringcited by 15
  21. 2025When Does Verification Pay Off? A Closer Look at LLMs as Solution Verifierscited by 15
  22. 2025Risk-Sensitive RL for Alleviating Exploration Dilemmas in Large Language Modelscited by 14
  23. 2025Entropy-guided sequence weighting for efficient exploration in RL-based LLM fine-tuningcited by 13
  24. 2025Examining False Positives under Inference Scaling for Mathematical Reasoningcited by 11
  25. 2025Unveiling Over-Memorization in Finetuning LLMs for Reasoning Taskscited by 11
  26. 2025Token Hidden Reward: Steering Exploration-Exploitation in Group Relative Deep Reinforcement Learningcited by 10
  27. 2025Reasoning Path Divergence: A New Metric and Curation Strategy to Unlock LLM Diverse Thinkingcited by 10
  28. 2026On-Policy Self-Distillation with Sampled Demonstrations Reduces Output Diversitycited by 10
  29. 2025First Finish Search: Efficient Test-Time Scaling in Large Language Modelscited by 9
  30. 2025Revisiting Test-Time Scaling: A Survey and a Diversity-Aware Method for Efficient Reasoningcited by 9
  31. 2026Why Pass@k Optimization Can Degrade Pass@1: Prompt Interference in LLM Post-trainingcited by 8
  32. 2026Vector Policy Optimization: Training for Diversity Improves Test-Time Searchcited by 8
  33. 2025Distilled Pretraining: A modern lens of Data, In-Context Learning and Test-Time Scalingcited by 7
  34. 2025Polychromic Objectives for Reinforcement Learningcited by 7
  35. 2025What Can You Do When You Have Zero Rewards During RL?cited by 7
  36. 2025Self-Reasoning Language Models: Unfold Hidden Reasoning Chains with Few Reasoning Catalystcited by 6
  37. 2025Why is Your Language Model a Poor Implicit Reward Model?cited by 6
  38. 2025Slm-mux: Orchestrating small language models for reasoningcited by 6

50 most cited of 89 All 89 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026Reinforcement Inference: Leveraging Uncertainty for Self-Correcting Language Model Reasoning
  2. 2026Revisiting the (Sub)Optimality of Best-of-N for Inference-Time Alignment
  3. 2026Compute Aligned Training: Optimizing for Test Time Inference
  4. 2026Supervised Fine-Tuning versus Reinforcement Learning: A Study of Post-Training Methods for Large Language Models
  5. 2026Reference-Free Reinforcement Learning Fine-Tuning for MT: A Seq2Seq Perspective
  6. 2025Enhancing Large Language Model Reasoning via Selective Critical Token Fine-Tuning
  7. 2025T1: Advancing Language Model Reasoning through Reinforcement Learning and Inference Scaling
  8. 2025When Life Gives You Samples: The Benefits of Scaling up Inference Compute for Multilingual LLMs