mlsanityData to 14 Oct 2026

Learning to Choose or Choosing to Learn: Best-of-N vs. Supervised Fine-Tuning for Bit String Generation

arXiv 2505.17288 ↗First posted 22 May 2025stat.ML cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Using the bit string generation problem as a case study, we theoretically compare two standard methods for adapting large language models to new tasks. The first, referred to as supervised fine-tuning, involves training a new next token predictor on good generations. The second method, Best-of-N, trains a reward model to select good responses from a collection generated by an unaltered base model. If the learning setting is realizable, we find that supervised fine-tuning outperforms BoN through a better dependence on the response length in its rate of convergence. If realizability fails, then depending on the failure mode, BoN can enjoy a better rate of convergence in either n or a rate of convergence with better dependence on the response length.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 47 of its 81 references that are in the map. First posted on arXiv 22 May 2025.

Builds on

47 of 81 references.

References · 47

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2017Proximal Policy Optimization Algorithmscited by 15,215
  3. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  4. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  5. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  6. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  7. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  8. 2022Large Language Models are Zero-Shot Reasonerscited by 4,374
  9. 2021Finetuned Language Models Are Zero-Shot Learnerscited by 3,780
  10. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Modelscited by 3,211
  11. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,879
  12. 2020Learning to summarize from human feedbackcited by 2,588
35 more
  1. 2015Trust Region Policy Optimizationcited by 2,580
  2. 2024Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameterscited by 1,986
  3. 2021Multitask Prompted Training Enables Zero-Shot Task Generalizationcited by 1,470
  4. 2021A General Language Assistant as a Laboratory for Alignmentcited by 1,055
  5. 2016The LAMBADA dataset: Word prediction requiring a broad discourse contextcited by 992
  6. 2024Large Language Monkeys: Scaling Inference Compute with Repeated Samplingcited by 991
  7. 2021P-Tuning v2: Prompt Tuning Can Be Comparable to Fine-tuning Universally Across Scales and Taskscited by 689
  8. 2023An Empirical Study of Catastrophic Forgetting in Large Language Models During Continual Fine-tuningcited by 639
  9. 2022Improving alignment of dialogue agents via targeted human judgementscited by 531
  10. 2024ReST-MCTS*: LLM Self-Training via Process Reward Guided Tree Searchcited by 344
  11. 2023A Survey on Model Compression for Large Language Modelscited by 269
  12. 2023Sentiment Analysis in the Era of Large Language Models: A Reality Checkcited by 268
  13. 2023Self-Evaluation Guided Beam Search for Reasoningcited by 196
  14. 2024The pitfalls of next-token predictioncited by 160
  15. 2024Stream of Search (SoS): Learning to Search in Languagecited by 145
  16. 2024Scaling Laws for Reward Model Overoptimization in Direct Alignment Algorithmscited by 143
  17. 2024From Decoding to Meta-Generation: Inference-time Algorithms for Large Language Modelscited by 137
  18. 2025LLM Post-Training: A Deep Dive into Reasoning Large Language Modelscited by 135
  19. 2024Toward Self-Improvement of LLMs via Imagination, Searching, and Criticizingcited by 125
  20. 2024Theoretical guarantees on the best-of-n alignment policycited by 124
  21. 2024BoNBoN Alignment for Large Language Models and the Sweetness of Best-of-n Samplingcited by 121
  22. 2024Fast Best-of-N Decoding via Speculative Rejectioncited by 112
  23. 2025Towards System 2 Reasoning in LLMs: Learning How to Think With Meta Chain-of-Thoughtcited by 109
  24. 2024Self-Improvement in Language Models: The Sharpening Mechanismcited by 104
  25. 2025Is Best-of-N the Best of Them? Coverage, Scaling, and Optimality in Inference-Time Alignmentcited by 97
  26. 2024Is Behavior Cloning All You Need? Understanding Horizon in Imitation Learningcited by 74
  27. 2023Auto-Regressive Next-Token Predictors are Universal Learnerscited by 67
  28. 2023Large Language Models Are Not Strong Abstract Reasonerscited by 53
  29. 2024Asymptotics of Language Model Alignmentcited by 47
  30. 2025Is a Good Foundation Necessary for Efficient Reinforcement Learning? The Computational Role of the Base Model in Explorationcited by 36
  31. 2024AI Hallucinations: A Misnomer Worth Clarifyingcited by 34
  32. 2024On the Impact of Fine-Tuning on Chain-of-Thought Reasoningcited by 32
  33. 2025A Theory of Learning with Autoregressive Chain of Thoughtcited by 23
  34. 2023Insights into Classifying and Mitigating LLMs' Hallucinationscited by 6
  35. 2025Tight Bounds on the Binomial CDF, and the Minimum of i.i.d Binomials, in terms of KL-Divergencecited by 3

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Efficient Response Generation Strategy Selection for Fine-Tuning Large Language Models Through Self-Aligned Perplexity
  2. 2025Enhancing Large Language Model Reasoning via Selective Critical Token Fine-Tuning
  3. 2025Discriminative Finetuning of Generative Large Language Models without Reward Models and Human Preference Data
  4. 2024Scalable Fine-tuning from Multiple Data Sources: A First-Order Approximation Approach
  5. 2026Goal-Conditioned Supervised Learning for LLM Fine-Tuning
  6. 2024Inference-Aware Fine-Tuning for Best-of-N Sampling in Large Language Models
  7. 2025Anchored Supervised Fine-Tuning
  8. 2026Supervised Fine-Tuning versus Reinforcement Learning: A Study of Post-Training Methods for Large Language Models