mlsanityData to 7 Oct 2026

TreeBoN: Enhancing Inference-Time Alignment with Speculative Tree-Search and Best-of-N Sampling

arXiv 2410.16033 ↗First posted 18 Oct 2024cs.CL cs.AI cs.LGcited by 61

On the map: LLM reasoning → Test-time compute and efficient thinking

also in 1 more
Abstract (arXiv)

Inference-time alignment enhances the performance of large language models without requiring additional training or fine-tuning but presents challenges due to balancing computational efficiency with high-quality output. Best-of-N (BoN) sampling, as a simple yet powerful approach, generates multiple responses and selects the best one, achieving improved performance but with a high computational cost. We propose TreeBoN, a novel framework that integrates a speculative tree-search strategy into Best-of-N (BoN) Sampling. TreeBoN maintains a set of parent nodes, iteratively branching and pruning low-quality responses, thereby reducing computational overhead while maintaining high output quality. Our approach also leverages token-level rewards from Direct Preference Optimization (DPO) to guide tree expansion and prune low-quality paths. We evaluate TreeBoN using AlpacaFarm, HH-RLHF, UltraFeedback, GSM8K, and TutorEval datasets, demonstrating consistent improvements. Specifically, TreeBoN achieves the highest win rate of 65% on TutorEval and around 60% win rates across other different datasets, outperforming standard BoN with the same computational cost and showcasing its scalability and alignment efficacy.

Key facts. Cited by 61 papers in the map (data through 7 Oct 2026). Placed in Test-time compute and efficient thinking, one of 17 directions in LLM reasoning. Builds on 57 of its 74 references that are in the map. First posted on arXiv 18 Oct 2024.

Builds on

57 of 74 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024Theoretical guarantees on the best-of-n alignment policy3 Jan

Other references · 56

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2022Training language models to follow instructions with human feedbackcited by 15,712
  3. 2024The Llama 3 Herd of Modelscited by 14,192
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  5. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  6. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  7. 2017Deep reinforcement learning from human preferencescited by 3,846
  8. 2022Training a Helpful and Harmless Assistant with Reinforcement Learning from Human Feedbackcited by 3,634
  9. 2023Let's Verify Step by Stepcited by 3,598
  10. 2020Learning to summarize from human feedbackcited by 2,562
  11. 2022Fast Inference from Transformers via Speculative Decodingcited by 1,615
  12. 2022Scaling Laws for Reward Model Overoptimizationcited by 1,056
38 more
  1. 2023Accelerating Large Language Model Decoding with Speculative Samplingcited by 975
  2. 2024SimPO: Simple Preference Optimization with a Reference-Free Rewardcited by 951
  3. 2024Large Language Monkeys: Scaling Inference Compute with Repeated Samplingcited by 946
  4. 2023A General Theoretical Paradigm to Understand Learning from Human Preferencescited by 906
  5. 2023Reasoning with Language Model is Planning with World Modelcited by 846
  6. 2024Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Headscited by 821
  7. 2023AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedbackcited by 738
  8. 2023RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignmentcited by 672
  9. 2024Safety Alignment Should Be Made More Than Just a Few Tokens Deepcited by 450
  10. 2023UltraFeedback: Boosting Language Models with Scaled AI Feedbackcited by 394
  11. 2023SpecInfer: Accelerating Generative Large Language Model Serving with Tree-based Speculative Inference and Verificationcited by 359
  12. 2023Iterative Preference Learning from Human Feedback: Bridging Theory and Practice for RLHF under KL-Constraintcited by 346
  13. 2023Statistical Rejection Sampling Improves Preference Optimizationcited by 321
  14. 2023The Unlocking Spell on Base LLMs: Rethinking Alignment via In-Context Learningcited by 251
  15. 2024From r to Q^*: Your Language Model is Secretly a Q-Functioncited by 242
  16. 2023Large Language Models as Commonsense Knowledge for Large-Scale Task Planningcited by 231
  17. 2024Self-Play Preference Optimization for Language Model Alignmentcited by 228
  18. 2023Reward Model Ensembles Help Mitigate Overoptimizationcited by 225
  19. 2024Direct Nash Optimization: Teaching Language Models to Self-Improve with General Preferencescited by 176
  20. 2024Teaching Large Language Models to Reason with Reinforcement Learningcited by 158
  21. 2023RAIN: Your Language Models Can Align Themselves without Finetuningcited by 149
  22. 2024Token-level Direct Preference Optimizationcited by 135
  23. 2023Controlled Decoding from Language Modelscited by 133
  24. 2023SpecTr: Fast Speculative Decoding via Optimal Transportcited by 129
  25. 2024BoNBoN Alignment for Large Language Models and the Sweetness of Best-of-n Samplingcited by 120
  26. 2024Sequoia: Scalable, Robust, and Hardware-aware Speculative Decodingcited by 103
  27. 2024ARGS: Alignment as Reward-Guided Searchcited by 99
  28. 2024Decoding-Time Language Model Alignment with Multiple Objectivescited by 90
  29. 2023Reward-Augmented Decoding: Efficient Controlled Text Generation With a Unidirectional Reward Modelcited by 79
  30. 2023Contrastive Preference Learning: Learning from Human Feedback without RLcited by 77
  31. 2024Dense Reward for Free in Reinforcement Learning from Human Feedbackcited by 77
  32. 2024SpecDec++: Boosting Speculative Decoding via Adaptive Candidate Lengthscited by 73
  33. 2024BOND: Aligning LLMs with Best-of-N Distillationcited by 73
  34. 2023Don't throw away your value model! Generating more preferable text with Value-Guided Monte-Carlo Tree Search decodingcited by 59
  35. 2024DeAL: Decoding-time Alignment for Large Language Modelscited by 55
  36. 2024Value Augmented Sampling for Language Model Alignment and Personalizationcited by 49
  37. 2024Variational Best-of-N Alignmentcited by 49
  38. 2024RS-DPO: A Hybrid Rejection Sampling and Direct Preference Optimization Method for Alignment of Large Language Modelscited by 48

50 most cited of 56 All 56 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 2; 2025: 51; 2026: 8.

61 papers.

Sort:

By area:

  1. 2025Towards Reasoning Era: A Survey of Long Chain-of-Thought for Reasoning Large Language Modelscited by 381
  2. 2025From System 1 to System 2: A Survey of Reasoning Large Language Modelscited by 319
  3. 2025A Survey on Test-Time Scaling in Large Language Models: What, How, Where, and How Well?cited by 215
  4. 2025A Survey of Efficient Reasoning for Large Reasoning Models: Language, Multimodality, and Beyondcited by 131
  5. 2025Is Best-of-N the Best of Them? Coverage, Scaling, and Optimality in Inference-Time Alignmentcited by 91
  6. 2025Revisiting the Test-Time Scaling of o1-like Models: Do they Truly Possess Test-Time Scaling Capabilities?cited by 69
  7. 2025Sampling-Efficient Test-Time Scaling: Self-Estimating the Best-of-N Sampling in Early Decodingcited by 62
  8. 2024Scaling of Search and Learning: A Roadmap to Reproduce o1 from Reinforcement Learning Perspectivecited by 49
  9. 2025Test-Time Preference Optimization: On-the-Fly Alignment via Iterative Textual Feedbackcited by 41
  10. 2025A Survey of Test-Time Compute: From Intuitive Inference to Deliberate Reasoningcited by 40
  11. 2025Improving LLM Safety Alignment with Dual-Objective Optimizationcited by 40
  12. 2025Harnessing the Reasoning Economy: A Survey of Efficient Reasoning for Large Language Modelscited by 35
38 more
  1. 2024InfAlign: Inference-aware language model alignmentcited by 34
  2. 2025Inference-Time Reward Hacking in Large Language Modelscited by 32
  3. 2025Chain-of-Reasoning: Towards Unified Mathematical Reasoning in Large Language Models via a Multi-Paradigm Perspectivecited by 29
  4. 2025Soft Best-of-n Sampling for Model Alignmentcited by 28
  5. 2025AlphaOne: Reasoning Models Thinking Slow and Fast at Test Timecited by 27
  6. 2025Accelerating Large Language Model Reasoning via Speculative Searchcited by 16
  7. 2025ETS: Efficient Tree Search for Inference-Time Scalingcited by 15
  8. 2025Every Rollout Counts: Optimal Resource Allocation for Efficient Test-Time Scalingcited by 15
  9. 2025Generative AI Act II: Test Time Scaling Drives Cognition Engineeringcited by 14
  10. 2025SSRL: Self-Search Reinforcement Learningcited by 13
  11. 2025Sample Complexity and Representation Ability of Test-time Scaling Paradigmscited by 12
  12. 2025A Survey on Feedback-based Multi-step Reasoning for Large Language Models on Mathematicscited by 11
  13. 2025Temporal Consistency for LLM Reasoning Process Error Identificationcited by 10
  14. 2025Zero-Shot Verification-guided Chain of Thoughtscited by 9
  15. 2025Agents Are All You Need for LLM Unlearningcited by 9
  16. 2025Guided Speculative Inference for Efficient Test-Time Alignment of LLMscited by 9
  17. 2025AdaBoN: Adaptive Best-of-N Alignmentcited by 8
  18. 2025Best-of-Majority: Minimax-Optimal Strategy for Pass@k Inference Scalingcited by 8
  19. 2025From Trial to Triumph: Advancing Long Video Understanding via Visual Context Sample Scaling and Self-reward Alignmentcited by 6
  20. 2025BiasFilter: An Inference-Time Debiasing Framework for Large Language Modelscited by 6
  21. 2025Shallow Preference Signals: Large Language Model Aligns Even Better with Truncated Data?cited by 5
  22. 2025LLM-ML Teaming: Integrated Symbolic Decoding and Gradient Search for Valid and Stable Generative Feature Transformationcited by 5
  23. 2025Monitoring Decoding: Mitigating Hallucination via Evaluating the Factuality of Partial Response during Generationcited by 4
  24. 2025Reward-Shifted Speculative Sampling Is An Efficient Test-Time Weak-to-Strong Alignercited by 4
  25. 2025Flow Autoencoders are Effective Protein Tokenizerscited by 4
  26. 2025Inference-Time Personalized Alignment with a Few User Preference Queriescited by 4
  27. 2026Identifying and Transferring Reasoning-Critical Neurons: Improving LLM Inference Reliability via Activation Steeringcited by 4
  28. 2026Cactus: Accelerating Auto-Regressive Decoding with Constrained Acceptance Speculative Samplingcited by 4
  29. 2025textttSPECS: Faster Test-Time Scaling through Speculative Draftscited by 3
  30. 2025Scale, Don't Fine-tune: Guiding Multimodal LLMs for Efficient Visual Place Recognition at Test-Timecited by 3
  31. 2025Reasoning over Boundaries: Enhancing Specification Alignment via Test-time Deliberationcited by 3
  32. 2025Test-Time Alignment for Large Language Models via Textual Model Predictive Controlcited by 2
  33. 2025PITA: Preference-Guided Inference-Time Alignment for LLM Post-Trainingcited by 2
  34. 2025A Survey on Training-free Alignment of Large Language Modelscited by 2
  35. 2025Navigating the Exploration-Exploitation Tradeoff in Inference-Time Scaling of Diffusion Modelscited by 2
  36. 2026Guided by the Plan: Enhancing Faithful Autoregressive Text-to-Audio Generation with Guided Decodingcited by 2
  37. 2025AnesSuite: A Comprehensive Benchmark and Dataset Suite for Anesthesiology Reasoning in LLMscited by 1
  38. 2025Aligning Frozen LLMs by Reinforcement Learning: An Iterative Reweight-then-Optimize Approachcited by 1

50 most cited of 61 All 61 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Inference Time Alignment with Reward-Guided Tree Search
  2. 2025W2S-AlignTree: Weak-to-Strong Inference-Time Alignment for Large Language Models via Monte Carlo Tree Search
  3. 2026Revisiting the (Sub)Optimality of Best-of-N for Inference-Time Alignment
  4. 2025On Almost Surely Safe Alignment of Large Language Models at Inference-Time
  5. 2024Fast Best-of-N Decoding via Speculative Rejection
  6. 2025Inference-time Alignment in Continuous Space
  7. 2025Bridging Draft Policy Misalignment: Group Tree Optimization for Speculative Decoding
  8. 2026TreeGraft: Adaptive Multi-Drafter Grafting for Tree-Based Speculative Decoding