mlsanityData to 7 Oct 2026

Fast Best-of-N Decoding via Speculative Rejection

arXiv 2410.20290 ↗First posted 26 Oct 2024cs.CLcited by 112

Not on the map. Why?

Cited by 67 in LLM reasoning, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

The safe and effective deployment of Large Language Models (LLMs) involves a critical step called alignment, which ensures that the model's responses are in accordance with human preferences. Prevalent alignment techniques, such as DPO, PPO and their variants, align LLMs by changing the pre-trained model weights during a phase called post-training. While predominant, these post-training methods add substantial complexity before LLMs can be deployed. Inference-time alignment methods avoid the complex post-training step and instead bias the generation towards responses that are aligned with human preferences. The best-known inference-time alignment method, called Best-of-N, is as effective as the state-of-the-art post-training procedures. Unfortunately, Best-of-N requires vastly more resources at inference time than standard decoding strategies, which makes it computationally not viable. In this work, we introduce Speculative Rejection, a computationally-viable inference-time alignment algorithm. It generates high-scoring responses according to a given reward model, like Best-of-N does, while being between 16 to 32 times more computationally efficient.

Key facts. Cited by 112 papers in the map (data through 7 Oct 2026). Builds on 60 of its 79 references that are in the map. First posted on arXiv 26 Oct 2024.

Builds on

60 of 79 references.

References · 60

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2022Training language models to follow instructions with human feedbackcited by 15,712
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  5. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  6. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 7,916
  7. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  8. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  9. 2017Deep reinforcement learning from human preferencescited by 3,846
  10. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  11. 2023Mistral 7Bcited by 3,065
  12. 2022Constitutional AI: Harmlessness from AI Feedbackcited by 2,950
38 more
  1. 2020Learning to summarize from human feedbackcited by 2,562
  2. 2021WebGPT: Browser-assisted question-answering with human feedbackcited by 1,722
  3. 2022Fast Inference from Transformers via Speculative Decodingcited by 1,615
  4. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,311
  5. 2022Scaling Laws for Reward Model Overoptimizationcited by 1,056
  6. 2023Accelerating Large Language Model Decoding with Speculative Samplingcited by 975
  7. 2023A General Theoretical Paradigm to Understand Learning from Human Preferencescited by 906
  8. 2024Length-Controlled AlpacaEval: A Simple Way to Debias Automatic Evaluatorscited by 865
  9. 2023Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedbackcited by 743
  10. 2023AlpacaFarm: A Simulation Framework for Methods that Learn from Human Feedbackcited by 738
  11. 2023RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignmentcited by 672
  12. 2017BranchyNet: Fast Inference via Early Exiting from Deep Neural Networkscited by 591
  13. 2022Improving alignment of dialogue agents via targeted human judgementscited by 527
  14. 2024Negative Preference Optimization: From Catastrophic Collapse to Effective Unlearningcited by 484
  15. 2023RRHF: Rank Responses to Align Language Models with Human Feedback without tearscited by 454
  16. 2023SLiC-HF: Sequence Likelihood Calibration with Human Feedbackcited by 395
  17. 2023Toxicity in ChatGPT: Analyzing Persona-assigned Language Modelscited by 356
  18. 2024ReST-MCTS*: LLM Self-Training via Process Reward Guided Tree Searchcited by 338
  19. 2023Statistical Rejection Sampling Improves Preference Optimizationcited by 321
  20. 2023Preference Ranking Optimization for Human Alignmentcited by 283
  21. 2022The Alignment Problem from a Deep Learning Perspectivecited by 246
  22. 2024From r to Q^*: Your Language Model is Secretly a Q-Functioncited by 242
  23. 2024Self-Play Preference Optimization for Language Model Alignmentcited by 228
  24. 2023Reward Model Ensembles Help Mitigate Overoptimizationcited by 225
  25. 2020FastBERT: a Self-distilling BERT with Adaptive Inference Timecited by 224
  26. 2024A Survey on Efficient Inference for Large Language Modelscited by 197
  27. 2023Helping or Herding? Reward Model Ensembles Mitigate but do not Eliminate Reward Hackingcited by 195
  28. 2022Fine-tuning language models to find agreement among humans with diverse preferencescited by 189
  29. 2024Accessing GPT-4 level Mathematical Olympiad Solutions via Monte Carlo Tree Self-refine with LLaMa-3 8Bcited by 156
  30. 2024Token-level Direct Preference Optimizationcited by 135
  31. 2023Controlled Decoding from Language Modelscited by 133
  32. 2023SpecTr: Fast Speculative Decoding via Optimal Transportcited by 129
  33. 2024DPO Meets PPO: Reinforced Token Optimization for RLHFcited by 126
  34. 2020The Right Tool for the Job: Matching Model and Instance Complexitiescited by 125
  35. 2023Training Language Models with Language Feedback at Scalecited by 125
  36. 2022Calibrating Sequence likelihood Improves Conditional Language Generationcited by 121
  37. 2024BoNBoN Alignment for Large Language Models and the Sweetness of Best-of-n Samplingcited by 120
  38. 2024Theoretical guarantees on the best-of-n alignment policycited by 116

50 most cited of 60 All 60 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 5; 2025: 78; 2026: 29.

112 papers.

Sort:

By area:

  1. 2025Stop Overthinking: A Survey on Efficient Reasoning for Large Language Modelscited by 456
  2. 2024LLMs-as-Judges: A Comprehensive Survey on LLM-based Evaluation Methodscited by 451
  3. 2025Reasoning Models Can Be Effective Without Thinkingcited by 182
  4. 2025DAST: Difficulty-Adaptive Slow-Thinking for Large Reasoning Modelscited by 154
  5. 2024ShadowKV: KV Cache in Shadows for High-Throughput Long-Context LLM Inferencecited by 132
  6. 2025A Survey of Efficient Reasoning for Large Reasoning Models: Language, Multimodality, and Beyondcited by 131
  7. 2025LLM Post-Training: A Deep Dive into Reasoning Large Language Modelscited by 129
  8. 2025Efficient Reasoning Models: A Surveycited by 93
  9. 2025Sample More to Think Less: Group Filtered Policy Optimization for Concise Reasoningcited by 75
  10. 2025Revisiting the Test-Time Scaling of o1-like Models: Do they Truly Possess Test-Time Scaling Capabilities?cited by 69
  11. 2025SpatialReasoner: Towards Explicit and Generalizable 3D Spatial Reasoningcited by 65
  12. 2025Wait, We Don't Need to "Wait"! Removing Thinking Tokens Improves Reasoning Efficiencycited by 65
38 more
  1. 2025Sampling-Efficient Test-Time Scaling: Self-Estimating the Best-of-N Sampling in Early Decodingcited by 62
  2. 2025A Survey on (M)LLM-Based GUI Agentscited by 61
  3. 2025Pangu Embedded: An Efficient Dual-system LLM Reasoner with Metacognitioncited by 61
  4. 2024Efficiently Scaling LLM Reasoning with Certaindexcited by 56
  5. 2024Scaling of Search and Learning: A Roadmap to Reproduce o1 from Reinforcement Learning Perspectivecited by 49
  6. 2025PonderLM: Pretraining Language Models to Ponder in Continuous Spacecited by 44
  7. 2025Harnessing the Reasoning Economy: A Survey of Efficient Reasoning for Large Language Modelscited by 35
  8. 2025ConCISE: Confidence-guided Compression in Step-by-step Efficient Reasoningcited by 33
  9. 2025InSTA: Towards Internet-Scale Training For Agentscited by 31
  10. 2025Reasoning on a Budget: A Survey of Adaptive and Controllable Test-Time Compute in LLMscited by 31
  11. 2025SPEED-RL: Faster Training of Reasoning Models via Online Curriculum Learningcited by 27
  12. 2025VeriThinker: Learning to Verify Makes Reasoning Model Efficientcited by 24
  13. 2025System-1.5 Reasoning: Traversal in Language and Latent Spaces with Dynamic Shortcutscited by 23
  14. 2025A Survey of Token Compression for Efficient Multimodal Large Language Modelscited by 22
  15. 2025Early Stopping Chain-of-thoughts in Large Language Modelscited by 19
  16. 2025A Survey of Slow Thinking-based Reasoning LLMs using Reinforced Learning and Inference-time Scaling Lawcited by 18
  17. 2024Search, Verify and Feedback: Towards Next Generation Post-training Paradigm of Foundation Models via Verifier Engineeringcited by 16
  18. 2025Learning from Peers in Reasoning Modelscited by 16
  19. 2025Causal Sufficiency and Necessity Improves Chain-of-Thought Reasoningcited by 16
  20. 2025AgentTTS: Large Language Model Agent for Test-time Compute-optimal Scaling Strategy in Complex Taskscited by 16
  21. 2026RAGEN-2: Reasoning Collapse in Agentic RLcited by 16
  22. 2025ETS: Efficient Tree Search for Inference-Time Scalingcited by 15
  23. 2025Generative AI Act II: Test Time Scaling Drives Cognition Engineeringcited by 14
  24. 2025Sailing by the Stars: A Survey on Reward Models and Learning Strategies for Learning from Rewardscited by 14
  25. 2025TrimR: Verifier-based Training-Free Thinking Compression for Efficient Test-Time Scalingcited by 14
  26. 2025PonderLM-2: Pretraining LLM with Latent Thoughts in Continuous Spacecited by 14
  27. 2025On Almost Surely Safe Alignment of Large Language Models at Inference-Timecited by 13
  28. 2026CycleVLA: Proactive Self-Correcting Vision-Language-Action Models via Subtask Backtracking and Minimum Bayes Risk Decodingcited by 13
  29. 2025φ-Decoding: Adaptive Foresight Sampling for Balanced Inference-Time Exploration and Exploitationcited by 12
  30. 2025Kinetics: Rethinking Test-Time Scaling Lawscited by 11
  31. 2025Revisiting Model Interpolation for Efficient Reasoningcited by 10
  32. 2025From Outcomes to Processes: Guiding PRM Learning from ORM for Inference-Time Alignmentcited by 9
  33. 2025When Life Gives You Samples: The Benefits of Scaling up Inference Compute for Multilingual LLMscited by 9
  34. 2025Iterative Deepening Sampling as Efficient Test-Time Scalingcited by 8
  35. 2025AdaBoN: Adaptive Best-of-N Alignmentcited by 8
  36. 2025How Far Are We from Optimal Reasoning Efficiency?cited by 8
  37. 2025Strategic Scaling of Test-Time Compute: A Bandit Learning Approachcited by 8
  38. 2025Scalable LLM Reasoning Acceleration with Low-rank Distillationcited by 7

50 most cited of 112 All 112 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025Judge Decoding: Faster Speculative Sampling Requires Going Beyond Model Alignment
  2. 2025Alignment-Augmented Speculative Decoding with Alignment Sampling and Conditional Verification
  3. 2025Is Best-of-N the Best of Them? Coverage, Scaling, and Optimality in Inference-Time Alignment
  4. 2024Nudging: Inference-time Alignment of LLMs via Guided Decoding
  5. 2025GRIFFIN: Effective Token Alignment for Faster Speculative Decoding
  6. 2025Guided Speculative Inference for Efficient Test-Time Alignment of LLMs
  7. 2024Inference Time Alignment with Reward-Guided Tree Search
  8. 2024Approximately Aligned Decoding