mlsanityData to 14 Oct 2026

Reward-Guided Speculative Decoding for Efficient LLM Reasoning

arXiv 2501.19324 ↗First posted 31 Jan 2025cs.CL cs.AIcited by 106

On the map: LLM reasoning → Test-time compute and efficient thinking

also in 2 more
Abstract (arXiv)

We introduce Reward-Guided Speculative Decoding (RSD), a novel framework aimed at improving the efficiency of inference in large language models (LLMs). RSD synergistically combines a lightweight draft model with a more powerful target model, incorporating a controlled bias to prioritize high-reward outputs, in contrast to existing speculative decoding methods that enforce strict unbiasedness. RSD employs a process reward model to evaluate intermediate decoding steps and dynamically decide whether to invoke the target model, optimizing the trade-off between computational cost and output quality. We theoretically demonstrate that a threshold-based mixture strategy achieves an optimal balance between resource utilization and performance. Extensive evaluations on challenging reasoning benchmarks, including Olympiad-level tasks, show that RSD delivers significant efficiency gains against decoding with the target model only (up to 4.4x fewer FLOPs), while achieving significant better accuracy than parallel decoding method on average (up to +3.5). These results highlight RSD as a robust and cost-effective approach for deploying LLMs in resource-intensive scenarios. The code is available at https://github.com/BaohaoLiao/RSD.

Key facts. Cited by 106 papers in the map (data through 14 Oct 2026). Placed in Test-time compute and efficient thinking, one of 17 directions in LLM reasoning. Builds on 57 of its 62 references that are in the map. First posted on arXiv 31 Jan 2025.

Builds on

57 of 62 references.

Closest ancestors · 4

Same direction, newest first

  1. 2024Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameters6 Aug
  2. 2024ThinK: Thinner Key Cache by Query-Driven Pruning30 Jul
  3. 2024EAGLE: Speculative Sampling Requires Rethinking Feature Uncertainty26 Jan
  4. 2023Tree of Thoughts: Deliberate Problem Solving with Large Language Models17 May

Other references · 53

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020Language Models are Few-Shot Learnerscited by 32,419
  3. 2024The Llama 3 Herd of Modelscited by 14,715
  4. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  5. 2020Measuring Massive Multitask Language Understandingcited by 7,963
  6. 2020Scaling Laws for Neural Language Modelscited by 6,586
  7. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  8. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  9. 2024Qwen2.5 Technical Reportcited by 4,942
  10. 2024GPT-4o System Cardcited by 4,588
  11. 2023Let's Verify Step by Stepcited by 3,789
  12. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,224
38 more
  1. 2022Training Compute-Optimal Large Language Modelscited by 3,019
  2. 2019The Curious Case of Neural Text Degenerationcited by 2,916
  3. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,879
  4. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,919
  5. 2022Fast Inference from Transformers via Speculative Decodingcited by 1,653
  6. 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,392
  7. 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,298
  8. 2024OlympiadBench: A Challenging Benchmark for Promoting AGI with Olympiad-Level Bilingual Multimodal Scientific Problemscited by 1,136
  9. 2024Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvementcited by 1,090
  10. 2023Accelerating Large Language Model Decoding with Speculative Samplingcited by 1,006
  11. 2024Large Language Monkeys: Scaling Inference Compute with Repeated Samplingcited by 991
  12. 2023Math-Shepherd: Verify and Reinforce LLMs Step-by-step without Human Annotationscited by 928
  13. 2024SnapKV: LLM Knows What You are Looking for Before Generationcited by 880
  14. 2024Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Headscited by 843
  15. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Modelscited by 832
  16. 2023A Simple and Effective Pruning Approach for Large Language Modelscited by 800
  17. 2023RAFT: Reward rAnked FineTuning for Generative Foundation Model Alignmentcited by 686
  18. 2017Deep Learning Scaling is Predictable, Empiricallycited by 582
  19. 2021Carbon Emissions and Large Neural Network Trainingcited by 567
  20. 2025The Lessons of Developing Process Reward Models in Mathematical Reasoningcited by 415
  21. 2023SpecInfer: Accelerating Generative Large Language Model Serving with Tree-based Speculative Inference and Verificationcited by 365
  22. 2024Skywork-Reward: Bag of Tricks for Reward Modeling in LLMscited by 327
  23. 2018Blockwise Parallel Decoding for Deep Autoregressive Modelscited by 323
  24. 2024LayerSkip: Enabling Early Exit Inference and Self-Speculative Decodingcited by 307
  25. 2024Break the Sequential Dependency of LLM Inference Using Lookahead Decodingcited by 298
  26. 2024ProcessBench: Identifying Process Errors in Mathematical Reasoningcited by 292
  27. 2024Unlocking Efficiency in Large Language Model Inference: A Comprehensive Survey of Speculative Decodingcited by 269
  28. 2024RLHF Workflow: From Reward Modeling to Online RLHFcited by 267
  29. 2024Arcee's MergeKit: A Toolkit for Merging Large Language Modelscited by 194
  30. 2024AlphaMath Almost Zero: Process Supervision without Processcited by 162
  31. 2023Beyond Chinchilla-Optimal: Accounting for Inference in Language Model Scaling Lawscited by 159
  32. 2024MathScale: Scaling Instruction Tuning for Mathematical Reasoningcited by 144
  33. 2024Mutual Reasoning Makes Smaller LLMs Stronger Problem-Solverscited by 138
  34. 2023SpecTr: Fast Speculative Decoding via Optimal Transportcited by 129
  35. 2024Sequoia: Scalable, Robust, and Hardware-aware Speculative Decodingcited by 103
  36. 2024TriForce: Lossless Acceleration of Long Sequence Generation with Hierarchical Speculative Decodingcited by 102
  37. 2023Cascade Speculative Drafting for Even Faster LLM Inferencecited by 83
  38. 2024MARIO: MAth Reasoning with code Interpreter Output -- A Reproducible Pipelinecited by 81

50 most cited of 53 All 53 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 68; 2026: 38.

106 papers.

Sort:

By area:

  1. 2025Stop Overthinking: A Survey on Efficient Reasoning for Large Language Modelscited by 464
  2. 2025Towards Reasoning Era: A Survey of Long Chain-of-Thought for Reasoning Large Language Modelscited by 384
  3. 2025Dynamic Early Exit in Reasoning Modelscited by 219
  4. 2025Reasoning Models Can Be Effective Without Thinkingcited by 187
  5. 2025Advances and Challenges in Foundation Agents: From Brain-Inspired Intelligence to Evolutionary, Collaborative, and Safe Systemscited by 126
  6. 2025Thinkless: LLM Learns When to Thinkcited by 113
  7. 2025Adaptive Chain-of-Focus Reasoning via Dynamic Visual Search and Zooming for Efficient VLMscited by 96
  8. 2025Efficient Reasoning Models: A Surveycited by 95
  9. 2025S-GRPO: Early Exit via Reinforcement Learning in Reasoning Modelscited by 87
  10. 2025Wait, We Don't Need to "Wait"! Removing Thinking Tokens Improves Reasoning Efficiencycited by 66
  11. 2025Efficient Inference for Large Reasoning Models: A Surveycited by 61
  12. 2025A Comprehensive Survey of Reward Models: Taxonomy, Applications, Challenges, and Futurecited by 53
38 more
  1. 2025Speculative Thinking: Enhancing Small-Model Reasoning with Large Model Guidance at Inference Timecited by 48
  2. 2025History Rhymes: Accelerating LLM Reinforcement Learning with RhymeRLcited by 38
  3. 2025Don't Overthink It: A Survey of Efficient R1-style Large Reasoning Modelscited by 37
  4. 2025Harnessing the Reasoning Economy: A Survey of Efficient Reasoning for Large Language Modelscited by 36
  5. 2025Scalable Chain of Thoughts via Elastic Reasoningcited by 34
  6. 2025R2R: Efficiently Navigating Divergent Reasoning Paths with Small-Large Model Token Routingcited by 29
  7. 2025Let LRMs Break Free from Overthinking via Self-Braking Tuningcited by 27
  8. 2025When to Continue Thinking: Adaptive Thinking Mode Switching for Efficient Reasoningcited by 26
  9. 2025VeriThinker: Learning to Verify Makes Reasoning Model Efficientcited by 24
  10. 2025LongSpec: Long-Context Lossless Speculative Decoding with Efficient Drafting and Verificationcited by 21
  11. 2026Self-Hinting Language Models Enhance Reinforcement Learningcited by 20
  12. 2026Does Your Reasoning Model Implicitly Know When to Stop Thinking?cited by 19
  13. 2025Lost at the Beginning of Reasoningcited by 15
  14. 2025StreamingThinker: Large Language Models Can Think While Readingcited by 15
  15. 2025Sailing by the Stars: A Survey on Reward Models and Learning Strategies for Learning from Rewardscited by 14
  16. 2025Taming the Long-Tail: Efficient Reasoning RL Training with Adaptive Draftercited by 14
  17. 2025A Survey on Feedback-based Multi-step Reasoning for Large Language Models on Mathematicscited by 11
  18. 2025Guided Speculative Inference for Efficient Test-Time Alignment of LLMscited by 11
  19. 2025AALC: Large Language Model Efficient Reasoning via Adaptive Accuracy-Length Controlcited by 11
  20. 2025Scaling LLM Test-Time Compute with Mobile NPU on Smartphonescited by 11
  21. 2025Exploring the System 1 Thinking Capability of Large Reasoning Modelscited by 10
  22. 2025Efficient Reasoning Through Suppression of Self-Affirmation Reflections in Large Reasoning Modelscited by 10
  23. 2025Confidence-Guided Stepwise Model Routing for Cost-Efficient Reasoningcited by 9
  24. 2025What makes Reasoning Models Different? Follow the Reasoning Leader for Efficient Decodingcited by 8
  25. 2025How Far Are We from Optimal Reasoning Efficiency?cited by 8
  26. 2025Enhancing Large Language Model Reasoning with Reward Models: An Analytical Surveycited by 8
  27. 2025Small Drafts, Big Verdict: Information-Intensive Visual Reasoning via Speculationcited by 8
  28. 2026TRIM: Hybrid Inference via Targeted Stepwise Routing in Multi-Step Reasoning Taskscited by 8
  29. 2026Neural Chain-of-Thought Search: Searching the Optimal Reasoning Path to Enhance Large Language Modelscited by 8
  30. 2026FoE: Forest of Errors Makes the First Solution the Best in Large Reasoning Modelscited by 8
  31. 2025Fractured Chain-of-Thought Reasoningcited by 7
  32. 2025Inverse Reinforcement Learning Meets Large Language Model Post-Training: Basics, Advances, and Opportunitiescited by 7
  33. 2025R-Stitch: Dynamic Trajectory Stitching for Efficient Reasoningcited by 7
  34. 2025Reward Models Identify Consistency, Not Causalitycited by 6
  35. 2025Think Before You Accept: Semantic Reflective Verification for Faster Speculative Decodingcited by 6
  36. 2025Simultaneous Multi-objective Alignment Across Verifiable and Non-verifiable Rewardscited by 6
  37. 2025PREMISE: Scalable and Strategic Prompt Optimization for Efficient Mathematical Reasoning in Large Modelscited by 5
  38. 2025Think How to Think: Mitigating Overthinking with Autonomous Difficulty Cognition in Large Reasoning Modelscited by 5

50 most cited of 106 All 106 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Closer Look at Efficient Inference Methods: A Survey of Speculative Decoding
  2. 2025Scaling Up, Speeding Up: A Benchmark of Speculative Decoding for Efficient LLM Test-Time Scaling
  3. 2024Recursive Speculative Decoding: Accelerating LLM Inference via Sampling Without Replacement
  4. 2025Speculative Decoding: Performance or Illusion?
  5. 2026From Tokens to Steps: Verification-Aware Speculative Decoding for Efficient Multi-Step Reasoning
  6. 2026Faster LLM Inference via Sequential Monte Carlo
  7. 2026Calibrated Speculative Decoding: Frequency-Guided Candidate Selection for Efficient Inference
  8. 2025Tutorial Proposal: Speculative Decoding for Efficient LLM Inference