mlsanityData to 14 Oct 2026

Speculative Reward Model Boosts Decision Making Ability of LLMs Cost-Effectively

arXiv 2506.00396 ↗First posted 31 May 2025cs.CLcited by 0

On the map: LLM alignment / RLHF / preference optimisation → Process reward models for reasoning

Abstract (arXiv)

Effective decision-making in Large Language Models (LLMs) is essential for handling intricate tasks. However, existing approaches prioritize performance but often overlook the balance between effectiveness and computational cost. To address this, we first introduce the 3E Criteria to systematically assess the cost-effectiveness of search strategies, revealing that existing methods often trade significant efficiency for marginal performance gains. To improve LLM decision-making while maintaining efficiency, we propose the Speculative Reward Model (SRM), a plug-and-play framework that seamlessly integrates with existing search strategies. Specifically, SRM employs an external reward assigner to predict optimal actions, reducing reliance on LLMs' internal self-evaluation. And a speculative verification mechanism is used to prune suboptimal choices and guide the search toward more promising steps. We evaluate SRM on several complex decision-making tasks including mathematical reasoning, planning and numerical reasoning in specialized domains. Experimental results show that SRM reduces costs to 1/10 of the original search framework on average while maintaining effectiveness.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Process reward models for reasoning, one of 10 directions in LLM alignment / RLHF / preference optimisation. Builds on 21 of its 39 references that are in the map. First posted on arXiv 31 May 2025.

Builds on

21 of 39 references.

Closest ancestors · 1

Same direction, newest first

  1. 2025PRMBench: A Fine-grained and Challenging Benchmark for Process-Level Reward Models6 Jan

Other references · 20

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2024The Llama 3 Herd of Modelscited by 14,715
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  4. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  5. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  6. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
  7. 2024Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameterscited by 1,986
  8. 2023Large Language Models Cannot Self-Correct Reasoning Yetcited by 1,017
  9. 2023Accelerating Large Language Model Decoding with Speculative Samplingcited by 1,006
  10. 2023Reasoning with Language Model is Planning with World Modelcited by 861
  11. 2021FinQA: A Dataset of Numerical Reasoning over Financial Datacited by 562
  12. 2015Compositional Semantic Parsing on Semi-Structured Tablescited by 515
8 more
  1. 2022PlanBench: An Extensible Benchmark for Evaluating Large Language Models on Planning and Reasoning about Changecited by 382
  2. 2024Quiet-STaR: Language Models Can Teach Themselves to Think Before Speakingcited by 301
  3. 2023Alphazero-like Tree-Search can Guide Large Language Model Decoding and Trainingcited by 295
  4. 2024Agent Q: Advanced Reasoning and Learning for Autonomous AI Agentscited by 208
  5. 2024Chain-of-Table: Evolving Tables in the Reasoning Chain for Table Understandingcited by 188
  6. 2022Speculative Decoding: Exploiting Speculative Execution for Accelerating Seq2seq Generationcited by 171
  7. 2023ToolChain*: Efficient Action Space Navigation in Large Language Models with A* Searchcited by 105
  8. 2023Everything of Thoughts: Defying the Law of Penrose Triangle for Thought Generationcited by 74

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Unifying Tree Search Algorithm and Reward Design for LLM Reasoning: A Survey
  2. 2024Enhancing LLM Reasoning with Reward-guided Tree Search
  3. 2025LLMs are Greedy Agents: Effects of RL Fine-tuning on Decision-Making Abilities
  4. 2025Accelerating Large Language Model Reasoning via Speculative Search
  5. 2024EVOLvE: Evaluating and Optimizing LLMs For In-Context Exploration
  6. 2026Breaking the Reward Barrier: Accelerating Tree-of-Thought Reasoning via Speculative Exploration
  7. 2025From Mathematical Reasoning to Code: Generalization of Process Reward Models in Test-Time Scaling
  8. 2024Cost-Effective Online Multi-LLM Selection with Versatile Reward Models