mlsanityData to 14 Oct 2026

Policy-Guided Stepwise Model Routing for Cost-Effective Reasoning

arXiv 2605.06116 ↗First posted 7 May 2026cs.AIcited by 0

On the map: LLM reasoning → Test-time compute and efficient thinking

Abstract (arXiv)

Inference-time computation has greatly enhanced the performance of large language models (LLMs) on challenging reasoning tasks, but this strategy can incur high inference costs. One solution is to route intermediate chain-of-thought (CoT) states to language models of different sizes; however, existing approaches rely on handcrafted routing strategies that limit performance, or on training large process reward models that may be infeasible in many applications. We formulate stepwise model routing as a constrained decision-making problem, which we solve by training a small control policy using reinforcement learning in conjunction with threshold calibration to tune the performance-efficiency tradeoff. We validate our method on three math benchmarks (GSM8K, MATH500, and OmniMath) on both open and closed models. Our method consistently improves the accuracy-cost tradeoff compared to handcrafted approaches, while achieving a comparable tradeoff to methods that require training large process reward models.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Test-time compute and efficient thinking, one of 17 directions in LLM reasoning. Builds on 29 of its 30 references that are in the map. First posted on arXiv 7 May 2026.

Builds on

29 of 30 references.

Closest ancestors · 3

Same direction, newest first

  1. 2025Confidence-Guided Stepwise Model Routing for Cost-Efficient Reasoning9 Nov
  2. 2025SpecReason: Fast and Accurate Inference-Time Compute via Speculative Reasoning10 Apr
  3. 2025Reward-Guided Speculative Decoding for Efficient LLM Reasoning31 Jan

Other references · 26

  1. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  2. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  3. 2020Scaling Laws for Neural Language Modelscited by 6,586
  4. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  5. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
  6. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  7. 2023Let's Verify Step by Stepcited by 3,789
  8. 2022Fast Inference from Transformers via Speculative Decodingcited by 1,653
  9. 2024Qwen2.5-Math Technical Report: Toward Mathematical Expert Model via Self-Improvementcited by 1,090
  10. 2023Accelerating Large Language Model Decoding with Speculative Samplingcited by 1,006
  11. 2024Large Language Monkeys: Scaling Inference Compute with Repeated Samplingcited by 991
  12. 2017Constrained Policy Optimizationcited by 983
14 more
  1. 2018IMPALA: Scalable Distributed Deep-RL with Importance Weighted Actor-Learner Architecturescited by 883
  2. 2023FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performancecited by 853
  3. 2024RouteLLM: Learning to Route LLMs with Preference Datacited by 682
  4. 2017Deep Learning Scaling is Predictable, Empiricallycited by 582
  5. 2024Hybrid LLM: Cost-Efficient and Quality-Aware Query Routingcited by 372
  6. 2023Robots That Ask For Help: Uncertainty Alignment for Large Language Model Plannerscited by 308
  7. 2024Omni-MATH: A Universal Olympiad Level Mathematic Benchmark For Large Language Modelscited by 284
  8. 2023Beyond Chinchilla-Optimal: Accounting for Inference in Language Model Scaling Lawscited by 159
  9. 2023Fly-Swat or Cannon? Cost-Effective Language Model Choice via Meta-Modelingcited by 95
  10. 2025Reasoning with Sampling: Your Base Model is Smarter Than You Thinkcited by 89
  11. 2024KOR-Bench: Benchmarking Language Models on Knowledge-Orthogonal Reasoning Taskscited by 70
  12. 2024Online conformal prediction with decaying step sizescited by 56
  13. 2025xVerify: Efficient Answer Verifier for Reasoning Model Evaluationscited by 55
  14. 2025Conformal Constrained Policy Optimization for Cost-Effective LLM Agentscited by 7

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Fast Thinking for Large Language Models
  2. 2025HierRouter: Coordinated Routing of Specialized Large Language Models via Reinforcement Learning
  3. 2026TRIM: Hybrid Inference via Targeted Stepwise Routing in Multi-Step Reasoning Tasks
  4. 2026CataOPD: Catalytic On-Policy Distillation for Large Language Model Reasoning
  5. 2026Reasoning Is Not Free: Robust Adaptive Cost-Efficient Routing for LLM-as-a-Judge
  6. 2026The Art of Efficient Reasoning: Data, Reward, and Optimization
  7. 2026Rubric-Guided Process Reward for Stepwise Model Routing
  8. 2025Route to Reason: Adaptive Routing for LLM and Reasoning Strategy Selection