mlsanityData to 14 Oct 2026

Rubric-Guided Process Reward for Stepwise Model Routing

arXiv 2605.29310 ↗First posted 28 May 2026cs.AI cs.CLcited by 2

On the map: LLM reasoning → Reinforcement learning for reasoning

also in 1 more
Abstract (arXiv)

Stepwise model routing improves the efficiency of Large Reasoning Models (LRMs) by assigning each reasoning step to a suitable model. Recent methods formulate routing as a sequential decision process and train the router with reinforcement learning. However, although they model routing as a process, they still supervise the router with outcome rewards. Such rewards only reflect final answer correctness and fail to evaluate intermediate routing decisions, which can weaken performance and generalization. To address this gap, we propose RoRo, a rubric-guided process reward framework for stepwise model routing. RoRo first collects diverse routing trajectories and constructs preference pairs based on outcome, cost, and process quality. It then trains a Rubricor to generate a query-specific evaluation rubric and a Judge to score routing trajectories under this rubric through alternating optimization. The resulting process rewards are combined with outcome rewards to optimize the routing policy via GRPO. Experiments on five reasoning benchmarks under both same-family and cross-family settings show that RoRo consistently outperforms strong baselines and achieves better accuracy and cost trade-offs.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 41 of its 71 references that are in the map. First posted on arXiv 28 May 2026.

Builds on

41 of 71 references.

Closest ancestors · 8

Same direction, newest first

  1. 2026Step-wise Rubric Rewards for LLM Reasoning17 May
  2. 2026One Refiner to Unlock Them All: Inference-Time Reasoning Elicitation via Reinforcement Query Refinement28 Apr
  3. 2026Outcome Accuracy is Not Enough: Aligning the Reasoning Process of Reward Models4 Feb
  4. 2025Scheduling Your LLM Reinforcement Learning with Reasoning Trees28 Oct
  5. 2025Rethinking Entropy Interventions in RLVR: An Entropy Change Perspective11 Oct
  6. 2025Breaking the Exploration Bottleneck: Rubric-Scaffolded Reinforcement Learning for General LLM Reasoning23 Aug
  7. 2023Let's reward step by step: Step-Level reward model as the Navigators for Reasoning16 Oct
  8. 2023Let's Verify Step by Step31 May

Other references · 33

  1. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  2. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  3. 2025Qwen3 Technical Reportcited by 8,820
  4. 2021Measuring Mathematical Problem Solving With the MATH Datasetcited by 5,371
  5. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,879
  6. 2024Scaling LLM Test-Time Compute Optimally can be More Effective than Scaling Model Parameterscited by 1,986
  7. 2023A Survey on Evaluation of Large Language Modelscited by 1,242
  8. 2024RouteLLM: Learning to Route LLMs with Preference Datacited by 682
  9. 2025Rubrics as Rewards: Reinforcement Learning Beyond Verifiable Domainscited by 366
  10. 2023Beyond Chinchilla-Optimal: Accounting for Inference in Language Model Scaling Lawscited by 159
  11. 2026Beyond Benchmarks: MathArena as an Evaluation Platform for Mathematics with LLMscited by 137
  12. 2025OpenRubrics: Towards Scalable Synthetic Rubric Generation for Reward Modeling and LLM Alignmentcited by 132
21 more
  1. 2025Reward-Guided Speculative Decoding for Efficient LLM Reasoningcited by 106
  2. 2025SpecReason: Fast and Accurate Inference-Time Compute via Speculative Reasoningcited by 59
  3. 2023DocMath-Eval: Evaluating Math Reasoning Capabilities of LLMs in Understanding Long and Specialized Documentscited by 45
  4. 2024Smoothie: Label Free Language Model Routingcited by 31
  5. 2026Look Inward to Explore Outward: Learning Temperature Policy from LLM Internal States via Hierarchical RLcited by 27
  6. 2026Open Rubric System: Scaling Reinforcement Learning with Pairwise Adaptive Rubriccited by 27
  7. 2026Reinforcing Chain-of-Thought Reasoning with Self-Evolving Rubricscited by 26
  8. 2026Not All Directions Matter: Towards Structured and Task-Aware Low-Rank Model Adaptationcited by 14
  9. 2025QUARK: Quantization-Enabled Circuit Sharing for Transformer Acceleration by Exploiting Common Patterns in Nonlinear Operationscited by 11
  10. 2025SpecQuant: Spectral Decomposition and Adaptive Truncation for Ultra-Low-Bit LLMs Quantizationcited by 10
  11. 2026AndroTMem: From Interaction Trajectories to Anchored Memory in Long-Horizon GUI Agentscited by 10
  12. 2025Confidence-Guided Stepwise Model Routing for Cost-Efficient Reasoningcited by 9
  13. 2026How Order-Sensitive Are LLMs? OrderProbe for Deterministic Structural Reconstructioncited by 9
  14. 2026EfficientPosterGen: Semantic-aware Efficient Poster Generation via Token Compression and Accurate Violation Detectioncited by 9
  15. 2025Auditing Data Membership in Reinforcement Learning With Verifiable Rewardscited by 8
  16. 2026TRIM: Hybrid Inference via Targeted Stepwise Routing in Multi-Step Reasoning Taskscited by 8
  17. 2026KBVQ-MoE: KLT-guided SVD with Bias-Corrected Vector Quantization for MoE Large Language Modelscited by 8
  18. 2025Enrich-on-Graph: Query-Graph Alignment for Complex Reasoning with LLM Enrichingcited by 7
  19. 2025Last Layer Logits to Logic: Empowering LLMs with Logic-Consistent Structured Knowledge Reasoningcited by 4
  20. 2024LLMs as Probabilistic Minimally Adequate Teachers for DFA Learningcited by 3
  21. 2026CirrusBench: Evaluating LLM-based Agents Beyond Correctness in Real-World Cloud Service Environmentscited by 3

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026AgenticRag-R1: Agentic Reinforcement Learning with Stack Memory for Multi-Step Reasoning, Retrieval and Memorizingcited by 2
  2. 2026All for 1-Bit: Towards Genuine 1-Bit Post-Training Quantization for LLMscited by 2

Similar, not cited

Close in wording, no citation either way

  1. 2026Policy-Guided Stepwise Model Routing for Cost-Effective Reasoning
  2. 2026Models Under SCOPE: Scalable and Controllable Routing via Pre-hoc Reasoning
  3. 2025Reward and Guidance through Rubrics: Promoting Exploration to Improve Multi-Domain Reasoning
  4. 2025StepWiser: Stepwise Generative Judges for Wiser Reasoning
  5. 2025A note on conditional PAC-efficient reasoning in large language model routing
  6. 2024PSPO*: An Effective Process-supervised Policy Optimization for Reasoning Alignment
  7. 2026RubricEM: Meta-RL with Rubric-guided Policy Decomposition beyond Verifiable Rewards
  8. 2025ReasonFlux-PRM: Trajectory-Aware PRMs for Long Chain-of-Thought Reasoning in LLMs