mlsanityData to 7 Oct 2026

CounterRoute: Self-Routed Reasoning via Hierarchical Counterfactual Credit Assignment

arXiv 2609.29109 ↗First posted 24 Sep 2026cs.AIcited by 0

On the map: LLM reasoning → Reinforcement learning for reasoning

Abstract (arXiv)

Reasoning-capable language models often produce long chains of thought when direct answers suffice, wasting inference compute. Dual-mode models offer both thinking and direct-answer modes, but typically leave mode selection to users. Automating this selection while improving responses under both modes is challenging because routing targets evolve with the policy, strong initial mode preferences can destabilize exploration, and sequence-level objectives entangle routing with response learning. We introduce CounterRoute, an online reinforcement-learning framework that jointly learns routing and mode-conditioned responses in one shared policy directly from a native dual-mode checkpoint, without method-specific SFT warm-up. Paired current-policy counterfactual rollouts estimate cross-mode routing credit, which is assigned only to the routing token, while within-mode GRPO trains response tokens. A paired-to-self-routed curriculum stabilizes early training with forced coverage of both modes before progressively transferring training to autonomous routing. Across nine benchmarks, CounterRoute better balances accuracy and efficiency than heuristic and learned adaptive-routing methods, improving macro-average accuracy at both Qwen3 scales while cutting mean generated tokens by 51% and 41%, respectively, versus always-thinking checkpoints. On instruction-following and commonsense benchmarks where direct answering is strong, think rates fall as low as 1% while response quality improves. Despite training only on math and synthetic instruction-following data, its routing and response quality generalize to held-out tasks in coding, science, knowledge, and commonsense.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Reinforcement learning for reasoning, one of 17 directions in LLM reasoning. Builds on 21 of its 25 references that are in the map. First posted on arXiv 24 Sep 2026.

Builds on

21 of 25 references.

Closest ancestors · 4

Same direction, newest first

  1. 2025Shop-R1: Rewarding LLMs to Simulate Human Behavior in Online Shopping via Reinforcement Learning23 Jul
  2. 2025DAPO: An Open-Source LLM Reinforcement Learning System at Scale18 Mar
  3. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning22 Jan
  4. 2023Let's Verify Step by Step31 May

Other references · 17

  1. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,302
  2. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  3. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 8,851
  4. 2025Qwen3 Technical Reportcited by 8,195
  5. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  6. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  7. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  8. 2023GPQA: A Graduate-Level Google-Proof Q&A Benchmarkcited by 2,748
  9. 2024MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmarkcited by 1,758
  10. 2025Generalizing Verifiable Instruction Followingcited by 178
  11. 2025AdaptThink: Reasoning Models Can Learn When to Thinkcited by 141
  12. 2025Thinkless: LLM Learns When to Thinkcited by 107
5 more
  1. 2024Self-play with Execution Feedback: Improving Instruction-following Capabilities of Large Language Modelscited by 73
  2. 2025Think Only When You Need with Large Hybrid-Reasoning Modelscited by 64
  3. 2025Learning When to Think: Shaping Adaptive Reasoning in R1-Style Models via Multi-Stage RLcited by 59
  4. 2025Think or Not? Selective Reasoning via Reinforcement Learning for Vision-Language Modelscited by 30
  5. 2026Learning When to Think: Adaptive Reasoning for Test-Time Compute Allocationcited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026T-Router: Learning Thalamic Routing for Reasoning with Parameter-Efficient Reinforcement Learning
  2. 2026Policy-Guided Stepwise Model Routing for Cost-Effective Reasoning
  3. 2026CataOPD: Catalytic On-Policy Distillation for Large Language Model Reasoning
  4. 2026Not All Tokens Deserve Equal Credit: Counterfactual Sensitivity Credit Reallocation for Long-CoT Reasoning
  5. 2026Sampling via Decision-Flow: Training-Free Extraction of Improved Latent Reasoning Paths in Large Language Models
  6. 2026SeqRoute: Global Budget-Aware Sequential LLM Routing via Offline Reinforcement Learning
  7. 2025HierRouter: Coordinated Routing of Specialized Large Language Models via Reinforcement Learning
  8. 2025Future Policy Approximation for Offline Reinforcement Learning in LLM Reasoning