mlsanityData to 14 Oct 2026

Router-R1: Teaching LLMs Multi-Round Routing and Aggregation via Reinforcement Learning

arXiv 2506.09033 ↗First posted 10 Jun 2025cs.CL cs.AI cs.LGcited by 72

Not on the map. Why?

Cited by 25 in LLM agents and tool use, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

The rapid emergence of diverse large language models (LLMs) has spurred the development of LLM routers that assign user queries to the most suitable model. However, existing LLM routers typically perform a single-round, one-to-one mapping (textiti.e., assigning each query to a single model in isolation), which limits their capability to tackle complex tasks that demand the complementary strengths of multiple LLMs. In this paper, we present Router-R1, a reinforcement learning (RL)-based framework that formulates multi-LLM routing and aggregation as a sequential decision process. Router-R1 instantiates the router itself as a capable LLM, leveraging its reasoning ability to interleave "think" actions (internal deliberation) with "route" actions (dynamic model invocation), and integrates each response into its evolving context. To facilitate learning, we employ a lightweight rule-based reward comprising format rewards, final outcome rewards, and a novel cost reward for optimizing the balance between performance and cost, opening a pathway toward enhancing performance-cost trade-offs via RL. Router-R1 also conditions only on simple model descriptors such as pricing, latency, and example performance, enabling strong generalization to unseen model selection. Experiments on seven general and multi-hop QA benchmarks show that Router-R1 outperforms several strong baselines, achieving superior performance while maintaining robust generalization and cost management.

Key facts. Cited by 72 papers in the map (data through 14 Oct 2026). Builds on 35 of its 39 references that are in the map. First posted on arXiv 10 Jun 2025.

Builds on

35 of 39 references.

References · 35

  1. 2022Training language models to follow instructions with human feedbackcited by 15,971
  2. 2017Proximal Policy Optimization Algorithmscited by 15,215
  3. 2024The Llama 3 Herd of Modelscited by 14,715
  4. 2022Chain-of-Thought Prompting Elicits Reasoning in Large Language Modelscited by 12,459
  5. 2024DeepSeekMath: Pushing the Limits of Mathematical Reasoning in Open Language Modelscited by 9,379
  6. 2023Direct Preference Optimization: Your Language Model is Secretly a Reward Modelcited by 8,065
  7. 2025DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learningcited by 6,330
  8. 2024Qwen2.5 Technical Reportcited by 4,942
  9. 2020Dense Passage Retrieval for Open-Domain Question Answeringcited by 3,824
  10. 2018HotpotQA: A Dataset for Diverse, Explainable Multi-hop Question Answeringcited by 3,754
  11. 2017TriviaQA: A Large Scale Distantly Supervised Challenge Dataset for Reading Comprehensioncited by 2,955
  12. 2024Gemma 2: Improving Open Language Models at a Practical Sizecited by 1,998
23 more
  1. 2024Mixtral of Expertscited by 1,823
  2. 2025Search-R1: Training LLMs to Reason and Leverage Search Engines with Reinforcement Learningcited by 1,444
  3. 2020Constructing A Multi-hop QA Dataset for Comprehensive Evaluation of Reasoning Stepscited by 1,282
  4. 2022Text Embeddings by Weakly-Supervised Contrastive Pre-trainingcited by 1,253
  5. 2023A Survey on Evaluation of Large Language Modelscited by 1,242
  6. 2022When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memoriescited by 1,128
  7. 2022Measuring and Narrowing the Compositionality Gap in Language Modelscited by 1,075
  8. 2024SimPO: Simple Preference Optimization with a Reference-Free Rewardcited by 963
  9. 2021MuSiQue: Multihop Questions via Single-hop Question Compositioncited by 916
  10. 2023FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performancecited by 853
  11. 2024RouteLLM: Learning to Route LLMs with Preference Datacited by 682
  12. 2023A Comprehensive Overview of Large Language Modelscited by 648
  13. 2023RLAIF vs. RLHF: Scaling Reinforcement Learning from Human Feedback with AI Feedbackcited by 625
  14. 2023RRHF: Rank Responses to Align Language Models with Human Feedback without tearscited by 463
  15. 2024Hybrid LLM: Cost-Efficient and Quality-Aware Query Routingcited by 372
  16. 2024RouterBench: A Benchmark for Multi-LLM Routing Systemcited by 238
  17. 2024GraphRouter: A Graph-based Router for LLM Selectionscited by 165
  18. 2023Fly-Swat or Cannon? Cost-Effective Language Model Choice via Meta-Modelingcited by 95
  19. 2024ChatQA: Surpassing GPT-4 on Conversational QA and RAGcited by 90
  20. 2024RouterDC: Query-Based Router by Dual Contrastive Learning for Assembling Large Language Modelscited by 88
  21. 2024TensorOpera Router: A Multi-Model Router for Efficient LLM Inferencecited by 60
  22. 2023Identifying and Mitigating Vulnerabilities in LLM-Integrated Applicationscited by 54
  23. 2024Cost-Effective Online Multi-LLM Selection with Versatile Reward Modelscited by 39

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 8; 2026: 64.

72 papers.

Sort:

By area:

  1. 2025Don't Overthink It: A Survey of Efficient R1-style Large Reasoning Modelscited by 37
  2. 2026MAS-Orchestra: Understanding and Improving Multi-Agent Reasoning Through Holistic Orchestration and Controlled Benchmarkscited by 32
  3. 2026SkillOrchestra: Learning to Route Agents via Skill Transfercited by 23
  4. 2025RouterArena: An Open Platform for Comprehensive Comparison of LLM Routerscited by 22
  5. 2026Dynamic Model Routing and Cascading for Efficient LLM Inference: A Surveycited by 19
  6. 2025xRouter: Training Cost-Aware LLMs Orchestration System via Reinforcement Learningcited by 17
  7. 2026EvolveRouter: Co-Evolving Routing and Prompt for Multi-Agent Question Answeringcited by 15
  8. 2026Learning Query-Aware Budget-Tier Routing for Runtime Agent Memorycited by 13
  9. 2026GraphPlanner: Graph Memory-Augmented Agentic Routing for Multi-Agent LLMscited by 12
  10. 2026SWE-Router: Routing in Multi-turn Agentic Software Engineering Taskscited by 11
  11. 2026Budget-Aware Agentic Routing via Boundary-Guided Trainingcited by 10
  12. 2026Workflow-R1: Group Sub-sequence Policy Optimization for Multi-turn Workflow Constructioncited by 8
38 more
  1. 2026The Replay Gap: Static Evaluation of Model Switching in LLM Agents Scores the Wrong Worldcited by 8
  2. 2026RACER: Risk-Aware Calibrated Efficient Routing for Large Language Modelscited by 7
  3. 2026vLLM Semantic Router: Signal Driven Decision Routing for Mixture-of-Modality Modelscited by 6
  4. 2026Uno-Orchestra: Parsimonious Agent Routing via Selective Delegationcited by 6
  5. 2025Bridging On-Device and Cloud LLMs for Collaborative Reasoning: A Unified Methodology for Local Routing and Post-Trainingcited by 5
  6. 2025Beyond the limitation of a single query: Train your LLM for query expansion with Reinforcement Learningcited by 5
  7. 2026GenRouter: Unified Workflow Routing for Agentic Image Generationcited by 5
  8. 2025PersonalizedRouter: Personalized LLM Routing via Graph-based User Preference Modelingcited by 4
  9. 2026PROTEUS: SLA-Aware Routing via Lagrangian RL for Multi-LLM Serving Systemscited by 4
  10. 2026Models Under SCOPE: Scalable and Controllable Routing via Pre-hoc Reasoningcited by 4
  11. 2026Chimera: Latency- and Performance-Aware Multi-agent Serving for Heterogeneous LLMscited by 4
  12. 2026Not All Turns Are Equally Hard: Adaptive Thinking Budgets For Efficient Multi-Turn Reasoning in Agentscited by 4
  13. 2026From Myopic Selection to Long-Horizon Awareness: Sequential LLM Routing for Multi-Turn Dialoguecited by 4
  14. 2026ATLAS: Agentic Test-time Learning-to-Allocate Scalingcited by 4
  15. 2026LLMRouter: Unified Infrastructure for Developing, Evaluating, and Deploying LLM Routerscited by 4
  16. 2025The Law of Multi-Model Collaboration: Scaling Limits of Model Ensembling for Large Language Modelscited by 3
  17. 2026System 1&2 Synergy via Dynamic Model Interpolationcited by 3
  18. 2026Federate the Router: Learning Language Model Routers with Sparse and Decentralized Evaluationscited by 3
  19. 2026Learning Agent Routing From Early Experiencecited by 3
  20. 2026DocArena: Turning Raw Documents into Controllable Training Environments for Document Search Agentscited by 3
  21. 2025GTAlign: Game-Theoretic Alignment of LLM Assistants for Social Welfarecited by 2
  22. 2026Dynamic Mixed-Precision Routing for Efficient Multi-step LLM Interactioncited by 2
  23. 2026No Single Best Model for Diversity: Learning a Router for Sample Diversitycited by 2
  24. 2026Local-First Intelligence via Proactive SLM Collaborating with Feedback-Providing LLMcited by 2
  25. 2026SeqRoute: Global Budget-Aware Sequential LLM Routing via Offline Reinforcement Learningcited by 2
  26. 2026Reward Modeling for Multi-Agent Orchestrationcited by 2
  27. 2026TSRouter: Dynamic Modality-Model Selection for Time Series Reasoningcited by 2
  28. 2026Markets, Not Planners: Decentralized Orchestration of LLM Agents with Private Informationcited by 2
  29. 2026IDSelect: A RL-Based Cost-Aware Selection Agent for Video-based Multi-Modal Person Recognitioncited by 1
  30. 2026The Workload-Router-Pool Architecture for LLM Inference Optimization: A Vision Paper from the vLLM Semantic Router Projectcited by 1
  31. 2026MemBoost: A Memory-Boosted Framework for Cost-Aware LLM Inferencecited by 1
  32. 2026SEAR: Schema-Based Evaluation and Routing for LLM Gatewayscited by 1
  33. 2026Privacy-Preserving LLMs Routingcited by 1
  34. 2026AgentSearchBench: A Benchmark for AI Agent Search in the Wildcited by 1
  35. 2026ModelLens: Finding the Best for Your Task from Myriads of Modelscited by 1
  36. 2026The Routing Plateau: Understanding the Accuracy Limits of LLM Routerscited by 1
  37. 2026Resample or Reroute? Recoverable Stopping Debt Without Identified Action Selectioncited by 1
  38. 2026MERA: Model Evolution and Routing with Skill Adaptation for Agentic Systems at Scalecited by 1

50 most cited of 72 All 72 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026ReCal: Reward Calibration for RL-based LLM Routing
  2. 2025HierRouter: Coordinated Routing of Specialized Large Language Models via Reinforcement Learning
  3. 2026Learning to Route LLMs from Implicit Cost-Performance Preferences via Meta-Learning
  4. 2024Exploring Domain Robust Lightweight Reward Models based on Router Mechanism
  5. 2026R2-Router: A New Paradigm for LLM Routing with Reasoning
  6. 2025R1-Searcher: Incentivizing the Search Capability in LLMs via Reinforcement Learning
  7. 2025Learning to Route LLMs from Bandit Feedback: One Policy, Many Trade-offs
  8. 2025Route-and-Reason: Scaling Large Language Model Reasoning with Reinforced Model Router