# HierRouter: Coordinated Routing of Specialized Large Language Models via Reinforcement Learning

arXiv: https://arxiv.org/abs/2511.09873 · HTML: https://mlsanity.com/paper/2511.09873/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 13 Nov 2025.

## Built on by (most cited first, 3 of 3)

- [ODAR: Principled Adaptive Routing for LLM Reasoning via Active Inference](https://mlsanity.com/paper/2602.23681/) (2026)
- [Do LLM-derived graph priors improve multi-agent coordination?](https://mlsanity.com/paper/2604.17191/) (2026)
- [IR3DE: A Linear Router for Large Language Models](https://mlsanity.com/paper/2606.06098/) (2026)

## Abstract (arXiv)

Large Language Models (LLMs) deliver state-of-the-art performance across many tasks but impose high computational and memory costs, limiting their deployment in resource-constrained or real-time settings. To address this, we propose HierRouter, a hierarchical routing approach that dynamically assembles inference pipelines from a pool of specialized, lightweight language models. Formulated as a finite-horizon Markov Decision Process (MDP), our approach trains a Proximal Policy Optimization (PPO)-based reinforcement learning agent to iteratively select which models to invoke at each stage of multi-hop inference. The agent conditions on the evolving context and accumulated cost to make context-aware routing decisions. Experiments with three open-source candidate LLMs across six benchmarks, including QA, code generation, and mathematical reasoning, show that HierRouter improves response quality by up to 2.4x compared to using individual models independently, while incurring only a minimal additional inference cost on average. These results highlight the promise of hierarchical routing for cost-efficient, high-performance LLM inference. All codes can be found here https://github.com/ Nikunj-Gupta/hierouter.