# SeqRoute: Global Budget-Aware Sequential LLM Routing via Offline Reinforcement Learning

arXiv: https://arxiv.org/abs/2605.25424 · HTML: https://mlsanity.com/paper/2605.25424/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data.
- Builds on 29 of its 39 references that are in the map.
- First posted on arXiv 25 May 2026.

## Placed in

- Offline RL and imitation learning from data / [Offline RL algorithms](https://mlsanity.com/offlinerl/offline-rl-algorithms/)

## Builds on (most cited first, 29 of 29)

- [Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems](https://mlsanity.com/paper/2005.01643/) (2020)
- [Conservative Q-Learning for Offline Reinforcement Learning](https://mlsanity.com/paper/2006.04779/) (2020)
- [Offline Reinforcement Learning with Implicit Q-Learning](https://mlsanity.com/paper/2110.06169/) (2021)
- [Hindsight Experience Replay](https://mlsanity.com/paper/1707.01495/) (2017)
- [FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performance](https://mlsanity.com/paper/2305.05176/) (2023)
- [RouteLLM: Learning to Route LLMs with Preference Data](https://mlsanity.com/paper/2406.18665/) (2024)
- [Hybrid LLM: Cost-Efficient and Quality-Aware Query Routing](https://mlsanity.com/paper/2404.14618/) (2024)
- [RouterBench: A Benchmark for Multi-LLM Routing System](https://mlsanity.com/paper/2403.12031/) (2024)
- [GraphRouter: A Graph-based Router for LLM Selections](https://mlsanity.com/paper/2410.03834/) (2024)
- [Router-R1: Teaching LLMs Multi-Round Routing and Aggregation via Reinforcement Learning](https://mlsanity.com/paper/2506.09033/) (2025)
- [Learning to Route LLMs with Confidence Tokens](https://mlsanity.com/paper/2410.13284/) (2024)
- [BEST-Route: Adaptive LLM Routing with Test-Time Optimal Compute](https://mlsanity.com/paper/2506.22716/) (2025)
- [RouterEval: A Comprehensive Benchmark for Routing LLMs to Explore Model-level Scaling Up in LLMs](https://mlsanity.com/paper/2503.10657/) (2025)
- [MixLLM: Dynamic Routing in Mixed Large Language Models](https://mlsanity.com/paper/2502.18482/) (2025)
- [SelfBudgeter: Adaptive Token Allocation for Efficient LLM Reasoning](https://mlsanity.com/paper/2505.11274/) (2025)
- [AgentRouter: A Knowledge-Graph-Guided LLM Router for Collaborative Multi-Agent Question Answering](https://mlsanity.com/paper/2510.05445/) (2025)
- [BudgetThinker: Empowering Budget-aware LLM Reasoning with Control Tokens](https://mlsanity.com/paper/2508.17196/) (2025)
- [ICL-Router: In-Context Learned Model Representations for LLM Routing](https://mlsanity.com/paper/2510.09719/) (2025)
- [Route to Reason: Adaptive Routing for LLM and Reasoning Strategy Selection](https://mlsanity.com/paper/2505.19435/) (2025)
- [RouterArena: An Open Platform for Comprehensive Comparison of LLM Routers](https://mlsanity.com/paper/2510.00202/) (2025)
- [CP-Router: An Uncertainty-Aware Router Between LLM and LRM](https://mlsanity.com/paper/2505.19970/) (2025)
- [RadialRouter: Structured Representation for Efficient and Robust Large Language Models Routing](https://mlsanity.com/paper/2506.03880/) (2025)
- [When Routing Collapses: On the Degenerate Convergence of LLM Routers](https://mlsanity.com/paper/2602.03478/) (2026)
- [Budget-Aware Agentic Routing via Boundary-Guided Training](https://mlsanity.com/paper/2602.21227/) (2026)
- [RAGRouter: Learning to Route Queries to Multiple Retrieval-Augmented Language Models](https://mlsanity.com/paper/2505.23052/) (2025)
- [AdaptEvolve: Improving Efficiency of Evolutionary AI Agents through Adaptive Model Selection](https://mlsanity.com/paper/2602.11931/) (2026)
- [GMTRouter: Personalized LLM Router over Multi-turn User Interactions](https://mlsanity.com/paper/2511.08590/) (2025)
- [PROTEUS: SLA-Aware Routing via Lagrangian RL for Multi-LLM Serving Systems](https://mlsanity.com/paper/2601.19402/) (2026)
- [RouteMoA: Dynamic Routing without Pre-Inference Boosts Efficient Mixture-of-Agents](https://mlsanity.com/paper/2601.18130/) (2026)

## Built on by (most cited first, 2 of 2)

- [Control the Harness, Control the Cost: Routing and Governing AI Coding Agents in the Enterprise](https://mlsanity.com/paper/2609.28919/) (2026)
- [HybridInfer: Thermal-Aware Reinforcement-Learning Tier Routing for On-Device, Edge, and Cloud LLM Inference](https://mlsanity.com/paper/2609.30270/) (2026)

## Abstract (arXiv)

Existing LLM routing frameworks treat queries as independent events, neglecting the sequential nature of real-world user sessions constrained by global computational budgets. This mismatch inevitably leads to budget bankruptcy: myopic routing policies exhaust resources on early interactions, forcing subsequent and often more complex queries onto inadequate models. We introduce SeqRoute, a framework that formulates multi-turn routing as a finite-horizon Markov Decision Process and solves it via offline reinforcement learning. By incorporating the remaining budget into the state space and training with Conservative Q-Learning (CQL), SeqRoute learns delayed gratification to strategically preserve resources for high-stakes turns later in the session. To overcome data starvation, we propose Hindsight Budget Relabeling (HBR). This technique retrospectively simulates historical trajectories under diverse hypothetical budgets, expanding 10,000 raw sessions into 2.38 million transitions enriched with critical bankruptcy signals. At deployment, a dynamic λ-sweep mechanism enables zero-shot navigation of the cost-quality Pareto frontier without retraining. Extensive evaluations demonstrate that SeqRoute reduces operational costs by 6.0-73.5% while maintaining or improving quality, and suppresses bankruptcy rates to under 1%, strictly dominating behavior cloning, budget-aware heuristics, and static baselines across the entire Pareto frontier.