mlsanityData to 7 Oct 2026

SeqRoute: Global Budget-Aware Sequential LLM Routing via Offline Reinforcement Learning

arXiv 2605.25424 ↗First posted 25 May 2026cs.LG cs.AIcited by 2

On the map: Offline RL and imitation learning from data → Offline RL algorithms

Abstract (arXiv)

Existing LLM routing frameworks treat queries as independent events, neglecting the sequential nature of real-world user sessions constrained by global computational budgets. This mismatch inevitably leads to budget bankruptcy: myopic routing policies exhaust resources on early interactions, forcing subsequent and often more complex queries onto inadequate models. We introduce SeqRoute, a framework that formulates multi-turn routing as a finite-horizon Markov Decision Process and solves it via offline reinforcement learning. By incorporating the remaining budget into the state space and training with Conservative Q-Learning (CQL), SeqRoute learns delayed gratification to strategically preserve resources for high-stakes turns later in the session. To overcome data starvation, we propose Hindsight Budget Relabeling (HBR). This technique retrospectively simulates historical trajectories under diverse hypothetical budgets, expanding 10,000 raw sessions into 2.38 million transitions enriched with critical bankruptcy signals. At deployment, a dynamic λ-sweep mechanism enables zero-shot navigation of the cost-quality Pareto frontier without retraining. Extensive evaluations demonstrate that SeqRoute reduces operational costs by 6.0-73.5% while maintaining or improving quality, and suppresses bankruptcy rates to under 1%, strictly dominating behavior cloning, budget-aware heuristics, and static baselines across the entire Pareto frontier.

Key facts. Cited by 2 papers in the map (data through 7 Oct 2026). Placed in Offline RL algorithms, one of 13 directions in Offline RL and imitation learning from data. Builds on 29 of its 39 references that are in the map. First posted on arXiv 25 May 2026.

Builds on

29 of 39 references.

Closest ancestors · 3

Same direction, newest first

  1. 2021Offline Reinforcement Learning with Implicit Q-Learning12 Oct
  2. 2020Conservative Q-Learning for Offline Reinforcement Learning8 Jun
  3. 2020Offline Reinforcement Learning: Tutorial, Review, and Perspectives on Open Problems4 May

Other references · 26

  1. 2017Hindsight Experience Replaycited by 1,096
  2. 2023FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performancecited by 791
  3. 2024RouteLLM: Learning to Route LLMs with Preference Datacited by 633
  4. 2024Hybrid LLM: Cost-Efficient and Quality-Aware Query Routingcited by 354
  5. 2024RouterBench: A Benchmark for Multi-LLM Routing Systemcited by 222
  6. 2024GraphRouter: A Graph-based Router for LLM Selectionscited by 154
  7. 2024Learning to Route LLMs with Confidence Tokenscited by 66
  8. 2025Router-R1: Teaching LLMs Multi-Round Routing and Aggregation via Reinforcement Learningcited by 64
  9. 2025BEST-Route: Adaptive LLM Routing with Test-Time Optimal Computecited by 63
  10. 2025RouterEval: A Comprehensive Benchmark for Routing LLMs to Explore Model-level Scaling Up in LLMscited by 58
  11. 2025MixLLM: Dynamic Routing in Mixed Large Language Modelscited by 56
  12. 2025SelfBudgeter: Adaptive Token Allocation for Efficient LLM Reasoningcited by 47
14 more
  1. 2025AgentRouter: A Knowledge-Graph-Guided LLM Router for Collaborative Multi-Agent Question Answeringcited by 29
  2. 2025BudgetThinker: Empowering Budget-aware LLM Reasoning with Control Tokenscited by 27
  3. 2025ICL-Router: In-Context Learned Model Representations for LLM Routingcited by 26
  4. 2025Route to Reason: Adaptive Routing for LLM and Reasoning Strategy Selectioncited by 23
  5. 2025CP-Router: An Uncertainty-Aware Router Between LLM and LRMcited by 19
  6. 2025RouterArena: An Open Platform for Comprehensive Comparison of LLM Routerscited by 19
  7. 2025RadialRouter: Structured Representation for Efficient and Robust Large Language Models Routingcited by 11
  8. 2026When Routing Collapses: On the Degenerate Convergence of LLM Routerscited by 10
  9. 2025RAGRouter: Learning to Route Queries to Multiple Retrieval-Augmented Language Modelscited by 8
  10. 2026Budget-Aware Agentic Routing via Boundary-Guided Trainingcited by 8
  11. 2026AdaptEvolve: Improving Efficiency of Evolutionary AI Agents through Adaptive Model Selectioncited by 6
  12. 2025GMTRouter: Personalized LLM Router over Multi-turn User Interactionscited by 5
  13. 2026PROTEUS: SLA-Aware Routing via Lagrangian RL for Multi-LLM Serving Systemscited by 4
  14. 2026RouteMoA: Dynamic Routing without Pre-Inference Boosts Efficient Mixture-of-Agentscited by 3

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026Control the Harness, Control the Cost: Routing and Governing AI Coding Agents in the Enterprisecited by 1
  2. 2026HybridInfer: Thermal-Aware Reinforcement-Learning Tier Routing for On-Device, Edge, and Cloud LLM Inferencecited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025xRouter: Training Cost-Aware LLMs Orchestration System via Reinforcement Learning
  2. 2025HierRouter: Coordinated Routing of Specialized Large Language Models via Reinforcement Learning
  3. 2026Policy-Guided Stepwise Model Routing for Cost-Effective Reasoning
  4. 2026CounterRoute: Self-Routed Reasoning via Hierarchical Counterfactual Credit Assignment
  5. 2026Learning Query-Aware Budget-Tier Routing for Runtime Agent Memory
  6. 2026Routing Dense Layouts with History-Aware Offline Reinforcement Learning using LSTM
  7. 2026From Sampled Outcomes to Capability Distributions: Rethinking Supervision for LLM Routing
  8. 2026TRIP-Bench: A Benchmark for Long-Horizon Interactive Agents in Real-World Scenarios