mlsanityData to 7 Oct 2026

TokenRouter: Efficient Serving System for Token-Level LLM Routing

arXiv 2610.12242 ↗First posted 8 Oct 2026cs.CLcited by 0

Not on the map. Why?

Abstract (arXiv)

Large language model (LLM) routing distributes inference work across different models, advancing the cost-quality Pareto frontier of LLM serving. While coarse-grained routing at the session or query level has been widely adopted in production systems, recent algorithmic work shows that fine-grained token-level routing can yield substantial efficiency and quality gains. However, efficiently serving token-level routed inference poses significant challenges to existing systems. Built on single-LLM assumptions, current systems suffer from severe step desynchronization and frequent batch admission delays under token-level routing, and they also impose high implementation complexity on developers. To address these challenges, we design TokenRouter, an efficient and developer-friendly serving system for token-level routed LLM inference. TokenRouter follows the principle of request-centric programming, model-centric execution: developers describe routing logic from the perspective of a single request, while the runtime launches a subserver for each LLM and dispatches requests asynchronously. Each subserver employs a delayed-batching scheduler, whose optimal hyperparameters are derived from a mathematical throughput model of the system. Across diverse routing algorithms, workloads, and model pairs, TokenRouter achieves 2.01-64.15x higher decoding throughput than existing systems, substantially advancing the serving efficiency of token-level LLM routing. Our code is available at https://github.com/thu-nics/TokenRouter.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 29 of its 38 references that are in the map. First posted on arXiv 8 Oct 2026.

Builds on

29 of 38 references.

References · 29

  1. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  2. 2025Qwen3 Technical Reportcited by 8,195
  3. 2022Fast Inference from Transformers via Speculative Decodingcited by 1,615
  4. 2025OpenAI GPT-5 System Cardcited by 910
  5. 2024Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Headscited by 821
  6. 2023FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performancecited by 791
  7. 2024RouteLLM: Learning to Route LLMs with Preference Datacited by 633
  8. 2024EAGLE: Speculative Sampling Requires Rethinking Feature Uncertaintycited by 564
  9. 2024Mixture-of-Agents Enhances Large Language Model Capabilitiescited by 420
  10. 2024Hybrid LLM: Cost-Efficient and Quality-Aware Query Routingcited by 354
  11. 2025SWE-smith: Scaling Data for Software Engineering Agentscited by 255
  12. 2024RouterBench: A Benchmark for Multi-LLM Routing Systemcited by 222
17 more
  1. 2024GraphRouter: A Graph-based Router for LLM Selectionscited by 154
  2. 2023Routing to the Expert: Efficient Reward-guided Ensemble of Large Language Modelscited by 141
  3. 2025RouterEval: A Comprehensive Benchmark for Routing LLMs to Explore Model-level Scaling Up in LLMscited by 58
  4. 2025Cache-to-Cache: Direct Semantic Communication Between Large Language Modelscited by 54
  5. 2025ToolOrchestra: Elevating Intelligence via Efficient Model and Tool Orchestrationcited by 44
  6. 2024Eagle: Efficient Training-Free Router for Multi-LLM Inferencecited by 34
  7. 2025CITER: Collaborative Inference for Efficient Large Language Model Decoding with Token-Level Routingcited by 29
  8. 2025R2R: Efficiently Navigating Divergent Reasoning Paths with Small-Large Model Token Routingcited by 28
  9. 2026GlimpRouter: Efficient Collaborative Inference by Glimpsing One Token of Thoughtscited by 16
  10. 2025Think-at-Hard: Dynamic Looped Transformers for Improved Reasoningcited by 12
  11. 2025SplitReason: Learning To Offload Reasoningcited by 11
  12. 2024Collaborative decoding of critical tokens for boosting factuality of large language modelscited by 9
  13. 2026Token-Level LLM Collaboration via FusionRoutecited by 9
  14. 2025R-Stitch: Dynamic Trajectory Stitching for Efficient Reasoningcited by 7
  15. 2026RelayLLM: Efficient Reasoning via Collaborative Decodingcited by 7
  16. 2026TARo: Token-level Adaptive Routing for LLM Test-time Alignmentcited by 1
  17. 2026Rethinking LLM Ensembling from the Perspective of Mixture Modelscited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Optimal Scheduling Algorithms for LLM Inference: Theory and Practice
  2. 2026Position: LLM Serving Needs Mathematical Optimization and Algorithmic Foundations, Not Just Heuristics
  3. 2026Accuracy Is Speed: Towards Long-Context-Aware Routing for Distributed LLM Serving
  4. 2026Token-Budget-Aware Pool Routing for Cost-Efficient LLM Inference
  5. 2025Niyama : Breaking the Silos of LLM Inference Serving
  6. 2026Cascade: Exploiting SLO-Aware latency budget for fair and high goodput LLM inference serving
  7. 2026AlignedServe: Orchestrating Prefix-aware Batching to Build a High-throughput and Computing-efficient LLM Serving System
  8. 2025Apt-Serve: Adaptive Request Scheduling on Hybrid Cache for Scalable LLM Inference Serving