mlsanityData to 14 Oct 2026

Lodestar: An Online-Learning LLM Inference Router

arXiv 2606.00946 ↗First posted 31 May 2026cs.DC cs.AI cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

Efficiently serving large language model (LLM) inference tasks is crucial both for user-perceived latency such as time-to-first-token (TTFT) and for GPU utilization. However, LLM request routing, that is, assigning each inference request to a GPU instance, is particularly challenging: execution is highly input-dependent; batching and KV-cache reuse create strong cross-request coupling; and latency responds nonlinearly to context length, model/engine settings, and heterogeneous accelerators. As a result, simple traditional load balancing algorithms, and even heuristics tailored for LLM inference, fail to achieve good performance. We present Lodestar, a novel learning-based request routing system for distributed GPU clusters. Lodestar continuously collects a snapshot of the cluster at per-request level, including real-time instance state, request characteristics, and observed performance, and trains an online reward predictor that it uses to route inference requests to the instance that will maximize given reward (e.g., minimizing TTFT). Lodestar is cloud-native and works seamlessly with existing serving stacks (vLLM). With continuous online adaptation to changing workloads and infrastructure conditions, Lodestar achieves 1.41x lower average TTFT and 1.47x lower P99 TTFT on average (up to 2.15x/1.86x on homogeneous and 4.38x/4.42x on heterogeneous clusters) compared to a state-of-the-art prefix cache and load-aware heuristic, and learns these efficient routing strategies within about 5 minutes, based on experiments in a public cloud GPU cluster.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 33 of its 62 references that are in the map. First posted on arXiv 31 May 2026.

Builds on

33 of 62 references.

References · 33

  1. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  2. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  3. 2022Fast Inference from Transformers via Speculative Decodingcited by 1,653
  4. 2024RULER: What's the Real Context Size of Your Long-Context Language Models?cited by 1,346
  5. 2023FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performancecited by 853
  6. 2024Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Headscited by 843
  7. 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Servingcited by 617
  8. 2024EAGLE: Speculative Sampling Requires Rethinking Feature Uncertaintycited by 578
  9. 2024Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Servecited by 526
  10. 2023LMSYS-Chat-1M: A Large-Scale Real-World LLM Conversation Datasetcited by 461
  11. 2022DeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scalecited by 407
  12. 2023Prompt Cache: Modular Attention Reuse for Low-Latency Inferencecited by 268
21 more
  1. 2024InfiniGen: Efficient Generative Inference of Large Language Models with Dynamic KV Cache Managementcited by 245
  2. 2023Fast Distributed Inference Serving for Large Language Modelscited by 145
  3. 2025LMCache: An Efficient KV Cache Layer for Enterprise-Scale LLM Inferencecited by 126
  4. 2020Serving DNNs like Clockwork: Performance Predictability from the Bottom Upcited by 120
  5. 2021GCR: Gradient Coreset Based Replay Buffer Selection For Continual Learningcited by 95
  6. 2024NanoFlow: Towards Optimal Large Language Model Serving Throughputcited by 90
  7. 2024Efficient LLM Scheduling by Learning to Rankcited by 85
  8. 2020TurboTransformers: An Efficient GPU Serving System For Transformer Modelscited by 71
  9. 2020Heterogeneity-Aware Cluster Scheduling Policies for Deep Learning Workloadscited by 56
  10. 2024Helix: Serving Large Language Models over Heterogeneous GPUs and Network via Max-Flowcited by 55
  11. 2025KVCache Cache in the Wild: Characterizing and Optimizing KVCache Cache at a Large Cloud Providercited by 41
  12. 2025Prism: Cost-Efficient Multi-LLM Serving via GPU Memory Ballooningcited by 24
  13. 2025PrefillOnly: An Inference Engine for Prefill-only Workloads in Large Language Model Applicationscited by 21
  14. 2025JITServe: SLO-aware LLM Serving with Imprecise Request Informationcited by 17
  15. 2026Prefill-as-a-Service: KVCache of Next-Generation Models Could Go Cross-Datacentercited by 11
  16. 2025Hetis: Serving LLMs in Heterogeneous GPU Clusters with Fine-grained and Dynamic Parallelismcited by 9
  17. 2025DynaServe: Unified and Elastic Execution for Dynamic Disaggregated LLM Servingcited by 6
  18. 2025Towards Efficient and Practical GPU Multitasking in the Era of LLMcited by 6
  19. 2025DeServe: Towards Affordable Offline LLM Inference via Decentralizationcited by 4
  20. 2025LLMRank: Understanding LLM Strengths for Model Routingcited by 4
  21. 2023Load is not what you should balance: Introducing Prequalcited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026RequestRouter: Request-Boundary Routing for Efficient Single-GPU LLM Inference
  2. 2025Efficient LLM Inference with Activation Checkpointing and Hybrid Caching
  3. 2024AQUA: Network-Accelerated Memory Offloading for LLMs in Scale-Up GPU Domains
  4. 2024NEO: Saving GPU Memory Crisis with CPU Offloading for Online LLM Inference
  5. 2025Parallax: Efficient LLM Inference Service over Decentralized Environment
  6. 2026Scaling LLM Inference Beyond Amdahl`s Limits via Eliminating Non-Scalable Overheads ↗
  7. 2025APEX: Asynchronous Parallel CPU-GPU Execution for Online LLM Inference on Constrained GPUs
  8. 2024KVDirect: Distributed Disaggregated LLM Inference