mlsanityData to 14 Oct 2026

Astrolabe: Balancing Load in LLM Serving with Randomized Prediction-Guided Scheduling

arXiv 2508.03611 ↗First posted 5 Aug 2025cs.DC cs.AIcited by 2

On the map: Efficient LLM inference and serving → Serving systems and request scheduling

Abstract (arXiv)

This paper presents Astrolabe, a randomized prediction-guided scheduler for one-shot request dispatch in multi-instance large language model (LLM) serving. Astrolabe improves load balancing without relying on migration-based rebalancing, whose KV-cache transfers can introduce substantial overhead and network contention under high load. It combines response-length estimation, per-instance simulation-based latency prediction, and a power-of-two-choices dispatch policy to balance load while avoiding request herding. On the default Llama-2-7B/ShareGPT setup, Astrolabe matches the SLO capacity of the best load-aware baseline (31.6 versus 31.5 QPS), while reducing mean time-to-first-token (TTFT) by 8 to 36 percent, P99 TTFT by 16 to 77 percent, and mean end-to-end (E2E) latency by up to 5.6 percent, with approximately six times fewer preemptions once capacity is reached. Under configuration shifts, Astrolabe improves SLO capacity by up to 6 percent on Qwen2-7B and 7.1 percent under tight batching, reduces mean E2E latency by 6 to 9 percent under bursty arrivals, and achieves an approximately 2.8-fold reduction in per-predictor CPU usage relative to full fanout. With migration enabled on A100 GPUs, Astrolabe outperforms Llumnix by up to 2.6 times in throughput while achieving orders-of-magnitude lower per-token latency at saturation.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Placed in Serving systems and request scheduling, one of 12 directions in Efficient LLM inference and serving. Builds on 40 of its 72 references that are in the map. First posted on arXiv 5 Aug 2025.

Builds on

40 of 72 references.

Closest ancestors · 10

Same direction, newest first

  1. 2025Prefill-Decode Aggregation or Disaggregation? Unifying Both for Goodput-Optimized LLM Serving4 Aug
  2. 2025Past-Future Scheduler for LLM Serving under SLA Guarantees14 Jul
  3. 2025ELIS: Efficient LLM Iterative Scheduling System with Response Length Predictor14 May
  4. 2024SCOOT: SLO-Oriented Performance Tuning for LLM Inference Engines8 Aug
  5. 2024Queue management for slo-oriented large language model serving5 Jun
  6. 2024Llumnix: Dynamic Scheduling for Large Language Model Serving5 Jun
  7. 2024Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve4 Mar
  8. 2024BurstGPT: A Real-world Workload Dataset to Optimize LLM Serving Systems31 Jan
  9. 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving18 Jan
  10. 2023Response Length Perception and Sequence Scheduling: An LLM-Empowered LLM Inference Pipeline22 May

Other references · 30

  1. 2017Attention Is All You Needcited by 60,361
  2. 2023GPT-4 Technical Reportcited by 18,396
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  4. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  5. 2025Qwen3 Technical Reportcited by 8,820
  6. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  7. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,598
  8. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,141
  9. 2024RouteLLM: Learning to Route LLMs with Preference Datacited by 682
  10. 2024A Survey on Large Language Models for Code Generationcited by 658
  11. 2021Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LMcited by 632
  12. 2022DeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scalecited by 407
18 more
  1. 2025FlashInfer: Efficient and Customizable Attention Engine for LLM Inference Servingcited by 258
  2. 2024GEAR: An Efficient KV Cache Compression Recipe for Near-Lossless Generative Inference of LLMcited by 158
  3. 2024Inference without Interference: Disaggregate LLM Inference for Mixed Downstream Workloadscited by 121
  4. 2024Vidur: A Large-Scale Simulation Framework For LLM Inferencecited by 117
  5. 2024DynamoLLM: Designing LLM Inference Clusters for Performance and Energy Efficiencycited by 112
  6. 2022Prompting Is Programming: A Query Language for Large Language Modelscited by 110
  7. 2023A Confederacy of Models: a Comprehensive Evaluation of LLMs on Creative Writingcited by 99
  8. 2023S3: Increasing GPU Utilization during Generative Inference for Higher Throughputcited by 81
  9. 2020TurboTransformers: An Efficient GPU Serving System For Transformer Modelscited by 71
  10. 2025JSONSchemaBench: A Rigorous Benchmark of Structured Outputs for Language Modelscited by 71
  11. 2025BEST-Route: Adaptive LLM Routing with Test-Time Optimal Computecited by 67
  12. 2024From LLM to NMT: Advancing Low-Resource Machine Translation with Claudecited by 58
  13. 2024POD-Attention: Unlocking Full Prefill-Decode Overlap for Faster LLM Inferencecited by 47
  14. 2025SLOs-Serve: Optimized Serving of Multi-SLO LLMscited by 19
  15. 2024LAVA: Lifetime-Aware VM Allocation with Learned Distributions and Adaptation to Mispredictionscited by 4
  16. 2025AccelGen: Heterogeneous SLO-Guaranteed High-Throughput LLM Inference Serving for Diverse Applicationscited by 2
  17. 2019Dynamic task scheduling in computing cluster environmentscited by 1
  18. 2024Navigating the Docker Ecosystem: A Comprehensive Taxonomy and Surveycited by 1

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 2.

2 papers.

  1. 2026RouteBalance: Fused Model Routing and Load Balancing for Heterogeneous LLM Servingcited by 1
  2. 2026Mind the Delay: Latency-aware Query Routing for Heterogeneous LLM poolscited by 1

Similar, not cited

Close in wording, no citation either way

  1. 2024Intelligent Router for LLM Workloads: Improving Performance Through Workload-Aware Load Balancing
  2. 2025SkyWalker: A Locality-Aware Cross-Region Load Balancer for LLM Inference
  3. 2026Beyond Binary Priorities: Multi-Tier SLA Scheduling for Large Language Model Serving
  4. 2026MAPS: Memory-Aware Predictive Scheduling Framework for Large Language Model Serving
  5. 2025Niyama : Breaking the Silos of LLM Inference Serving
  6. 2026Cascade: Exploiting SLO-Aware latency budget for fair and high goodput LLM inference serving
  7. 2026Tackling the Data-Parallel Load Balancing Bottleneck in LLM Serving: Practical Online Routing at Scale
  8. 2026Taming Request Imbalance: SLO-Aware Scheduling for Disaggregated LLM Inference