mlsanityData to 14 Oct 2026

ExaServe: Large-Scale LLM Serving on Exascale HPC Systems

arXiv 2609.10812 ↗First posted 9 Sep 2026cs.DCcited by 0

On the map: Efficient LLM inference and serving → Serving systems and request scheduling

Abstract (arXiv)

Cloud-native LLM serving frameworks have made deployment routine in data centers, yet deploying them on leadership-class supercomputers remains an engineering challenge requiring scheduler integration, MPI launch, accelerator selection, node-local weight staging, and platform-specific patches. We present ExaServe, a pip-installable framework that transforms a declarative YAML specification into a reproducible large-scale LLM serving deployment. Using ExaServe, we deploy LLM serving on ALCF Aurora from 1 to 256 nodes (3072 vLLM replicas). Non-streaming inference scales nearly linearly to 256 nodes, reaching 27.1k requests/s (3.8M tokens/s). Token streaming scales differently: a centralized proxy plateaus at about 4.7k requests/s despite the model servers remaining within the service-level objective. We also identify an O(N^2) Ray Serve control-plane bottleneck that increases cluster bring-up to roughly 30 minutes at 256 nodes. ExaServe provides a practical, reproducible deployment path while exposing key barriers to future exascale LLM serving.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Serving systems and request scheduling, one of 12 directions in Efficient LLM inference and serving. Builds on 21 of its 52 references that are in the map. First posted on arXiv 9 Sep 2026.

Builds on

21 of 52 references.

Closest ancestors · 6

Same direction, newest first

  1. 2025On Evaluating Performance of LLM Inference Serving Systems11 Jul
  2. 2024Revisiting Service Level Objectives and System Level Metrics in Large Language Model Serving18 Oct
  3. 2024Llumnix: Dynamic Scheduling for Large Language Model Serving5 Jun
  4. 2024Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve4 Mar
  5. 2024BurstGPT: A Real-world Workload Dataset to Optimize LLM Serving Systems31 Jan
  6. 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving18 Jan

Other references · 15

  1. 2024The Llama 3 Herd of Modelscited by 14,715
  2. 2025Qwen3 Technical Reportcited by 8,820
  3. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  4. 2025gpt-oss-120b & gpt-oss-20b Model Cardcited by 1,386
  5. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,141
  6. 2023ChemCrow: Augmenting large-language models with chemistry toolscited by 631
  7. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 575
  8. 2023Splitwise: Efficient generative LLM inference using phase splittingcited by 558
  9. 2017Ray: A Distributed Framework for Emerging AI Applicationscited by 527
  10. 2022DeepSpeed Inference: Enabling Efficient Inference of Transformer Models at Unprecedented Scalecited by 407
  11. 2023Distributed Inference and Fine-tuning of Large Language Models Over The Internetcited by 42
  12. 2024SkyServe: Serving AI Models across Regions and Clouds with Spot Instancescited by 20
3 more
  1. 2020An In-Depth Analysis of the Slingshot Interconnectcited by 13
  2. 2025FIRST: Federated Inference Resource Scheduling Toolkit for Scientific AI Model Accesscited by 5
  3. 2025Aurora: Architecting Argonne's First Exascale Supercomputer for Accelerated Scientific Discoverycited by 2

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025ThunderServe: High-performance and Cost-efficient LLM Serving in Cloud Environments
  2. 2025SageServe: Optimizing LLM Serving on Cloud Data Centers with Forecast Aware Auto-Scaling
  3. 2026OpScale: Operator-level Provisioning and Autoscaling for LLM Serving
  4. 2025HydraServe: Minimizing Cold Start Latency for Serverless LLM Serving in Public Clouds
  5. 2025DynaServe: Unified and Elastic Execution for Dynamic Disaggregated LLM Serving
  6. 2026Cross-Model Autoscaling for Shared LLM Serving
  7. 2026OServe: Accelerating LLM Serving via Spatial-Temporal Workload Orchestration
  8. 2025LLMServingSim2.0: A Unified Simulator for Heterogeneous Hardware and Serving Techniques in LLM Infrastructure