mlsanityData to 14 Oct 2026

WANSpec: Leveraging Global Compute Capacity for LLM Inference

arXiv 2602.18931 ↗First posted 21 Feb 2026cs.DCcited by 0

On the map: Efficient LLM inference and serving → Serving systems and request scheduling

Abstract (arXiv)

Data centers capable of running large language models (LLMs) are spread across the globe. Some have high end GPUs for running the most advanced models (100B+ parameters), and others are only suitable for smaller models (1B parameters). The most capable GPUs are under high demand thanks to the rapidly expanding applications of LLMs. Choosing the right location to run an LLM inference workload can have consequences on the latency of requests due to these high demands. In this work, we explore options to shift some aspects of inference to the under-utilized data centers. We first observe the varying delays affecting inference in AWS services from different regions, demonstrating that load is not spread evenly. We then introduce WANSpec, which offloads part of LLM generation to the under-utilized data centers. In doing so, WANSpec can mitigate capacity issues as well as effectively use on-site compute (ie at universities) to augment cloud providers. This is done with speculative decoding, a widely used technique to speed up auto-regressive decoding, by moving the draft model to the under-utilized compute resources. Our experiments in simulation and cloud deployments show that WANSpec can judiciously employ redundancy to avoid increases in latency while still reducing the forward passes of speculative decoding's draft model in high demand data centers by over 50%.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Serving systems and request scheduling, one of 12 directions in Efficient LLM inference and serving. Builds on 20 of its 36 references that are in the map. First posted on arXiv 21 Feb 2026.

Builds on

20 of 36 references.

Closest ancestors · 3

Same direction, newest first

  1. 2024Efficient LLM Scheduling by Learning to Rank28 Aug
  2. 2024Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve4 Mar
  3. 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving18 Jan

Other references · 17

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,626
  3. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  4. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  5. 2022Fast Inference from Transformers via Speculative Decodingcited by 1,653
  6. 2023Accelerating Large Language Model Decoding with Speculative Samplingcited by 1,006
  7. 2024Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Headscited by 843
  8. 2024EAGLE-2: Faster Inference of Language Models with Dynamic Draft Treescited by 370
  9. 2023SpecInfer: Accelerating Generative Large Language Model Serving with Tree-based Speculative Inference and Verificationcited by 365
  10. 2024Unlocking Efficiency in Large Language Model Inference: A Comprehensive Survey of Speculative Decodingcited by 269
  11. 2024DynamoLLM: Designing LLM Inference Clusters for Performance and Energy Efficiencycited by 112
  12. 2020EdgeBERT: Sentence-Level Energy Optimizations for Latency-Aware Multi-Task NLP Inferencecited by 70
5 more
  1. 2024OPT-Tree: Speculative Decoding with Adaptive Draft Tree Structurecited by 67
  2. 2024SpecExec: Massively Parallel Speculative Decoding for Interactive LLM Inference on Consumer Devicescited by 46
  3. 2025LServe: Efficient Long-sequence LLM Serving with Unified Sparse Attentioncited by 41
  4. 2024Dovetail: A CPU/GPU Heterogeneous Speculative Decoding for LLM inferencecited by 13
  5. 2019Judicious QoS using Cloud Overlayscited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026WISP: Waste- and Interference-Suppressed Distributed Speculative LLM Serving at the Edge via Dynamic Drafting and SLO-Aware Batching
  2. 2025AdaSpec: Adaptive Speculative Decoding for Fast, SLO-Aware Large Language Model Serving
  3. 2026ConfigSpec: Profiling-Based Configuration Selection for Distributed Edge--Cloud Speculative LLM Serving
  4. 2025TAPAS: Thermal- and Power-Aware Scheduling for LLM Inference in Cloud Platforms
  5. 2026A Measurement Study of LLM Inference Trade-offs Across Edge Continuum Hardware
  6. 2025SageServe: Optimizing LLM Serving on Cloud Data Centers with Forecast Aware Auto-Scaling
  7. 2024AcceLLM: Accelerating LLM Inference using Redundancy for Load Balancing and Data Locality
  8. 2025Parallax: Efficient LLM Inference Service over Decentralized Environment