mlsanityData to 7 Oct 2026

MAPS: Memory-Aware Predictive Scheduling Framework for Large Language Model Serving

arXiv 2609.15359 ↗First posted 14 Sep 2026cs.AIcited by 0

On the map: Efficient LLM inference and serving → Serving systems and request scheduling

Abstract (arXiv)

The surge of large language model (LLM) applications on personal devices imposes massive, bursty workloads on cloud serving infrastructure. While prefill-decode disaggregation improves throughput and scalability, memory-bound decode instances often suffer from persistent load imbalance, as output lengths are unknown when requests arrive at the cloud. To address this, we propose MAPS, a Memory-Aware Predictive Scheduling framework tailored for disaggregated LLM serving. MAPS performs device-assisted speculative output length prediction overlapped with cloud-side prefilling, incurring negligible latency overhead. To handle generation uncertainty, MAPS applies uncertainty-aware calibration to derive output-length upper bounds with target coverage, enabling safe scheduling decisions. Building on these bounds, MAPS employs a hierarchical global-local scheduling strategy to mitigate inter-decoder queue buildup and intra-decoder head-of-line blocking. Extensive experiments on two real-world workloads and two LLMs show that MAPS significantly outperforms three state-of-the-art systems, reducing average end-to-end latency by 42.6 and tail latency by up to 84.8.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Serving systems and request scheduling, one of 12 directions in Efficient LLM inference and serving. Builds on 21 of its 37 references that are in the map. First posted on arXiv 14 Sep 2026.

Builds on

21 of 37 references.

Closest ancestors · 7

Same direction, newest first

  1. 2024Efficient LLM Scheduling by Learning to Rank28 Aug
  2. 2024Llumnix: Dynamic Scheduling for Large Language Model Serving5 Jun
  3. 2024Efficient Interactive LLM Serving with Proxy Model-based Sequence Length Prediction12 Apr
  4. 2024BurstGPT: A Real-world Workload Dataset to Optimize LLM Serving Systems31 Jan
  5. 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving18 Jan
  6. 2023SpotServe: Serving Generative Large Language Models on Preemptible Instances27 Nov
  7. 2023Response Length Perception and Sequence Scheduling: An LLM-Empowered LLM Inference Pipeline22 May

Other references · 14

  1. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  2. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  3. 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,234
  4. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,107
  5. 2024DynamoLLM: Designing LLM Inference Clusters for Performance and Energy Efficiencycited by 106
  6. 2023S3: Increasing GPU Utilization during Generative Inference for Higher Throughputcited by 81
  7. 2019Latent-Variable Non-Autoregressive Neural Machine Translation with Deterministic Inference Using a Delta Posteriorcited by 58
  8. 2024Characterization of Large Language Model Development in the Datacentercited by 46
  9. 2024PerLLM: Personalized Inference Scheduling with Edge-Cloud Collaboration for Diverse LLM Servicescited by 23
  10. 2024Atomic Fact Decomposition Helps Attributed Question Answeringcited by 21
  11. 2024Enabling Efficient Batch Serving for LMaaS via Generation Length Predictioncited by 20
  12. 2026State of AI: An Empirical 100 Trillion Token Study with OpenRoutercited by 15
2 more
  1. 2024SplitLLM: Collaborative Inference of LLMs for Model Placement and Throughput Optimizationcited by 10
  2. 2025Splitwise: Collaborative Edge-Cloud Inference for LLMs via Lyapunov-Assisted DRLcited by 3

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Towards Multi-Model LLM Schedulers: Empirical Insights into Offloading and Preemption
  2. 2026OServe: Accelerating LLM Serving via Spatial-Temporal Workload Orchestration
  3. 2026Beyond Prediction: Tail-Aware Scheduling for LLM Inference
  4. 2025Niyama : Breaking the Silos of LLM Inference Serving
  5. 2025DOPD: A Dynamic PD-Disaggregation Architecture for Maximizing Goodput in LLM Inference Serving
  6. 2025LLM Serving Optimization with Variable Prefill and Decode Lengths
  7. 2026AlignedServe: Orchestrating Prefix-aware Batching to Build a High-throughput and Computing-efficient LLM Serving System
  8. 2026SLO-Aware Compute Resource Allocation for Prefill-Decode Disaggregated LLM Inference