mlsanityData to 14 Oct 2026

MorphServe: Efficient and Workload-Aware LLM Serving via Runtime Quantized Layer Swapping and KV Cache Resizing

arXiv 2506.02006 ↗First posted 24 May 2025cs.DC cs.LGcited by 6

On the map: Efficient LLM inference and serving → Serving systems and request scheduling

Abstract (arXiv)

Efficiently serving large language models (LLMs) under dynamic and bursty workloads remains a key challenge for real-world deployment. Existing serving frameworks and static model compression techniques fail to adapt to workload fluctuations, leading to either service-level objective (SLO) violations under full-precision serving or persistent accuracy degradation with static quantization. We present MorphServe, a dynamic, workload-aware LLM serving framework based on morphological adaptation. MorphServe introduces two asynchronous, token-level runtime mechanisms: quantized layer swapping, which selectively replaces less impactful layers with quantized alternatives during high-load periods, and pressure-aware KV cache resizing, which dynamically adjusts KV cache capacity in response to memory pressure. These mechanisms enable state-preserving transitions with minimum runtime overhead and are fully compatible with modern scheduling and attention techniques. Extensive experiments on Vicuna and Llama family models with real-world workloads demonstrate that MorphServe reduces average SLO violations by 92.45 percent and improves the P95 TTFT latency by 2.2x-3.9x compared to full-precision serving, without compromising generation quality. These results establish MorphServe as a practical and elastic solution for LLM deployment in dynamic environments.

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Placed in Serving systems and request scheduling, one of 12 directions in Efficient LLM inference and serving. Builds on 57 of its 76 references that are in the map. First posted on arXiv 24 May 2025.

Builds on

57 of 76 references.

Closest ancestors · 5

Same direction, newest first

  1. 2024NEO: Saving GPU Memory Crisis with CPU Offloading for Online LLM Inference2 Nov
  2. 2024Llumnix: Dynamic Scheduling for Large Language Model Serving5 Jun
  3. 2024Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve4 Mar
  4. 2024BurstGPT: A Real-world Workload Dataset to Optimize LLM Serving Systems31 Jan
  5. 2023Fast Distributed Inference Serving for Large Language Models10 May

Other references · 52

  1. 2017Attention Is All You Needcited by 60,361
  2. 2024The Llama 3 Herd of Modelscited by 14,715
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  5. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  6. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,598
  7. 2023Code Llama: Open Foundation Models for Codecited by 2,139
  8. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,919
  9. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,417
  10. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,386
  11. 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,298
  12. 2024DeepSeek-V2: A Strong, Economical, and Efficient Mixture-of-Experts Language Modelcited by 1,135
38 more
  1. 2024SnapKV: LLM Knows What You are Looking for Before Generationcited by 880
  2. 2023LLM-Pruner: On the Structural Pruning of Large Language Modelscited by 650
  3. 2020Up or Down? Adaptive Rounding for Post-Training Quantizationcited by 621
  4. 2023Splitwise: Efficient generative LLM inference using phase splittingcited by 558
  5. 2022ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformerscited by 526
  6. 2021A White Paper on Neural Network Quantizationcited by 472
  7. 2023Scissorhands: Exploiting the Persistence of Importance Hypothesis for LLM KV Cache Compression at Test Timecited by 439
  8. 2024PyramidKV: Dynamic KV Cache Compression based on Pyramidal Information Funnelingcited by 432
  9. 2019Multi-News: a Large-Scale Multi-Document Summarization Dataset and Abstractive Hierarchical Modelcited by 431
  10. 2023OmniQuant: Omnidirectionally Calibrated Quantization for Large Language Modelscited by 424
  11. 2019HAWQ: Hessian AWare Quantization of Neural Networks with Mixed-Precisioncited by 402
  12. 2023SpQR: A Sparse-Quantized Representation for Near-Lossless LLM Weight Compressioncited by 367
  13. 2021QMSum: A New Benchmark for Query-based Multi-domain Meeting Summarizationcited by 365
  14. 2021Efficient Attentions for Long Document Summarizationcited by 345
  15. 2024LayerSkip: Enabling Early Exit Inference and Self-Speculative Decodingcited by 307
  16. 2023SqueezeLLM: Dense-and-Sparse Quantizationcited by 294
  17. 2018Model compression via distillation and quantizationcited by 289
  18. 2023Atom: Low-bit Quantization for Efficient and Accurate LLM Servingcited by 272
  19. 2024Ada-KV: Optimizing KV Cache Eviction by Adaptive Budget Allocation for Efficient LLM Inferencecited by 235
  20. 2024QServe: W4A8KV4 Quantization and System Co-design for Efficient LLM Servingcited by 208
  21. 2024DuQuant: Distributing Outliers via Dual Transformation Makes Stronger Quantized LLMscited by 167
  22. 2024Cost-Efficient Large Language Model Serving for Multi-turn Conversations with CachedAttentioncited by 163
  23. 2024DynamoLLM: Designing LLM Inference Clusters for Performance and Energy Efficiencycited by 112
  24. 2017DuReader: a Chinese Machine Reading Comprehension Dataset from Real-world Applicationscited by 110
  25. 2024LLM Pruning and Distillation in Practice: The Minitron Approachcited by 82
  26. 2024ServerlessLLM: Low-Latency Serverless Inference for Large Language Modelscited by 80
  27. 2024AffineQuant: Affine Transformation Quantization for Large Language Modelscited by 67
  28. 2024Transformer Layers as Painterscited by 54
  29. 2024DynamicKV: Task-Aware Adaptive KV Cache Compression for Long Context LLMscited by 42
  30. 2024Tracing Representation Progression: Analyzing and Enhancing Layer-Wise Similaritycited by 36
  31. 2023Norm Tweaking: High-performance Low-bit Quantization of Large Language Modelscited by 33
  32. 2025Adaptive Layer-skipping in Pre-trained LLMscited by 27
  33. 2024DISP-LLM: Dimension-Independent Structural Pruning for Large Language Modelscited by 24
  34. 2025TransMLA: Multi-Head Latent Attention Is All You Needcited by 23
  35. 2024Progressive Mixed-Precision Decoding for Efficient LLM Inferencecited by 18
  36. 2024LayerKV: Optimizing Large Language Model Serving with Layer-wise KV Cache Managementcited by 16
  37. 2024Optimised Grouped-Query Attention Mechanism for Transformerscited by 12
  38. 2022Channel-wise Mixed-precision Assignment for DNN Inference on Constrained Edge Nodescited by 9

50 most cited of 52 All 52 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 5.

6 papers.

  1. 2025TokenScale: Timely and Accurate Autoscaling for Disaggregated LLM Serving with Token Velocitycited by 12
  2. 2026ObjectCache: Layerwise Object-Storage Retrieval for KV Cache Reusecited by 1
  3. 2026Pricing the Risk of Runtime Compression: Anytime-Valid Admission and a Served-Output Law for Compressed Serving Statecited by 0
  4. 2026DPS: Dual-Mode Precision LLM Serving with Semi-Unified Memorycited by 0
  5. 2026The Price of Precision: A Calibrated Cost Model for Mixed-Precision LLM Servingcited by 0
  6. 2026Context-Aware Precision Scheduling for Small Language Modelscited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026OServe: Accelerating LLM Serving via Spatial-Temporal Workload Orchestration
  2. 2026Towards Efficient Large Language Model Serving: A Survey on System-Aware KV Cache Optimization
  3. 2025Unlock the Potential of Fine-grained LLM Serving via Dynamic Module Scaling
  4. 2026RedKnot: Efficient Long-Context LLM Serving with Head-Aware KV Reuse and SegPagedAttention
  5. 2025FastCache: Optimizing Multimodal LLM Serving through Lightweight KV-Cache Compression Framework
  6. 2026Cascade: Exploiting SLO-Aware latency budget for fair and high goodput LLM inference serving
  7. 2026The KV Cache Working Set: Online Capacity Planning for LLM Inference Systems
  8. 2025Apt-Serve: Adaptive Request Scheduling on Hybrid Cache for Scalable LLM Inference Serving