mlsanityData to 14 Oct 2026

DPS: Dual-Mode Precision LLM Serving with Semi-Unified Memory

arXiv 2609.34380 ↗First posted 28 Sep 2026cs.DC cs.AI cs.ETcited by 0

On the map: Efficient LLM inference and serving → Serving systems and request scheduling

Abstract (arXiv)

Existing LLM serving systems virtualize and optimize KV-cache memory, but treat model-weight memory as fixed throughout execution. Recent work on multi-precision model representations challenges this design by allowing a single stored model to support both full-accuracy and lower-precision execution, making the effective weight footprint runtime-dependent. This creates an opportunity under bursty workloads, where temporary spikes in KV-cache demand often determine throughput and SLO compliance. We present DPS, a dual-precision LLM serving system that turns weight memory into an elastic resource: under normal load, DPS serves the full-accuracy model; under KV pressure, it switches to a nested, lower-precision variant and repurposes unused weight memory for KV cache blocks. DPS is built on Semi-Unified Memory (SUM), which partitions the weight region into a persistent lower-precision sub-region and a shared region that alternates between residual weight tensors and KV-cache blocks, preserving compatibility with paged KV-cache management. We implement DPS on top of vLLM and evaluate it across both dense and MoE models and various production workload traces. Our results show that sysname improves sustained throughput by 2.1--3.3× and effective pass@1 by up to +41,pp over Static FP16, while preserving FP16-class accuracy.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Serving systems and request scheduling, one of 12 directions in Efficient LLM inference and serving. Builds on 34 of its 49 references that are in the map. First posted on arXiv 28 Sep 2026.

Builds on

34 of 49 references.

Closest ancestors · 5

Same direction, newest first

  1. 2025MorphServe: Efficient and Workload-Aware LLM Serving via Runtime Quantized Layer Swapping and KV Cache Resizing24 May
  2. 2024vAttention: Dynamic Memory Management for Serving LLMs without PagedAttention7 May
  3. 2024Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Serve4 Mar
  4. 2024BurstGPT: A Real-world Workload Dataset to Optimize LLM Serving Systems31 Jan
  5. 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving18 Jan

Other references · 29

  1. 2020Language Models are Few-Shot Learnerscited by 32,419
  2. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  3. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  4. 2025Qwen3 Technical Reportcited by 8,820
  5. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  6. 2023SWE-bench: Can Language Models Resolve Real-World GitHub Issues?cited by 3,373
  7. 2024Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phonecited by 1,917
  8. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,386
  9. 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,298
  10. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,141
  11. 2017In-Datacenter Performance Analysis of a Tensor Processing Unitcited by 618
  12. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 575
17 more
  1. 2022DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scalecited by 404
  2. 2024QServe: W4A8KV4 Quantization and System Co-design for Efficient LLM Servingcited by 208
  3. 2023TPU v4: An Optically Reconfigurable Supercomputer for Machine Learning with Hardware Support for Embeddingscited by 202
  4. 2022FP8 Quantization: The Power of the Exponentcited by 88
  5. 2024MoE-Lightning: High-Throughput MoE Inference on Memory-constrained GPUscited by 73
  6. 2026Qwen3-Coder-Next Technical Reportcited by 71
  7. 2023EE-LLM: Large-Scale Training and Inference of Early-Exit Large Language Models with 3D Parallelismcited by 69
  8. 2024Any-Precision LLM: Low-Cost Deployment of Multiple, Different-Sized LLMscited by 54
  9. 2019Any-Precision Deep Neural Networkscited by 40
  10. 2023Efficient Post-training Quantization with FP8 Formatscited by 29
  11. 2025SpecEE: Accelerating Large Language Model Inference with Speculative Early Exitingcited by 23
  12. 2025SLOs-Serve: Optimized Serving of Multi-SLO LLMscited by 19
  13. 2024QQQ: Quality Quattuor-Bit Quantization for Large Language Modelscited by 16
  14. 2025An Inquiry into Datacenter TCO for LLM Inference with FP8cited by 5
  15. 2025eLLM: Elastic Memory Management Framework for Efficient LLM Servingcited by 5
  16. 2025NestedFP: High-Performance, Memory-Efficient Dual-Precision Floating Point Support for LLMscited by 3
  17. 2026MatGPTQ: Efficient and Accurate Inference over Nested Quantized Modelscited by 3

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026From Tensor Buffer to Distributed Memory Hierarchy: A Survey of KV Cache Management for LLM Serving
  2. 2026The Price of Precision: A Calibrated Cost Model for Mixed-Precision LLM Serving
  3. 2026The KV Cache Is the New Memory Wall
  4. 2025DynaServe: Unified and Elastic Execution for Dynamic Disaggregated LLM Serving
  5. 2025Understanding Bottlenecks for Efficiently Serving LLM Inference With KV Offloading
  6. 2026CrossPool: Efficient Multi-LLM Serving for Cold MoE Models through KV-Cache and Weight Disaggregation
  7. 2026RedKnot: Efficient Long-Context LLM Serving with Head-Aware KV Reuse and SegPagedAttention
  8. 2026More GPUs or a Smaller Cache? Tensor Parallelism versus KV Compression for Memory-Bound LLM Serving