mlsanityData to 14 Oct 2026

FineServe: Precision-Aware KV Slab and Two-Level Scheduling for Heterogeneous Precision LLM Serving

arXiv 2509.06261 ↗First posted 8 Sep 2025cs.DC cs.LGcited by 3

On the map: Efficient LLM inference and serving → KV cache quantization

Abstract (arXiv)

Recent advances in Post-Training Quantization (PTQ) techniques have significantly increased demand for serving quantized large language models (LLMs), enabling higher throughput and substantially reduced memory usage with minimal accuracy loss. Quantized models address memory constraints in LLMs and enhance GPU resource utilization through efficient GPU sharing. However, quantized models have smaller KV block sizes than non-quantized models, causing limited memory efficiency due to memory fragmentation. Also, distinct resource usage patterns between quantized and non-quantized models require efficient scheduling to maximize throughput. To address these challenges, we propose FineServe, an inference serving framework for mixed-precision LLMs. FineServe's key contributions include: (1) KV Slab, a precision-aware adaptive memory management technique dynamically allocating KV cache based on model quantization characteristics, significantly reducing GPU memory fragmentation, and (2) a two-level scheduling framework comprising a global scheduler that places models to GPUs based on request rates, latency SLOs, and memory constraints and efficiency, and a local scheduler that adaptively adjusts batch sizes according to real-time request fluctuations. Experimental results demonstrate that FineServe achieves up to 2.2x higher SLO attainment and 1.8x higher token generation throughput compared to the state-of-the-art GPU sharing systems.

Key facts. Cited by 3 papers in the map (data through 14 Oct 2026). Placed in KV cache quantization, one of 12 directions in Efficient LLM inference and serving. Builds on 11 of its 35 references that are in the map. First posted on arXiv 8 Sep 2025.

Builds on

11 of 35 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024QServe: W4A8KV4 Quantization and System Co-design for Efficient LLM Serving7 May

Other references · 10

  1. 2025Qwen3 Technical Reportcited by 8,820
  2. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  3. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,919
  4. 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,298
  5. 2023Splitwise: Efficient generative LLM inference using phase splittingcited by 558
  6. 2020Serving DNNs like Clockwork: Performance Predictability from the Bottom Upcited by 120
  7. 2024vAttention: Dynamic Memory Management for Serving LLMs without PagedAttentioncited by 98
  8. 2024MuxServe: Flexible Spatial-Temporal Multiplexing for Multiple LLM Servingcited by 52
  9. 2024Queue management for slo-oriented large language model servingcited by 36
  10. 2025Prism: Cost-Efficient Multi-LLM Serving via GPU Memory Ballooningcited by 24

Built on by

Cited by year

Citing papers per year: 2025: 3.

3 papers.

  1. 2025Nightjar: Dynamic Adaptive Speculative Decoding for Large Language Models Servingcited by 6
  2. 2025A Predictive and Synergistic Two-Layer Scheduling Framework for LLM Servingcited by 0
  3. 2025ProServe: Unified Multi-Priority Request Scheduling for LLM Servingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024LLM-PQ: Serving LLM on Heterogeneous Clusters with Phase-Aware Partition and Adaptive Quantization
  2. 2025Cocktail: Chunk-Adaptive Mixed-Precision Quantization for Long-Context LLM Inference
  3. 2025Systematic Characterization of LLM Quantization: A Performance, Energy, and Quality Perspective
  4. 2024LLMEasyQuant: Scalable Quantization for Parallel and Distributed LLM Inference
  5. 2026Understanding Efficiency: Quantization, Batching, and Serving Strategies in LLM Energy Use
  6. 2026FAMPWQ: Fisher Information-based Adaptive Mixed Precision Weight Quantization for Effective LLM Inference
  7. 2024ABQ-LLM: Arbitrary-Bit Quantized Inference Acceleration for Large Language Models
  8. 2024QUICK: Quantization-aware Interleaving and Conflict-free Kernel for efficient LLM inference