mlsanityData to 7 Oct 2026

Does Execution Require Target KV Fidelity? A Mixed-Fidelity KV Runtime for LLM Serving

arXiv 2609.33536 ↗First posted 27 Sep 2026cs.LGcited by 0

On the map: Efficient LLM inference and serving → Serving systems and request scheduling

Abstract (arXiv)

Large language model (LLM) serving is increasingly constrained by the GPU memory consumed by key-value (KV) caches. Existing compression, eviction, and offloading techniques alleviate this pressure, but serving runtimes typically treat only the configured target KV representation as execution-ready. Under memory pressure, this target-only contract can turn KV shortage into request stalls and preemptions. We present ElasticKV, a mixed-fidelity KV runtime built on the observation that target fidelity need not gate execution. ElasticKV introduces a compact intermediate KV state, making fidelity a runtime-managed execution property. To realize this state in a paged serving runtime, ElasticKV combines (i) a pair-structured layout that turns fidelity reduction into reusable GPU capacity, (ii) a dual-mode attention backend that directly consumes the compact state while preserving the native target-only path, and (iii) pressure-aware fidelity management that adapts KV fidelity to memory pressure. Our extensive evaluation across diverse workloads, model families and scales, and GPU platforms demonstrates the effectiveness and generality of ElasticKV. Under high concurrency, ElasticKV achieves 3.8-4.0× lower time-to-first-token (TTFT) and 9.1× lower P90 TTFT than vLLM while preserving generation quality.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in Serving systems and request scheduling, one of 12 directions in Efficient LLM inference and serving. Builds on 52 of its 56 references that are in the map. First posted on arXiv 27 Sep 2026.

Builds on

52 of 56 references.

Closest ancestors · 4

Same direction, newest first

  1. 2026Coordinated Scheduling for MoE LLM Serving13 Jun
  2. 2024FastSwitch: Optimizing Context Switching Efficiency in Fairness-aware Large Language Model Serving27 Nov
  3. 2024Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving24 Jun
  4. 2023Towards Efficient Generative Large Language Model Serving: A Survey from Algorithms to Systems23 Dec

Other references · 48

  1. 2024The Llama 3 Herd of Modelscited by 14,192
  2. 2017Towards Deep Learning Models Resistant to Adversarial Attackscited by 6,326
  3. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  4. 2019Benchmarking Neural Network Robustness to Common Corruptions and Perturbationscited by 2,857
  5. 2023A Survey of Large Language Modelscited by 2,537
  6. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  7. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,378
  8. 2019Certified Adversarial Robustness via Randomized Smoothingcited by 1,324
  9. 2019Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shiftcited by 1,274
  10. 2019Adversarial Examples Are Not Bugs, They Are Featurescited by 950
  11. 2024SnapKV: LLM Knows What You are Looking for Before Generationcited by 848
  12. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Modelscited by 805
36 more
  1. 2024KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cachecited by 604
  2. 2024KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantizationcited by 571
  3. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 567
  4. 2021Fast Model Editing at Scalecited by 491
  5. 2024PyramidKV: Dynamic KV Cache Compression based on Pyramidal Information Funnelingcited by 420
  6. 2025FlashInfer: Efficient and Customizable Attention Engine for LLM Inference Servingcited by 248
  7. 2024InfiniGen: Efficient Generative Inference of Large Language Models with Dynamic KV Cache Managementcited by 242
  8. 2024Ada-KV: Optimizing KV Cache Eviction by Adaptive Budget Allocation for Efficient LLM Inferencecited by 230
  9. 2024QServe: W4A8KV4 Quantization and System Co-design for Efficient LLM Servingcited by 200
  10. 2023CacheGen: KV Cache Compression and Streaming for Fast Large Language Model Servingcited by 199
  11. 2024Cost-Efficient Large Language Model Serving for Multi-turn Conversations with CachedAttentioncited by 159
  12. 2024GEAR: An Efficient KV Cache Compression Recipe for Near-Lossless Generative Inference of LLMcited by 155
  13. 2025LMCache: An Efficient KV Cache Layer for Enterprise-Scale LLM Inferencecited by 119
  14. 2024ZipCache: Accurate and Efficient KV Cache Quantization with Salient Token Identificationcited by 86
  15. 2024KV Cache is 1 Bit Per Channel: Efficient Large Language Model Inference with Coupled Quantizationcited by 74
  16. 2025RotateKV: Accurate and Robust 2-Bit KV Cache Quantization for LLMs via Outlier-Aware Adaptive Rotationscited by 42
  17. 2025LServe: Efficient Long-sequence LLM Serving with Unified Sparse Attentioncited by 41
  18. 2025KVTuner: Sensitivity-Aware Layer-Wise Mixed-Precision KV Cache Quantization for Efficient and Nearly Lossless LLM Inferencecited by 36
  19. 2024DiffKV: Differentiated Memory Management for Large Language Models with Parallel KV Compactioncited by 26
  20. 2025CommVQ: Commutative Vector Quantization for KV Cache Compressioncited by 25
  21. 2025ThinKV: Thought-Adaptive KV Cache Compression for Efficient Reasoning Modelscited by 21
  22. 2025PM-KVQ: Progressive Mixed-precision KV Cache Quantization for Long-CoT LLMscited by 19
  23. 2025Accurate KV Cache Quantization with Outlier Tokens Tracingcited by 15
  24. 2024KV-Compress: Paged KV-Cache Compression with Variable Compression Rates per Attention Headcited by 12
  25. 2025VQ-LLM: High-performance Code Generation for Vector Quantization Augmented LLM Inferencecited by 12
  26. 2025MoQAE: Mixed-Precision Quantization for Long-Context LLM Inference via Mixture of Quantization-Aware Expertscited by 12
  27. 2024Regulating Model Reliance on Non-Robust Features by Smoothing Input Marginal Densitycited by 10
  28. 2025LaCache: Ladder-Shaped KV Caching for Efficient Long-Context Modeling of Large Language Modelscited by 9
  29. 2025NSNQuant: A Double Normalization Approach for Calibration-Free Low-Bit Vector Quantization of KV Cachecited by 7
  30. 2025PagedEviction: Structured Block-wise KV Cache Pruning for Efficient Large Language Model Inferencecited by 7
  31. 2025KVmix: Gradient-Based Layer Importance-Aware Mixed-Precision Quantization for KV Cachecited by 4
  32. 2026Zipage: Maintain High Request Concurrency for LLM Reasoning through Compressed PagedAttentioncited by 4
  33. 2026Attribution-Guided Model Rectification of Unreliable Neural Network Behaviorscited by 3
  34. 2026Towards Efficient Large Language Model Serving: A Survey on System-Aware KV Cache Optimizationcited by 2
  35. 2026Semantic Robustness Certification for Vision-Language Modelscited by 1
  36. 2026Lynx: Progressive Speculative Quantization for accelerating KV Transfer in Long-Context Inferencecited by 1

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025eLLM: Elastic Memory Management Framework for Efficient LLM Serving
  2. 2026The KV Cache Is the New Memory Wall
  3. 2026RedKnot: Efficient Long-Context LLM Serving with Head-Aware KV Reuse and SegPagedAttention
  4. 2026TempoKV: Timely Staging of LLM KV Caches for Memory-Semantic Flash
  5. 2026Elastic KV Cache for LLM Serving:A Working Reclamation Mechanism, and Why Chunked Prefill Already Closes the Gap
  6. 2026KV Cache Optimization Strategies for Scalable and Efficient LLM Inference
  7. 2024LayerKV: Optimizing Large Language Model Serving with Layer-wise KV Cache Management
  8. 2026Dynamic Flow, Static Graph: KV Cache Reuse for Efficient LLM Serving on Mobile NPUs