mlsanityData to 14 Oct 2026

SemKV: Semantic Mixed-Precision KV Cache Quantization Guided by the Quality Cliff for Long-Context LLM Inference

arXiv 2608.28911 ↗First posted 28 Aug 2026cs.LG cs.CL cs.ITcited by 0

On the map: Efficient LLM inference and serving → KV cache quantization

Abstract (arXiv)

The key-value (KV) cache is the dominant memory bottleneck of long-context large language model (LLM) inference, growing linearly with context length. We show that uniform KV quantization on a fractional-bit grid does not degrade gracefully: under a prespecified multi-seed statistical protocol, Llama-3.1-8B-Instruct with an affine quantizer is statistically indistinguishable from FP16 KV down to 2.322 code bits/value and collapses at 2.0 bits - a quality cliff in (2.0, 2.322] that reappears in generation-time quantization and multi-turn dialogue and transfers to Mistral-7B. The cliff reframes importance-aware mixed precision: above it, eight model-internal importance indicators are statistically interchangeable, so the benefit of mixing is grid interpolation, reaching average precisions uniform quantization cannot realize. SemKV preserves every token, ranks tokens by a model-internal score, and assigns two adjacent above-cliff precisions, achieving a measured 6.0x storage reduction with no statistically detectable quality difference from full KV (n=900, three seeds), and outperforming FP16 token pruning granted a 1.5x larger memory budget. Replacing the affine base with a distortion-optimized quantizer (TurboQuant-MSE) lowers the cliff in every protocol tested, raising the no-detectable-loss operating point to 7.9x. The recipe: measure the cliff for the target deployment setting, then interpolate above it.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in KV cache quantization, one of 12 directions in Efficient LLM inference and serving. Builds on 70 of its 74 references that are in the map. First posted on arXiv 28 Aug 2026.

Builds on

70 of 74 references.

Closest ancestors · 9

Same direction, newest first

  1. 2025RotateKV: Accurate and Robust 2-Bit KV Cache Quantization for LLMs via Outlier-Aware Adaptive Rotations25 Jan
  2. 2024ZipCache: Accurate and Efficient KV Cache Quantization with Salient Token Identification23 May
  3. 2024SKVQ: Sliding-window Key and Value Cache Quantization for Large Language Models10 May
  4. 2024QServe: W4A8KV4 Quantization and System Co-design for Efficient LLM Serving7 May
  5. 2024QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMs30 Mar
  6. 2024QAQ: Quality Adaptive Quantization for LLM KV Cache7 Mar
  7. 2024KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache5 Feb
  8. 2024KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantization31 Jan
  9. 2023Atom: Low-bit Quantization for Efficient and Accurate LLM Serving29 Oct

Other references · 61

  1. 2017Attention Is All You Needcited by 60,361
  2. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  4. 2014Adam: A Method for Stochastic Optimizationcited by 37,240
  5. 2020Language Models are Few-Shot Learnerscited by 32,419
  6. 2023GPT-4 Technical Reportcited by 18,396
  7. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 14,132
  8. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  9. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,703
  10. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,673
  11. 2023Lost in the Middle: How Language Models Use Long Contextscited by 3,455
  12. 2020Longformer: The Long-Document Transformercited by 3,413
38 more
  1. 2023Mistral 7Bcited by 3,164
  2. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
  3. 2022Training Compute-Optimal Large Language Modelscited by 3,019
  4. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,598
  5. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
  6. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,919
  7. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,868
  8. 2020Big Bird: Transformers for Longer Sequencescited by 1,684
  9. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,631
  10. 2019Generating Long Sequences with Sparse Transformerscited by 1,546
  11. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,417
  12. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,386
  13. 2019ZeRO: Memory Optimizations Toward Training Trillion Parameter Modelscited by 1,301
  14. 2017Mixed Precision Trainingcited by 1,156
  15. 2021Train Short, Test Long: Attention with Linear Biases Enables Input Length Extrapolationcited by 1,103
  16. 2022Token Merging: Your ViT But Fastercited by 1,021
  17. 2022LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scalecited by 906
  18. 2024SnapKV: LLM Knows What You are Looking for Before Generationcited by 880
  19. 2021DynamicViT: Efficient Vision Transformers with Dynamic Token Sparsificationcited by 864
  20. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Modelscited by 832
  21. 2020Efficient Transformers: A Surveycited by 790
  22. 2019Fast Transformer Decoding: One Write-Head is All You Needcited by 717
  23. 2021Efficient Large-Scale Language Model Training on GPU Clusters Using Megatron-LMcited by 632
  24. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 575
  25. 2022ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformerscited by 526
  26. 2021BRECQ: Pushing the Limit of Post-Training Quantization by Block Reconstructioncited by 479
  27. 2023Model Tells You What to Discard: Adaptive KV Cache Compression for LLMscited by 474
  28. 2023Scissorhands: Exploiting the Persistence of Importance Hypothesis for LLM KV Cache Compression at Test Timecited by 439
  29. 2024PyramidKV: Dynamic KV Cache Compression based on Pyramidal Information Funnelingcited by 432
  30. 2019HAWQ: Hessian AWare Quantization of Neural Networks with Mixed-Precisioncited by 402
  31. 2023Learning to Compress Prompts with Gist Tokenscited by 390
  32. 2024LongBench v2: Towards Deeper Understanding and Reasoning on Realistic Long-context Multitaskscited by 379
  33. 2024∞Bench: Extending Long Context Evaluation Beyond 100K Tokenscited by 367
  34. 2021AdaViT: Adaptive Tokens for Efficient Vision Transformercited by 362
  35. 2019Data-Free Quantization Through Weight Equalization and Bias Correctioncited by 336
  36. 2022Reducing Activation Recomputation in Large Transformer Modelscited by 302
  37. 2014One weird trick for parallelizing convolutional neural networkscited by 264
  38. 2024InfLLM: Training-Free Long-Context Extrapolation for LLMs with an Efficient Context Memorycited by 197

50 most cited of 61 All 61 (JSON)

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Cocktail: Chunk-Adaptive Mixed-Precision Quantization for Long-Context LLM Inference
  2. 2025KV Pareto: Systems-Level Optimization of KV Cache and Model Compression for Long Context Inference
  3. 2025PM-KVQ: Progressive Mixed-precision KV Cache Quantization for Long-CoT LLMs
  4. 2025KVmix: Gradient-Based Layer Importance-Aware Mixed-Precision Quantization for KV Cache
  5. 2025MILLION: Mastering Long-Context LLM Inference Via Outlier-Immunized KV Product Quantization
  6. 2025KVLinC : KV Cache Quantization with Hadamard Rotation and Linear Correction
  7. 2025KVTuner: Sensitivity-Aware Layer-Wise Mixed-Precision KV Cache Quantization for Efficient and Nearly Lossless LLM Inference
  8. 2025MoQAE: Mixed-Precision Quantization for Long-Context LLM Inference via Mixture of Quantization-Aware Experts