mlsanityData to 13 Oct 2026

Cocktail: Chunk-Adaptive Mixed-Precision Quantization for Long-Context LLM Inference

arXiv 2503.23294 ↗First posted 30 Mar 2025cs.CLcited by 5

On the map: Efficient LLM inference and serving → KV cache quantization

Abstract (arXiv)

Recently, large language models (LLMs) have been able to handle longer and longer contexts. However, a context that is too long may cause intolerant inference latency and GPU memory usage. Existing methods propose mixed-precision quantization to the key-value (KV) cache in LLMs based on token granularity, which is time-consuming in the search process and hardware inefficient during computation. This paper introduces a novel approach called Cocktail, which employs chunk-adaptive mixed-precision quantization to optimize the KV cache. Cocktail consists of two modules: chunk-level quantization search and chunk-level KV cache computation. Chunk-level quantization search determines the optimal bitwidth configuration of the KV cache chunks quickly based on the similarity scores between the corresponding context chunks and the query, maintaining the model accuracy. Furthermore, chunk-level KV cache computation reorders the KV cache chunks before quantization, avoiding the hardware inefficiency caused by mixed-precision quantization in inference computation. Extensive experiments demonstrate that Cocktail outperforms state-of-the-art KV cache quantization methods on various models and datasets.

Key facts. Cited by 5 papers in the map (data through 13 Oct 2026). Placed in KV cache quantization, one of 12 directions in Efficient LLM inference and serving. Builds on 33 of its 40 references that are in the map. First posted on arXiv 30 Mar 2025.

Builds on

33 of 40 references.

Closest ancestors · 5

Same direction, newest first

  1. 2024QServe: W4A8KV4 Quantization and System Co-design for Efficient LLM Serving7 May
  2. 2024QAQ: Quality Adaptive Quantization for LLM KV Cache7 Mar
  3. 2024KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache5 Feb
  4. 2024KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantization31 Jan
  5. 2023Atom: Low-bit Quantization for Efficient and Accurate LLM Serving29 Oct

Other references · 28

  1. 2017Attention Is All You Needcited by 60,106
  2. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,979
  3. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,221
  4. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,831
  5. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,637
  6. 2023Mistral 7Bcited by 3,123
  7. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,570
  8. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,137
  9. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,884
  10. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,386
  11. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,365
  12. 2023GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpointscited by 1,323
16 more
  1. 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,272
  2. 2021Unsupervised Dense Information Retrieval with Contrastive Learningcited by 1,205
  3. 2024Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Headscited by 829
  4. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Modelscited by 813
  5. 2023Extending Context Window of Large Language Models via Positional Interpolationcited by 727
  6. 2019Fast Transformer Decoding: One Write-Head is All You Needcited by 705
  7. 2023YaRN: Efficient Context Window Extension of Large Language Modelscited by 589
  8. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 570
  9. 2023Model Tells You What to Discard: Adaptive KV Cache Compression for LLMscited by 466
  10. 2023SpQR: A Sparse-Quantized Representation for Near-Lossless LLM Weight Compressioncited by 365
  11. 2023SqueezeLLM: Dense-and-Sparse Quantizationcited by 292
  12. 2022Memorizing Transformerscited by 222
  13. 2023TEST: Text Prototype Aligned Embedding to Activate LLM's Ability for Time Seriescited by 166
  14. 2024No Token Left Behind: Reliable KV Cache Compression via Importance-Aware Mixed Precision Quantizationcited by 99
  15. 2021Pyserini: An Easy-to-Use Python Toolkit to Support Replicable IR Research with Sparse and Dense Representationscited by 60
  16. 2023Large Language Models for Healthcare Data Augmentation: An Example on Patient-Trial Matchingcited by 30

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2025: 3; 2026: 2.

5 papers.

  1. 2025MoQAE: Mixed-Precision Quantization for Long-Context LLM Inference via Mixture of Quantization-Aware Expertscited by 12
  2. 2026Don't Waste Bits! Adaptive KV-Cache Quantization for Lightweight On-Device LLMscited by 7
  3. 2025Lethe: Layer- and Time-Adaptive KV Cache Pruning for Reasoning-Intensive LLM Servingcited by 5
  4. 2026Runtime-Certified Bounded-Error Quantized Attentioncited by 4
  5. 2025MADLLM: Multivariate Anomaly Detection via Pre-trained LLMscited by 3

Similar, not cited

Close in wording, no citation either way

  1. 2025MILLION: Mastering Long-Context LLM Inference Via Outlier-Immunized KV Product Quantization
  2. 2026SemKV: Semantic Mixed-Precision KV Cache Quantization Guided by the Quality Cliff for Long-Context LLM Inference
  3. 2025PM-KVQ: Progressive Mixed-precision KV Cache Quantization for Long-CoT LLMs
  4. 2025KV Pareto: Systems-Level Optimization of KV Cache and Model Compression for Long Context Inference
  5. 2025KVTuner: Sensitivity-Aware Layer-Wise Mixed-Precision KV Cache Quantization for Efficient and Nearly Lossless LLM Inference
  6. 2025NQKV: A KV Cache Quantization Scheme Based on Normal Distribution Characteristics
  7. 2025VecInfer: Efficient LLM Inference with Low-Bit KV Cache via Outlier-Suppressed Vector Quantization
  8. 2025FlexQuant: A Flexible and Efficient Dynamic Precision Switching Framework for LLM Quantization