mlsanityData to 14 Oct 2026

PatternKV: Flattening KV Representation Expands Quantization Headroom

arXiv 2510.05176 ↗First posted 5 Oct 2025cs.LG cs.AIcited by 0

On the map: LLM reasoning → Test-time compute and efficient thinking

also in 1 more
Abstract (arXiv)

KV cache in autoregressive LLMs eliminates redundant recomputation but has emerged as the dominant memory and bandwidth bottleneck during inference, notably with long contexts and test-time scaling. KV quantization is a key lever for reducing cache cost, but accuracy drops sharply as the native KV distribution lacks flatness and thus maintains a wide quantization range. Prior work focuses on isolating outliers, which caps their error but fails to flatten the overall distribution, leaving performance fragile under low-bit settings. In this work, we show that the K cache maintains a stable, context-evolving structure, while the V cache carries latent semantic regularities, with both contributing to the organization of vectors into shared patterns. Building on these insights, we propose PatternKV, a pattern-aligned residual quantization scheme. It mines representative pattern vectors online, aligns each KV vector to its nearest pattern, and quantizes only the residual. This reshaping of the KV distribution flattens the quantization target and narrows its range, thereby improving the fidelity of low-bit KV quantization. Across long-context and test-time scaling settings on multiple backbones, PatternKV delivers consistent 2-bit gains, with a 0.08% average 4-bit drop relative to FP16, improves test-time scaling accuracy by 10% on average, and raises throughput by 1.5x while supporting 1.25x larger batches.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Test-time compute and efficient thinking, one of 17 directions in LLM reasoning. Builds on 36 of its 44 references that are in the map. First posted on arXiv 5 Oct 2025.

Builds on

36 of 44 references.

Closest ancestors · 4

Same direction, newest first

  1. 2025s1: Simple test-time scaling31 Jan
  2. 2024Inference Scaling Laws: An Empirical Analysis of Compute-Optimal Inference for Problem-Solving with Language Models1 Aug
  3. 2024ThinK: Thinner Key Cache by Query-Driven Pruning30 Jul
  4. 2023Self-Evaluation Guided Beam Search for Reasoning1 May

Other references · 32

  1. 2023GPT-4 Technical Reportcited by 18,396
  2. 2021Training Verifiers to Solve Math Word Problemscited by 10,109
  3. 2020Retrieval-Augmented Generation for Knowledge-Intensive NLP Taskscited by 9,297
  4. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,944
  5. 2022Self-Consistency Improves Chain of Thought Reasoning in Language Modelscited by 5,708
  6. 2024Qwen2.5 Technical Reportcited by 4,942
  7. 2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,926
  8. 2020Longformer: The Long-Document Transformercited by 3,413
  9. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,186
  10. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,919
  11. 2023LongBench: A Bilingual, Multitask Benchmark for Long Context Understandingcited by 1,417
  12. 2024SnapKV: LLM Knows What You are Looking for Before Generationcited by 880
20 more
  1. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Modelscited by 832
  2. 2024KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cachecited by 637
  3. 2024QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMscited by 634
  4. 2024KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantizationcited by 591
  5. 2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 575
  6. 2024Quest: Query-Aware Sparsity for Efficient Long-Context LLM Inferencecited by 501
  7. 2023Scissorhands: Exploiting the Persistence of Importance Hypothesis for LLM KV Cache Compression at Test Timecited by 439
  8. 2025MathArena: Evaluating LLMs on Uncontaminated Math Competitionscited by 297
  9. 2024GEAR: An Efficient KV Cache Compression Recipe for Near-Lossless Generative Inference of LLMcited by 158
  10. 2024ZipCache: Accurate and Efficient KV Cache Quantization with Salient Token Identificationcited by 93
  11. 2024KV Cache is 1 Bit Per Channel: Efficient Large Language Model Inference with Coupled Quantizationcited by 76
  12. 2024SKVQ: Sliding-window Key and Value Cache Quantization for Large Language Modelscited by 44
  13. 2024TokenSelect: Efficient Long-Context Inference and Length Extrapolation for LLMs via Dynamic Token-Level KV Cache Selectioncited by 39
  14. 2025LLMs Know What to Drop: Self-Attention Guided KV Cache Eviction for Efficient Long-Context Inferencecited by 20
  15. 2024AsymKV: Enabling 1-Bit Quantization of KV Cache with Layer-Wise Asymmetric Quantization Configurationscited by 19
  16. 2025Accurate KV Cache Quantization with Outlier Tokens Tracingcited by 18
  17. 2024Residual vector quantization for KV cache compression in large language modelcited by 7
  18. 2025PagedEviction: Structured Block-wise KV Cache Pruning for Efficient Large Language Model Inferencecited by 7
  19. 2024KVPruner: Structural Pruning for Faster and Memory-Efficient Large Language Modelscited by 6
  20. 2024HashEvict: A Pre-Attention KV Cache Eviction Strategy using Locality-Sensitive Hashingcited by 6

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025KVTuner: Sensitivity-Aware Layer-Wise Mixed-Precision KV Cache Quantization for Efficient and Nearly Lossless LLM Inference
  2. 2025KVSink: Understanding and Enhancing the Preservation of Attention Sinks in KV Cache Quantization for LLMs
  3. 2025NQKV: A KV Cache Quantization Scheme Based on Normal Distribution Characteristics
  4. 2026OScaR: The Occam's Razor for Extreme KV Cache Quantization in LLMs and Beyond
  5. 2026High-accuracy Low-Bit KV-Cache Quantization via Local Distribution Restoration
  6. 2025MILLION: Mastering Long-Context LLM Inference Via Outlier-Immunized KV Product Quantization
  7. 2026KVSculpt: KV Cache Compression as Distillation
  8. 2026Fractal KV-Cache Archives: Lossless Symbolic Storage with In-Place Retrieval for Long-Context LLM Inference