mlsanityData to 7 Oct 2026

IntactKV: Improving Large Language Model Quantization by Keeping Pivot Tokens Intact

arXiv 2403.01241 ↗First posted 2 Mar 2024cs.CL cs.AIcited by 52

Not on the map. Why?

Abstract (arXiv)

Large language models (LLMs) excel in natural language processing but demand intensive computation. To mitigate this, various quantization methods have been explored, yet they compromise LLM performance. This paper unveils a previously overlooked type of outliers in LLMs. Such outliers are found to allocate most of the attention scores on initial tokens of input, termed as pivot tokens, which are crucial to the performance of quantized LLMs. Given that, we propose IntactKV to generate the KV cache of pivot tokens losslessly from the full-precision model. The approach is simple and easy to combine with existing quantization solutions with no extra inference overhead. Besides, IntactKV can be calibrated as additional LLM parameters to boost the quantized LLMs further with minimal training costs. Mathematical analysis also proves that IntactKV effectively reduces the upper bound of quantization error. Empirical results show that IntactKV brings consistent improvement over various quantization methods across different LLMs and downstream tasks, leading to the new state-of-the-art for LLM quantization. The codes are available at https://github.com/ruikangliu/IntactKV.

Key facts. Cited by 52 papers in the map (data through 7 Oct 2026). Builds on 20 of its 32 references that are in the map. First posted on arXiv 2 Mar 2024.

Builds on

20 of 32 references.

References · 20

  1. 2020Language Models are Few-Shot Learnerscited by 32,199
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2023Judging LLM-as-a-Judge with MT-Bench and Chatbot Arenacited by 8,393
  5. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  6. 2018Think you have Solved Question Answering? Try ARC, the AI2 Reasoning Challengecited by 4,710
  7. 2019HellaSwag: Can a Machine Really Finish Your Sentence?cited by 4,018
  8. 2016Pointer Sentinel Mixture Modelscited by 2,992
  9. 2023Sparks of Artificial General Intelligence: Early experiments with GPT-4cited by 2,579
  10. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  11. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,140
  12. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
8 more
  1. 2018Can a Suit of Armor Conduct Electricity? A New Dataset for Open Book Question Answeringcited by 2,114
  2. 2022Fast Inference from Transformers via Speculative Decodingcited by 1,615
  3. 2023SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shotcited by 1,090
  4. 2023Accelerating Large Language Model Decoding with Speculative Samplingcited by 975
  5. 2016The LAMBADA dataset: Word prediction requiring a broad discourse contextcited by 960
  6. 2024Medusa: Simple LLM Inference Acceleration Framework with Multiple Decoding Headscited by 821
  7. 2024KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantizationcited by 571
  8. 2023Quantizable Transformers: Removing Outliers by Helping Attention Heads Do Nothingcited by 147

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 13; 2025: 24; 2026: 15.

52 papers.

Sort:

By area:

  1. 2024When Attention Sink Emerges in Language Models: An Empirical Viewcited by 200
  2. 2024DuQuant: Distributing Outliers via Dual Transformation Makes Stronger Quantized LLMscited by 165
  3. 2024A Survey on Large Language Model Acceleration based on KV Cache Managementcited by 125
  4. 2025Why do LLMs attend to the first token?cited by 122
  5. 2025FlexPrefill: A Context-Aware Sparse Attention Mechanism for Efficient Long-Sequence Inferencecited by 121
  6. 2024FlatQuant: Flatness Matters for LLM Quantizationcited by 116
  7. 2024LOOK-M: Look-Once Optimization in KV Cache for Efficient Multimodal Long-Context Inferencecited by 93
  8. 2024PTQ4DiT: Post-training Quantization for Diffusion Transformerscited by 77
  9. 2025Quantization Hurts Reasoning? An Empirical Study on Quantized Reasoning Modelscited by 68
  10. 2024Squeezed Attention: Accelerating Long Context Length LLM Inferencecited by 56
  11. 2024Active-Dormant Attention Heads: Mechanistically Demystifying Extreme-Token Phenomena in LLMscited by 54
  12. 2025RotateKV: Accurate and Robust 2-Bit KV Cache Quantization for LLMs via Outlier-Aware Adaptive Rotationscited by 42
38 more
  1. 2025KVTuner: Sensitivity-Aware Layer-Wise Mixed-Precision KV Cache Quantization for Efficient and Nearly Lossless LLM Inferencecited by 36
  2. 2026The Spike, the Sparse and the Sink: Anatomy of Massive Activations and Attention Sinkscited by 23
  3. 2025Cache Me If You Must: Adaptive Key-Value Quantization for Large Language Modelscited by 21
  4. 2025Quantization Meets dLLMs: A Systematic Study of Post-training Quantization for Diffusion LLMscited by 21
  5. 2024AsymKV: Enabling 1-Bit Quantization of KV Cache with Layer-Wise Asymmetric Quantization Configurationscited by 19
  6. 2025PM-KVQ: Progressive Mixed-precision KV Cache Quantization for Long-CoT LLMscited by 19
  7. 2025Probe Pruning: Accelerating LLMs through Dynamic Pruning via Model-Probingcited by 14
  8. 2025Outlier-Safe Pre-Training for Robust 4-Bit Quantization of Large Language Modelscited by 13
  9. 2024PrefixQuant: Eliminating Outliers by Prefixed Tokens for Large Language Models Quantizationcited by 11
  10. 2025Do we really have to filter out random noise in pre-training data for language models?cited by 11
  11. 2025ZeroTuning: Unlocking the Initial Token's Power to Enhance Large Language Models Without Trainingcited by 11
  12. 2025LRQ-DiT: Log-Rotation Post-Training Quantization of Diffusion Transformers for Image and Video Generationcited by 10
  13. 2025AKVQ-VL: Attention-Aware KV Cache Adaptive 2-Bit Quantization for Vision-Language Modelscited by 9
  14. 2026What Makes Low-Bit Quantization-Aware Training Work for Reasoning LLMs? A Systematic Studycited by 9
  15. 2026ResidualKV: Residual-Based KV Cache Compression for Efficient Long-Context Inferencecited by 8
  16. 2026DuQuant++: Fine-grained Rotation Enhances Microscaling FP4 Quantizationcited by 8
  17. 2025Rethinking Key-Value Cache Compression Techniques for Large Language Model Servingcited by 7
  18. 2025SQuat: Subspace-orthogonal KV Cache Quantizationcited by 7
  19. 2025Rethinking Causal Mask Attention for Vision-Language Inferencecited by 6
  20. 2024AlignedKV: Reducing Memory Access of KV-Cache with Precision-Aligned Quantizationcited by 4
  21. 2024Deploying Foundation Model Powered Agent Services: A Surveycited by 4
  22. 2025A Simple Linear Patch Revives Layer-Pruned Large Language Modelscited by 4
  23. 2025KVmix: Gradient-Based Layer Importance-Aware Mixed-Precision Quantization for KV Cachecited by 4
  24. 2025XQuant: Breaking the Memory Wall for LLM Inference with KV Cache Rematerializationcited by 4
  25. 2026Attention Sink Forges Native MoE in Attention Layers: Sink-Aware Training to Address Head Collapsecited by 4
  26. 2024Harnessing Your DRAM and SSD for Sustainable and Accessible LLM Inference with Mixed-Precision and Multi-level Cachingcited by 3
  27. 2025On the Existence and Behavior of Secondary Attention Sinkscited by 3
  28. 2026Predict, Don't Iterate: Efficient Adaptive-Length Infilling for Diffusion Language Modelscited by 3
  29. 2026One Token Is Enough: Improving Diffusion Language Models with a Sink Tokencited by 2
  30. 2026Search Your Block Floating Point Scales!cited by 2
  31. 2025Binary Neural Networks for Large Language Model: A Surveycited by 1
  32. 2025Outlier Smoothing with Closed-Form Rotations for W4A4 Large Language Model Quantizationcited by 1
  33. 2026SinkTrack: Attention Sink based Context Anchoring for Large Language Modelscited by 1
  34. 2025WindowKV: Task-Adaptive Group-Wise KV Cache Window Selection for Efficient LLM Inferencecited by 0
  35. 2026MALLOC: Benchmarking the Memory-aware Long Sequence Compression for Large Sequential Recommendationcited by 0
  36. 2026The Structural Origin of Attention Sink: Variance Discrepancy, Super Neurons, and Dimension Disparitycited by 0
  37. 2026InfoQuant: Shaping Activation Distributions for Low-Bit LLM Quantizationcited by 0
  38. 2026SemKV: Semantic Mixed-Precision KV Cache Quantization Guided by the Quality Cliff for Long-Context LLM Inferencecited by 0

50 most cited of 52 All 52 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2025MILLION: Mastering Long-Context LLM Inference Via Outlier-Immunized KV Product Quantization
  2. 2024LLMC: Benchmarking Large Language Model Quantization with a Versatile Compression Toolkit
  3. 2023LLM-QAT: Data-Free Quantization Aware Training for Large Language Models
  4. 2024WKVQuant: Quantizing Weight and Key/Value Cache for Large Language Models Gains More
  5. 2026High-accuracy Low-Bit KV-Cache Quantization via Local Distribution Restoration
  6. 2025Mixed-Precision Quantization for Language Models: Techniques and Prospects
  7. 2025Cocktail: Chunk-Adaptive Mixed-Precision Quantization for Long-Context LLM Inference
  8. 2025Rethinking the Outlier Distribution in Large Language Models: An In-depth Study