# When Quantization Is Free: An int4 KV Cache That Outruns fp16 on Apple Silicon

arXiv: https://arxiv.org/abs/2605.05699 · HTML: https://mlsanity.com/paper/2605.05699/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in KV cache quantization, one of 12 directions in Efficient LLM inference and serving.
- Builds on 18 of its 33 references that are in the map.
- First posted on arXiv 7 May 2026.

## Placed in

- Efficient LLM inference and serving / [KV cache quantization](https://mlsanity.com/inference/kv-cache-quantization/)

## Builds on (most cited first, 18 of 18)

- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awareness](https://mlsanity.com/paper/2205.14135/) (2022)
- [Qwen2 Technical Report](https://mlsanity.com/paper/2407.10671/) (2024)
- [GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers](https://mlsanity.com/paper/2210.17323/) (2022)
- [LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale](https://mlsanity.com/paper/2208.07339/) (2022)
- [KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache](https://mlsanity.com/paper/2402.02750/) (2024)
- [QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMs](https://mlsanity.com/paper/2404.00456/) (2024)
- [SpinQuant: LLM quantization with learned rotations](https://mlsanity.com/paper/2405.16406/) (2024)
- [QuIP: 2-Bit Quantization of Large Language Models With Guarantees](https://mlsanity.com/paper/2307.13304/) (2023)
- [QuIP#: Even Better LLM Quantization with Hadamard Incoherence and Lattice Codebooks](https://mlsanity.com/paper/2402.04396/) (2024)
- [SmolLM2: When Smol Goes Big -- Data-Centric Training of a Small Language Model](https://mlsanity.com/paper/2502.02737/) (2025)
- [TurboQuant: Online Vector Quantization with Near-optimal Distortion Rate](https://mlsanity.com/paper/2504.19874/) (2025)
- [DartQuant: Efficient Rotational Distribution Calibration for LLM Quantization](https://mlsanity.com/paper/2511.04063/) (2025)
- [FPTQuant: Function-Preserving Transforms for LLM Quantization](https://mlsanity.com/paper/2506.04985/) (2025)
- [KurTail : Kurtosis-based LLM Quantization](https://mlsanity.com/paper/2503.01483/) (2025)
- [ButterflyQuant: Ultra-low-bit LLM Quantization through Learnable Orthogonal Butterfly Transforms](https://mlsanity.com/paper/2509.09679/) (2025)
- [Beyond Homogeneous Attention: Memory-Efficient LLMs via Fourier-Approximated KV Cache](https://mlsanity.com/paper/2506.11886/) (2025)
- [Beating vDSP: A 138 GFLOPS Radix-8 Stockham FFT on Apple Silicon via Two-Tier Register-Threadgroup Memory Decomposition](https://mlsanity.com/paper/2603.27569/) (2026)

## Abstract (arXiv)

KV-cache quantization is framed as a quality--latency trade-off. We show it is inverted on Apple Silicon's unified memory: a single fused Metal kernel (sign-randomized FFT + per-channel λ + per-group abs-max + int4 nibble pack), exposed as a HuggingFace textttCache subclass, runs faster than fp16 across 256--4096-token prefixes on Gemma-3 1B (-3 to -8% ms/tok) and at short context on Qwen2.5-1.5B (-0.7 to -2.6% through 1K), with 3× persistent memory compression and quality preserved (dPPL = 0.000 Qwen short-prompt; +3.6 hook dPPL Gemma). The kernel's ~!25,ns/vec overhead is below the bandwidth savings from 3× compression. The fused kernel also closes Qwen's 4-bit per-token catastrophe (dPPL = +7975 → +638.6, 12.5× reduction) at 182,GFLOPS / D=128. Supporting findings: SRFT and SRHT are statistically indistinguishable for KV quality (we pick SRFT for mixed-radix and matrix-multiply alignment); a learned-rotation ablation surfaces a regularization role for the fixed random SRFT base (learning R+λ without SRFT lowers calibration MSE 84.9% vs 50.3% but yields worse PPL); Householder rotations at k=d/2 reflectors are effectively lossless at d=256.