When Quantization Is Free: An int4 KV Cache That Outruns fp16 on Apple Silicon
On the map: Efficient LLM inference and serving → KV cache quantization
Abstract (arXiv)
KV-cache quantization is framed as a quality--latency trade-off. We show it is inverted on Apple Silicon's unified memory: a single fused Metal kernel (sign-randomized FFT + per-channel λ + per-group abs-max + int4 nibble pack), exposed as a HuggingFace textttCache subclass, runs faster than fp16 across 256--4096-token prefixes on Gemma-3 1B (-3 to -8% ms/tok) and at short context on Qwen2.5-1.5B (-0.7 to -2.6% through 1K), with 3× persistent memory compression and quality preserved (dPPL = 0.000 Qwen short-prompt; +3.6 hook dPPL Gemma). The kernel's ~!25,ns/vec overhead is below the bandwidth savings from 3× compression. The fused kernel also closes Qwen's 4-bit per-token catastrophe (dPPL = +7975 → +638.6, 12.5× reduction) at 182,GFLOPS / D=128. Supporting findings: SRFT and SRHT are statistically indistinguishable for KV quality (we pick SRFT for mixed-radix and matrix-multiply alignment); a learned-rotation ablation surfaces a regularization role for the fixed random SRFT base (learning R+λ without SRFT lowers calibration MSE 84.9% vs 50.3% but yields worse PPL); Householder rotations at k=d/2 reflectors are effectively lossless at d=256.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in KV cache quantization, one of 12 directions in Efficient LLM inference and serving. Builds on 18 of its 33 references that are in the map. First posted on arXiv 7 May 2026.
Builds on
18 of 33 references.
Closest ancestors · 3
Same direction, newest first
- 2024SpinQuant: LLM quantization with learned rotations26 May
- 2024QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMs30 Mar
- 2024KIVI: A Tuning-Free Asymmetric 2bit Quantization for KV Cache5 Feb
Other references · 15
- 2024The Llama 3 Herd of Modelscited by 14,192
- 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
- 2024Qwen2 Technical Reportcited by 2,076
- 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,838
- 2022LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scalecited by 881
- 2023QuIP: 2-Bit Quantization of Large Language Models With Guaranteescited by 397
- 2024QuIP#: Even Better LLM Quantization with Hadamard Incoherence and Lattice Codebookscited by 354
- 2025SmolLM2: When Smol Goes Big -- Data-Centric Training of a Small Language Modelcited by 307
- 2025TurboQuant: Online Vector Quantization with Near-optimal Distortion Ratecited by 116
- 2025FPTQuant: Function-Preserving Transforms for LLM Quantizationcited by 16
- 2025DartQuant: Efficient Rotational Distribution Calibration for LLM Quantizationcited by 14
- 2025KurTail : Kurtosis-based LLM Quantizationcited by 12
3 more
- 2025ButterflyQuant: Ultra-low-bit LLM Quantization through Learnable Orthogonal Butterfly Transformscited by 8
- 2025Beyond Homogeneous Attention: Memory-Efficient LLMs via Fourier-Approximated KV Cachecited by 5
- 2026Beating vDSP: A 138 GFLOPS Radix-8 Stockham FFT on Apple Silicon via Two-Tier Register-Threadgroup Memory Decompositioncited by 1
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2026SAW-INT4: System-Aware 4-Bit KV-Cache Quantization for Real-World LLM Serving
- 2026TurboAngle: Near-Lossless KV Cache Compression via Uniform Angle Quantization
- 2026OScaR: The Occam's Razor for Extreme KV Cache Quantization in LLMs and Beyond
- 2026RateQuant: Optimal Mixed-Precision KV Cache Quantization via Rate-Distortion Theory
- 2024AsymKV: Enabling 1-Bit Quantization of KV Cache with Layer-Wise Asymmetric Quantization Configurations
- 2026Quality Recovery for Quantized KV Caches via Low-Rank Attention Adaptation
- 2025Titanus: Enabling KV Cache Pruning and Quantization On-the-Fly for LLM Acceleration
- 2025Kitty: Accurate and Efficient 2-bit KV Cache Quantization with Dynamic Channel-wise Precision Boost