# SplitZip: Ultra Fast Lossless KV Compression for Disaggregated LLM Serving

arXiv: https://arxiv.org/abs/2605.01708 · HTML: https://mlsanity.com/paper/2605.01708/

## Key facts

- Cited by 3 papers in the map (data through 7 Oct 2026).
- Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving.
- Builds on 25 of its 31 references that are in the map.
- First posted on arXiv 3 May 2026.

## Placed in

- Efficient LLM inference and serving / [KV cache optimization](https://mlsanity.com/inference/kv-cache-optimization/)

## Builds on (most cited first, 25 of 25)

- [Training Verifiers to Solve Math Word Problems](https://mlsanity.com/paper/2110.14168/) (2021)
- [Evaluating Large Language Models Trained on Code](https://mlsanity.com/paper/2107.03374/) (2021)
- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [Longformer: The Long-Document Transformer](https://mlsanity.com/paper/2004.05150/) (2020)
- [Pointer Sentinel Mixture Models](https://mlsanity.com/paper/1609.07843/) (2016)
- [Efficient Streaming Language Models with Attention Sinks](https://mlsanity.com/paper/2309.17453/) (2023)
- [SnapKV: LLM Knows What You are Looking for Before Generation](https://mlsanity.com/paper/2404.14469/) (2024)
- [H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models](https://mlsanity.com/paper/2306.14048/) (2023)
- [DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Serving](https://mlsanity.com/paper/2401.09670/) (2024)
- [KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantization](https://mlsanity.com/paper/2401.18079/) (2024)
- [Splitwise: Efficient generative LLM inference using phase splitting](https://mlsanity.com/paper/2311.18677/) (2023)
- [Mooncake: A KVCache-centric Disaggregated Architecture for LLM Serving](https://mlsanity.com/paper/2407.00079/) (2024)
- [KV Cache is 1 Bit Per Channel: Efficient Large Language Model Inference with Coupled Quantization](https://mlsanity.com/paper/2405.03917/) (2024)
- [Head-Driven Phrase Structure Grammar Parsing on Penn Treebank](https://mlsanity.com/paper/1907.02684/) (2019)
- [70% Size, 100% Accuracy: Lossless LLM Compression for Efficient GPU Inference via Dynamic-Length Float (DFloat11)](https://mlsanity.com/paper/2504.11651/) (2025)
- [PM-KVQ: Progressive Mixed-precision KV Cache Quantization for Long-CoT LLMs](https://mlsanity.com/paper/2505.18610/) (2025)
- [Prefill-as-a-Service: KVCache of Next-Generation Models Could Go Cross-Datacenter](https://mlsanity.com/paper/2604.15039/) (2026)
- [FlowKV: A Disaggregated Inference Framework with Low-Latency KV Cache Transfer and Load-Aware Scheduling](https://mlsanity.com/paper/2504.03775/) (2025)
- [KVLinC : KV Cache Quantization with Hadamard Rotation and Linear Correction](https://mlsanity.com/paper/2510.05373/) (2025)
- [ZipServ: Fast and Memory-Efficient LLM Inference with Hardware-Aware Lossless Compression](https://mlsanity.com/paper/2603.17435/) (2026)
- [Huff-LLM: End-to-End Lossless Compression for Efficient LLM Inference](https://mlsanity.com/paper/2502.00922/) (2025)
- [ZipLLM: Efficient LLM Storage via Model-Aware Synergistic Data Deduplication and Compression](https://mlsanity.com/paper/2505.06252/) (2025)
- [Single-Stage Huffman Encoder for ML Compression](https://mlsanity.com/paper/2601.10673/) (2026)
- [A High-Throughput GPU Framework for Adaptive Lossless Compression of Floating-Point Data](https://mlsanity.com/paper/2511.04140/) (2025)
- [Quad Length Codes for Lossless Compression of e4m3](https://mlsanity.com/paper/2602.17849/) (2026)

## Built on by (most cited first, 3 of 3)

- [SpectrumKV: Per-Token Mixed-Precision KV Cache Transfer for Prefill-Decode Disaggregated LLM Serving](https://mlsanity.com/paper/2606.08635/) (2026)
- [Unified AI Gateway: A Framework for Joint Model Routing and KV Cache Management](https://mlsanity.com/paper/2609.06940/) (2026)
- [D-Quant: Driftable Entropy Coding for KV Cache Quantization](https://mlsanity.com/paper/2609.19880/) (2026)

## Abstract (arXiv)

Contemporary systems serving large language models (LLMs) have adopted prefill-decode disaggregation to load-balance between the compute-bound prefill phase and the memory-bound decode phase. Under this design, prefill workers generate a KV cache that must be transferred to decode workers before generation can begin. With these workers residing on different physical systems, this transfer becomes a significant bottleneck to serving LLMs at scale, especially for long-input and agentic workloads. Existing lossless codecs are unsuitable here as they primarily target offline weight compression, run on CPUs, or use variable-length coding whose compression cannot keep up with KV production during prefill. We introduce SplitZip, a GPU-friendly lossless compressor for KV cache transfer that preserves KV tensors bitwise and integrates into existing serving frameworks without modifying model execution. SplitZip exploits redundancy in floating-point exponents of KV activations, encoding frequent exponent values with fixed-length codes and routing rare exponents through a sparse escape stream of (position, value). A calibrated top-16 exponent codebook eliminates online histogramming, while the regular dense path and sparse escape correction make both encoding and decoding efficient on GPUs. On real BF16 activation tensors, SplitZip achieves 613.3 GB/s compression throughput and 2181.8 GB/s decompression throughput, outperforming prior lossless compressors on the critical codec path. End-to-end transfer experiments show up to 1.32× speedup for BF16 KV cache transfer, 1.30× speedup for TTFT, and 1.23× increase in Request Throughput. The same approach extends to FP8 KV caches, providing up to 1.14× compression over native E5M2. Code is available at https://github.com/Intelligent-Microsystems-Lab/SplitZip