mlsanityData to 7 Oct 2026

SplitZip: Ultra Fast Lossless KV Compression for Disaggregated LLM Serving

arXiv 2605.01708 ↗First posted 3 May 2026cs.DC cs.AI cs.LGcited by 3

On the map: Efficient LLM inference and serving → KV cache optimization

Abstract (arXiv)

Contemporary systems serving large language models (LLMs) have adopted prefill-decode disaggregation to load-balance between the compute-bound prefill phase and the memory-bound decode phase. Under this design, prefill workers generate a KV cache that must be transferred to decode workers before generation can begin. With these workers residing on different physical systems, this transfer becomes a significant bottleneck to serving LLMs at scale, especially for long-input and agentic workloads. Existing lossless codecs are unsuitable here as they primarily target offline weight compression, run on CPUs, or use variable-length coding whose compression cannot keep up with KV production during prefill. We introduce SplitZip, a GPU-friendly lossless compressor for KV cache transfer that preserves KV tensors bitwise and integrates into existing serving frameworks without modifying model execution. SplitZip exploits redundancy in floating-point exponents of KV activations, encoding frequent exponent values with fixed-length codes and routing rare exponents through a sparse escape stream of (position, value). A calibrated top-16 exponent codebook eliminates online histogramming, while the regular dense path and sparse escape correction make both encoding and decoding efficient on GPUs. On real BF16 activation tensors, SplitZip achieves 613.3 GB/s compression throughput and 2181.8 GB/s decompression throughput, outperforming prior lossless compressors on the critical codec path. End-to-end transfer experiments show up to 1.32× speedup for BF16 KV cache transfer, 1.30× speedup for TTFT, and 1.23× increase in Request Throughput. The same approach extends to FP8 KV caches, providing up to 1.14× compression over native E5M2. Code is available at https://github.com/Intelligent-Microsystems-Lab/SplitZip

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving. Builds on 25 of its 31 references that are in the map. First posted on arXiv 3 May 2026.

Builds on

25 of 31 references.

Closest ancestors · 5

Same direction, newest first

  1. 2026Prefill-as-a-Service: KVCache of Next-Generation Models Could Go Cross-Datacenter16 Apr
  2. 2026ZipServ: Fast and Memory-Efficient LLM Inference with Hardware-Aware Lossless Compression18 Mar
  3. 2025Huff-LLM: End-to-End Lossless Compression for Efficient LLM Inference2 Feb
  4. 2024SnapKV: LLM Knows What You are Looking for Before Generation22 Apr
  5. 2023H2O: Heavy-Hitter Oracle for Efficient Generative Inference of Large Language Models24 Jun

Other references · 20

  1. 2021Training Verifiers to Solve Math Word Problemscited by 9,743
  2. 2021Evaluating Large Language Models Trained on Codecited by 8,053
  3. 2020Measuring Massive Multitask Language Understandingcited by 7,768
  4. 2020Longformer: The Long-Document Transformercited by 3,362
  5. 2016Pointer Sentinel Mixture Modelscited by 2,992
  6. 2023Efficient Streaming Language Models with Attention Sinkscited by 2,115
  7. 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Servingcited by 595
  8. 2024KVQuant: Towards 10 Million Context Length LLM Inference with KV Cache Quantizationcited by 571
  9. 2023Splitwise: Efficient generative LLM inference using phase splittingcited by 536
  10. 2024Mooncake: A KVCache-centric Disaggregated Architecture for LLM Servingcited by 193
  11. 2024KV Cache is 1 Bit Per Channel: Efficient Large Language Model Inference with Coupled Quantizationcited by 74
  12. 2019Head-Driven Phrase Structure Grammar Parsing on Penn Treebankcited by 43
8 more
  1. 202570% Size, 100% Accuracy: Lossless LLM Compression for Efficient GPU Inference via Dynamic-Length Float (DFloat11)cited by 23
  2. 2025PM-KVQ: Progressive Mixed-precision KV Cache Quantization for Long-CoT LLMscited by 19
  3. 2025FlowKV: A Disaggregated Inference Framework with Low-Latency KV Cache Transfer and Load-Aware Schedulingcited by 10
  4. 2025KVLinC : KV Cache Quantization with Hadamard Rotation and Linear Correctioncited by 9
  5. 2025ZipLLM: Efficient LLM Storage via Model-Aware Synergistic Data Deduplication and Compressioncited by 5
  6. 2026Single-Stage Huffman Encoder for ML Compressioncited by 2
  7. 2025A High-Throughput GPU Framework for Adaptive Lossless Compression of Floating-Point Datacited by 1
  8. 2026Quad Length Codes for Lossless Compression of e4m3cited by 1

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 3.

3 papers.

  1. 2026SpectrumKV: Per-Token Mixed-Precision KV Cache Transfer for Prefill-Decode Disaggregated LLM Servingcited by 1
  2. 2026Unified AI Gateway: A Framework for Joint Model Routing and KV Cache Managementcited by 0
  3. 2026D-Quant: Driftable Entropy Coding for KV Cache Quantizationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026VeriCache: Turning Lossy KV Cache into Lossless LLM Inference
  2. 2026KV-CoRE: Benchmarking Data-Dependent Low-Rank Compressibility of KV-Caches in LLMs
  3. 2025TraCT: Disaggregated LLM Serving with CXL Shared Memory KV Cache at Rack-Scale
  4. 2025TRACE: Unlocking Effective CXL Bandwidth via Lossless Compression and Precision Scaling
  5. 2026Fast KVzip: Efficient and Accurate LLM Inference with Gated KV Eviction
  6. 2024KVDirect: Distributed Disaggregated LLM Inference
  7. 2026More GPUs or a Smaller Cache? Tensor Parallelism versus KV Compression for Memory-Bound LLM Serving
  8. 2025PackKV: Reducing KV Cache Memory Footprint through LLM-Aware Lossy Compression