mlsanityData to 7 Oct 2026

ZipServ: Fast and Memory-Efficient LLM Inference with Hardware-Aware Lossless Compression

arXiv 2603.17435 ↗First posted 18 Mar 2026cs.DC cs.AR cs.LGcited by 8

On the map: Efficient LLM inference and serving → KV cache optimization

Abstract (arXiv)

Lossless model compression holds tremendous promise for alleviating the memory and bandwidth bottlenecks in bit-exact Large Language Model (LLM) serving. However, existing approaches often result in substantial inference slowdowns due to fundamental design mismatches with GPU architectures: at the kernel level, variable-length bitstreams produced by traditional entropy codecs break SIMT parallelism; at the system level, decoupled pipelines lead to redundant memory traffic. We present ZipServ, a lossless compression framework co-designed for efficient LLM inference. ZipServ introduces Tensor-Core-Aware Triple Bitmap Encoding (TCA-TBE), a novel fixed-length format that enables constant-time, parallel decoding, together with a fused decompression-GEMM (ZipGEMM) kernel that decompresses weights on-the-fly directly into Tensor Core registers. This "load-compressed, compute-decompressed" design eliminates intermediate buffers and maximizes compute intensity. Experiments show that ZipServ reduces the model size by up to 30%, achieves up to 2.21x kernel-level speedup over NVIDIA's cuBLAS, and expedites end-to-end inference by an average of 1.22x over vLLM. ZipServ is the first lossless compression system that provides both storage savings and substantial acceleration for LLM inference on GPUs.

Key facts. Cited by 8 papers in the map (data through 7 Oct 2026). Placed in KV cache optimization, one of 12 directions in Efficient LLM inference and serving. Builds on 46 of its 80 references that are in the map. First posted on arXiv 18 Mar 2026.

Builds on

46 of 80 references.

Closest ancestors · 3

Same direction, newest first

  1. 2025Huff-LLM: End-to-End Lossless Compression for Efficient LLM Inference2 Feb
  2. 2023CacheGen: KV Cache Compression and Streaming for Fast Large Language Model Serving11 Oct
  3. 2023Efficient Memory Management for Large Language Model Serving with PagedAttention12 Sep

Other references · 43

  1. 2024The Llama 3 Herd of Modelscited by 14,192
  2. 2020Scaling Laws for Neural Language Modelscited by 6,411
  3. 2022FlashAttention: Fast and Memory-Efficient Exact Attention with IO-Awarenesscited by 3,611
  4. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  5. 2023FlashAttention-2: Faster Attention with Better Parallelism and Work Partitioningcited by 2,551
  6. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,838
  7. 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,234
  8. 2024Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preferencecited by 1,151
  9. 2023SGLang: Efficient Execution of Structured Language Model Programscited by 1,107
  10. 2023SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shotcited by 1,090
  11. 2024QuaRot: Outlier-Free 4-Bit Inference in Rotated LLMscited by 598
  12. 2024DistServe: Disaggregating Prefill and Decoding for Goodput-optimized Large Language Model Servingcited by 595
31 more
  1. 2024Taming Throughput-Latency Tradeoff in LLM Inference with Sarathi-Servecited by 507
  2. 2024FlashAttention-3: Fast and Accurate Attention with Asynchrony and Low-precisioncited by 507
  3. 2024SpinQuant: LLM quantization with learned rotationscited by 446
  4. 2023Atom: Low-bit Quantization for Efficient and Accurate LLM Servingcited by 255
  5. 2023PowerInfer: Fast Large Language Model Serving with a Consumer-grade GPUcited by 210
  6. 2023TPU v4: An Optically Reconfigurable Supercomputer for Machine Learning with Hardware Support for Embeddingscited by 198
  7. 2024Llumnix: Dynamic Scheduling for Large Language Model Servingcited by 167
  8. 2024Physics of Language Models: Part 3.3, Knowledge Capacity Scaling Lawscited by 137
  9. 2024DeepSpeed-FastGen: High-throughput Text Generation for LLMs via MII and DeepSpeed-Inferencecited by 106
  10. 2022LUT-GEMM: Quantized Matrix Multiplication based on LUTs for Efficient Inference in Large-Scale Generative Language Modelscited by 98
  11. 2024ServerlessLLM: Low-Latency Serverless Inference for Large Language Modelscited by 78
  12. 2025Quantization Hurts Reasoning? An Empirical Study on Quantized Reasoning Modelscited by 68
  13. 2024Physics of Language Models: Part 2.2, How to Learn From Mistakes on Grade-School Math Problemscited by 49
  14. 2024BESA: Pruning Large Language Models with Blockwise Parameter-Efficient Sparsity Allocationcited by 44
  15. 2025Quantization Meets Reasoning: Exploring LLM Low-Bit Quantization Degradation for Mathematical Reasoningcited by 36
  16. 2024STBLLM: Breaking the 1-Bit Barrier with Structured Binary LLMscited by 31
  17. 2024Benchmarking and Dissecting the Nvidia Hopper GPU Architecturecited by 23
  18. 202570% Size, 100% Accuracy: Lossless LLM Compression for Efficient GPU Inference via Dynamic-Length Float (DFloat11)cited by 23
  19. 2025Past-Future Scheduler for LLM Serving under SLA Guaranteescited by 20
  20. 2021Bolt: Bridging the Gap between Auto-tuners and Hardware-native Performancecited by 18
  21. 2025Can Compressed LLMs Truly Act? An Empirical Evaluation of Agentic Capabilities in LLM Compressioncited by 15
  22. 2023Flash-LLM: Enabling Cost-Effective and Highly-Efficient Large Generative Model Inference with Unstructured Sparsitycited by 14
  23. 2022Deep neural networks with dependent weights: Gaussian Process mixture limit, heavy tails, sparsity and compressibilitycited by 13
  24. 2024NeuZip: Memory-Efficient Training and Inference with Dynamic Compression of Neural Networkscited by 12
  25. 2024ZipNN: Lossless Compression for AI Modelscited by 12
  26. 2025Ecco: Improving Memory Bandwidth and Capacity for LLMs via Entropy-aware Cache Compressioncited by 12
  27. 2019Buddy Compression: Enabling Larger Memory for Deep Learning and HPC Workloads on GPUscited by 10
  28. 2025Unveiling the Mystery of Weight in Large Foundation Models: Gaussian Distribution Never Fadescited by 8
  29. 2024OAC: Output-adaptive Calibration for Accurate Post-training Quantizationcited by 7
  30. 2025Lossless Compression for LLM Tensor Incremental Snapshotscited by 2
  31. 2025DECA: A Near-Core LLM Decompression Accelerator Grounded on a 3D Roofline Modelcited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 8.

8 papers.

  1. 2026SplitZip: Ultra Fast Lossless KV Compression for Disaggregated LLM Servingcited by 3
  2. 2026UCCL-Zip: Lossless Compression Supercharged GPU Communicationcited by 2
  3. 2026ZipCCL: Efficient Lossless Data Compression of Communication Collectives for Accelerating LLM Trainingcited by 1
  4. 2026FluxMoE: Decoupling Expert Residency for High-Performance MoE Servingcited by 0
  5. 2026Cassandra: Enabling Reasoning LLMs at Edge via Self-Speculative Decodingcited by 0
  6. 2026Reducing the GPU Memory Bottleneck with Lossless Compression for ML -- Extendedcited by 0
  7. 2026Lossless Tensor Compression as Program Synthesiscited by 0
  8. 2026CIERA: Cross-Iteration Exponent Reuse for Lossless Allgather in Sharded MoE Trainingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Reimagining Memory Access for LLM Inference: Compression-Aware Memory Controller Design
  2. 2023Efficient LLM inference solution on Intel GPU
  3. 2025MCBP: A Memory-Compute Efficient LLM Inference Accelerator Leveraging Bit-Slice-enabled Sparsity and Repetitiveness
  4. 2026Unified Static-Dynamic Pruning for Efficient LLM Inference
  5. 2024GEAR: An Efficient KV Cache Compression Recipe for Near-Lossless Generative Inference of LLM
  6. 2025Speculate Deep and Accurate: Lossless and Training-Free Acceleration for Offloaded LLMs via Substitute Speculative Decoding
  7. 2026VeriCache: Turning Lossy KV Cache into Lossless LLM Inference
  8. 2024Faster and Lighter LLMs: A Survey on Current Challenges and Way Forward