# ZipNN: Lossless Compression for AI Models

arXiv: https://arxiv.org/abs/2411.05239 · HTML: https://mlsanity.com/paper/2411.05239/

## Key facts

- Cited by 14 papers in the map (data through 14 Oct 2026).
- Builds on 39 of its 76 references that are in the map.
- First posted on arXiv 7 Nov 2024.

## Builds on (most cited first, 30 of 39)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [The Llama 3 Herd of Models](https://mlsanity.com/paper/2407.21783/) (2024)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [QLoRA: Efficient Finetuning of Quantized LLMs](https://mlsanity.com/paper/2305.14314/) (2023)
- [Mistral 7B](https://mlsanity.com/paper/2310.06825/) (2023)
- [Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity](https://mlsanity.com/paper/2101.03961/) (2021)
- [Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding](https://mlsanity.com/paper/1510.00149/) (2015)
- [HuggingFace's Transformers: State-of-the-art Natural Language Processing](https://mlsanity.com/paper/1910.03771/) (2019)
- [GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers](https://mlsanity.com/paper/2210.17323/) (2022)
- [Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism](https://mlsanity.com/paper/1909.08053/) (2019)
- [Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling](https://mlsanity.com/paper/2304.01373/) (2023)
- [Knowledge Distillation: A Survey](https://mlsanity.com/paper/2006.05525/) (2020)
- [Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time](https://mlsanity.com/paper/2203.05482/) (2022)
- [Editing Models with Task Arithmetic](https://mlsanity.com/paper/2212.04089/) (2022)
- [TIES-Merging: Resolving Interference When Merging Models](https://mlsanity.com/paper/2306.01708/) (2023)
- [A Survey of Quantization Methods for Efficient Neural Network Inference](https://mlsanity.com/paper/2103.13630/) (2021)
- [Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuning](https://mlsanity.com/paper/2012.13255/) (2020)
- [PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallel](https://mlsanity.com/paper/2304.11277/) (2023)
- [Merging Models with Fisher-Weighted Averaging](https://mlsanity.com/paper/2111.09832/) (2021)
- [To prune, or not to prune: exploring the efficacy of pruning for model compression](https://mlsanity.com/paper/1710.01878/) (2017)
- [GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projection](https://mlsanity.com/paper/2403.03507/) (2024)
- [AdapterHub: A Framework for Adapting Transformers](https://mlsanity.com/paper/2007.07779/) (2020)
- [Fine-Tuning Pretrained Language Models: Weight Initializations, Data Orders, and Early Stopping](https://mlsanity.com/paper/2002.06305/) (2020)
- [Model compression via distillation and quantization](https://mlsanity.com/paper/1802.05668/) (2018)
- [ReLoRA: High-Rank Training Through Low-Rank Updates](https://mlsanity.com/paper/2307.05695/) (2023)
- [Marian: Fast Neural Machine Translation in C++](https://mlsanity.com/paper/1804.00344/) (2018)
- [Branch-Train-Merge: Embarrassingly Parallel Training of Expert Language Models](https://mlsanity.com/paper/2208.03306/) (2022)
- [Bayesian Optimization is Superior to Random Search for Machine Learning Hyperparameter Tuning: Analysis of the Black-Box Optimization Challenge 2020](https://mlsanity.com/paper/2104.10201/) (2021)
- [Fusing finetuned models for better pretraining](https://mlsanity.com/paper/2204.03044/) (2022)
- [LLM360: Towards Fully Transparent Open-Source LLMs](https://mlsanity.com/paper/2312.06550/) (2023)

## Built on by (most cited first, 14 of 14)

- [70% Size, 100% Accuracy: Lossless LLM Compression for Efficient GPU Inference via Dynamic-Length Float (DFloat11)](https://mlsanity.com/paper/2504.11651/) (2025)
- [ZipServ: Fast and Memory-Efficient LLM Inference with Hardware-Aware Lossless Compression](https://mlsanity.com/paper/2603.17435/) (2026)
- [Statistically-Lossless Quantization of Large Language Models](https://mlsanity.com/paper/2605.02404/) (2026)
- [ZipMoE: Efficient On-Device MoE Serving via Lossless Compression and Cache-Affinity Scheduling](https://mlsanity.com/paper/2601.21198/) (2026)
- [ENEC: A Lossless AI Model Compression Method Enabling Fast Inference on Ascend NPUs](https://mlsanity.com/paper/2604.03298/) (2026)
- [UCCL-Zip: Lossless Compression Supercharged GPU Communication](https://mlsanity.com/paper/2604.17172/) (2026)
- [ZipCCL: Efficient Lossless Data Compression of Communication Collectives for Accelerating LLM Training](https://mlsanity.com/paper/2604.27844/) (2026)
- [Lossless Tensor Compression as Program Synthesis](https://mlsanity.com/paper/2608.02162/) (2026)
- [Embedding Compression via Spherical Coordinates](https://mlsanity.com/paper/2602.00079/) (2026)
- [FluxMoE: Decoupling Expert Residency for High-Performance MoE Serving](https://mlsanity.com/paper/2604.02715/) (2026)
- [TensorDex: A Compact, Tensor-Centric Storage System for Modern AI Models](https://mlsanity.com/paper/2604.17104/) (2026)
- [EntroPack: Fast and Accurate Entropy-Coded Weight Compression at Arbitrary Bitrates](https://mlsanity.com/paper/2609.34185/) (2026)
- [NeuralZip: Reusable Setup for Fast Lossless Compression](https://mlsanity.com/paper/2610.09916/) (2026)
- [ULPack: Reference-Conditioned Lossless Compression of Checkpoint Trajectories](https://mlsanity.com/paper/2610.14184/) (2026)

## Abstract (arXiv)

With the growth of model sizes and the scale of their deployment, their sheer size burdens the infrastructure requiring more network and more storage to accommodate these. While there is a vast model compression literature deleting parts of the model weights for faster inference, we investigate a more traditional type of compression - one that represents the model in a compact form and is coupled with a decompression algorithm that returns it to its original form and size - namely lossless compression. We present ZipNN a lossless compression tailored to neural networks. Somewhat surprisingly, we show that specific lossless compression can gain significant network and storage reduction on popular models, often saving 33% and at times reducing over 50% of the model size. We investigate the source of model compressibility and introduce specialized compression variants tailored for models that further increase the effectiveness of compression. On popular models (e.g. Llama 3) ZipNN shows space savings that are over 17% better than vanilla compression while also improving compression and decompression speeds by 62%. We estimate that these methods could save over an ExaByte per month of network traffic downloaded from a large model hub like Hugging Face.