mlsanityData to 7 Oct 2026

ZipNN: Lossless Compression for AI Models

arXiv 2411.05239 ↗First posted 7 Nov 2024cs.LG cs.IT math.ITcited by 12

Not on the map. Why?

Abstract (arXiv)

With the growth of model sizes and the scale of their deployment, their sheer size burdens the infrastructure requiring more network and more storage to accommodate these. While there is a vast model compression literature deleting parts of the model weights for faster inference, we investigate a more traditional type of compression - one that represents the model in a compact form and is coupled with a decompression algorithm that returns it to its original form and size - namely lossless compression. We present ZipNN a lossless compression tailored to neural networks. Somewhat surprisingly, we show that specific lossless compression can gain significant network and storage reduction on popular models, often saving 33% and at times reducing over 50% of the model size. We investigate the source of model compressibility and introduce specialized compression variants tailored for models that further increase the effectiveness of compression. On popular models (e.g. Llama 3) ZipNN shows space savings that are over 17% better than vanilla compression while also improving compression and decompression speeds by 62%. We estimate that these methods could save over an ExaByte per month of network traffic downloaded from a large model hub like Hugging Face.

Key facts. Cited by 12 papers in the map (data through 7 Oct 2026). Builds on 39 of its 76 references that are in the map. First posted on arXiv 7 Nov 2024.

Builds on

39 of 76 references.

References · 39

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2024The Llama 3 Herd of Modelscited by 14,192
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  4. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  5. 2023Mistral 7Bcited by 3,065
  6. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,014
  7. 2015Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Codingcited by 2,291
  8. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,201
  9. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,838
  10. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
  11. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,734
  12. 2020Knowledge Distillation: A Surveycited by 1,578
27 more
  1. 2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,366
  2. 2022Editing Models with Task Arithmeticcited by 1,076
  3. 2021A Survey of Quantization Methods for Efficient Neural Network Inferencecited by 798
  4. 2023TIES-Merging: Resolving Interference When Merging Modelscited by 795
  5. 2020Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuningcited by 658
  6. 2023PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallelcited by 641
  7. 2021Merging Models with Fisher-Weighted Averagingcited by 613
  8. 2017To prune, or not to prune: exploring the efficacy of pruning for model compressioncited by 596
  9. 2024GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projectioncited by 447
  10. 2020AdapterHub: A Framework for Adapting Transformerscited by 441
  11. 2020Fine-Tuning Pretrained Language Models: Weight Initializations, Data Orders, and Early Stoppingcited by 377
  12. 2018Model compression via distillation and quantizationcited by 287
  13. 2018Marian: Fast Neural Machine Translation in C++cited by 202
  14. 2023ReLoRA: High-Rank Training Through Low-Rank Updatescited by 200
  15. 2022Branch-Train-Merge: Embarrassingly Parallel Training of Expert Language Modelscited by 187
  16. 2021Bayesian Optimization is Superior to Random Search for Machine Learning Hyperparameter Tuning: Analysis of the Black-Box Optimization Challenge 2020cited by 146
  17. 2022Fusing finetuned models for better pretrainingcited by 110
  18. 2023LLM360: Towards Fully Transparent Open-Source LLMscited by 100
  19. 2021Distributed Deep Learning in Open Collaborationscited by 58
  20. 2023Knowledge is a Region in Weight Space for Fine-tuned Language Modelscited by 58
  21. 2019The Knowledge Within: Methods for Data-Free Model Compressioncited by 54
  22. 2022ColD Fusion: Collaborative Descent for Distributed Multitask Finetuningcited by 52
  23. 2019Scalable Model Compression by Entropy Penalized Reparameterizationcited by 31
  24. 2021Effective Model Sparsification by Scheduled Grow-and-Prune Methodscited by 25
  25. 2023Training trajectories, mini-batch losses and the curious role of the learning ratecited by 20
  26. 2023Git-Theta: A Git Extension for Collaborative Development of Machine Learning Modelscited by 15
  27. 2023GraVAC: Adaptive Compression for Communication-Efficient Distributed DL Trainingcited by 10

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 1; 2026: 11.

12 papers.

  1. 202570% Size, 100% Accuracy: Lossless LLM Compression for Efficient GPU Inference via Dynamic-Length Float (DFloat11)cited by 23
  2. 2026ZipServ: Fast and Memory-Efficient LLM Inference with Hardware-Aware Lossless Compressioncited by 8
  3. 2026ZipMoE: Efficient On-Device MoE Serving via Lossless Compression and Cache-Affinity Schedulingcited by 4
  4. 2026Statistically-Lossless Quantization of Large Language Modelscited by 4
  5. 2026ENEC: A Lossless AI Model Compression Method Enabling Fast Inference on Ascend NPUscited by 3
  6. 2026UCCL-Zip: Lossless Compression Supercharged GPU Communicationcited by 2
  7. 2026ZipCCL: Efficient Lossless Data Compression of Communication Collectives for Accelerating LLM Trainingcited by 1
  8. 2026Embedding Compression via Spherical Coordinatescited by 0
  9. 2026FluxMoE: Decoupling Expert Residency for High-Performance MoE Servingcited by 0
  10. 2026TStore: Rethinking AI Model Hub with Tensor-Centric Compressioncited by 0
  11. 2026Lossless Tensor Compression as Program Synthesiscited by 0
  12. 2026EntroPack: Fast and Accurate Entropy-Coded Weight Compression at Arbitrary Bitratescited by 0