mlsanityData to 14 Oct 2026

Lossless and Near-Lossless Compression for Foundation Models

arXiv 2404.15198 ↗First posted 5 Apr 2024cs.LG cs.IT math.ITcited by 5

Not on the map. Why?

Abstract (arXiv)

With the growth of model sizes and scale of their deployment, their sheer size burdens the infrastructure requiring more network and more storage to accommodate these. While there is a vast literature about reducing model sizes, we investigate a more traditional type of compression -- one that compresses the model to a smaller form and is coupled with a decompression algorithm that returns it to its original size -- namely lossless compression. Somewhat surprisingly, we show that such lossless compression can gain significant network and storage reduction on popular models, at times reducing over 50% of the model size. We investigate the source of model compressibility, introduce compression variants tailored for models and categorize models to compressibility groups. We also introduce a tunable lossy compression technique that can further reduce size even on the less compressible models with little to no effect on the model accuracy. We estimate that these methods could save over an ExaByte per month of network traffic downloaded from a large model hub like HuggingFace.

Key facts. Cited by 5 papers in the map (data through 14 Oct 2026). Builds on 52 of its 85 references that are in the map. First posted on arXiv 5 Apr 2024.

Builds on

52 of 85 references.

References · 52

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  3. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,599
  5. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  6. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 13,028
  7. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,148
  8. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,418
  9. 2023Mistral 7Bcited by 3,164
  10. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
  11. 2015Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Codingcited by 2,329
  12. 2022BLOOM: A 176B-Parameter Open-Access Multilingual Language Modelcited by 1,984
38 more
  1. 2022GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformerscited by 1,919
  2. 2018A Call for Clarity in Reporting BLEU Scorescited by 1,799
  3. 2023Pythia: A Suite for Analyzing Large Language Models Across Training and Scalingcited by 1,786
  4. 2020Knowledge Distillation: A Surveycited by 1,603
  5. 2022Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference timecited by 1,396
  6. 2018Don't Give Me the Details, Just the Summary! Topic-Aware Convolutional Neural Networks for Extreme Summarizationcited by 1,317
  7. 2023AWQ: Activation-aware Weight Quantization for LLM Compression and Accelerationcited by 1,298
  8. 2022Editing Models with Task Arithmeticcited by 1,120
  9. 2023TIES-Merging: Resolving Interference When Merging Modelscited by 818
  10. 2021A Survey of Quantization Methods for Efficient Neural Network Inferencecited by 815
  11. 2016Abstractive Text Summarization Using Sequence-to-Sequence RNNs and Beyondcited by 812
  12. 2020Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuningcited by 671
  13. 2023PyTorch FSDP: Experiences on Scaling Fully Sharded Data Parallelcited by 660
  14. 2021Merging Models with Fisher-Weighted Averagingcited by 621
  15. 2017To prune, or not to prune: exploring the efficacy of pruning for model compressioncited by 603
  16. 2023The Falcon Series of Open Language Modelscited by 535
  17. 2024GaLore: Memory-Efficient LLM Training by Gradient Low-Rank Projectioncited by 463
  18. 2020AdapterHub: A Framework for Adapting Transformerscited by 444
  19. 2020Fine-Tuning Pretrained Language Models: Weight Initializations, Data Orders, and Early Stoppingcited by 384
  20. 2018Model compression via distillation and quantizationcited by 289
  21. 2022ASQA: Factoid Questions Meet Long-Form Answerscited by 234
  22. 2023Camels in a Changing Climate: Enhancing LM Adaptation with Tulu 2cited by 229
  23. 2023ReLoRA: High-Rank Training Through Low-Rank Updatescited by 204
  24. 2018Marian: Fast Neural Machine Translation in C++cited by 203
  25. 2022Branch-Train-Merge: Embarrassingly Parallel Training of Expert Language Modelscited by 191
  26. 2021Bayesian Optimization is Superior to Random Search for Machine Learning Hyperparameter Tuning: Analysis of the Black-Box Optimization Challenge 2020cited by 148
  27. 2023The Truth is in There: Improving Reasoning in Language Models with Layer-Selective Rank Reductioncited by 123
  28. 2022Fusing finetuned models for better pretrainingcited by 110
  29. 2023LLM360: Towards Fully Transparent Open-Source LLMscited by 104
  30. 2021Distributed Deep Learning in Open Collaborationscited by 58
  31. 2023Knowledge is a Region in Weight Space for Fine-tuned Language Modelscited by 58
  32. 2019The Knowledge Within: Methods for Data-Free Model Compressioncited by 54
  33. 2022ColD Fusion: Collaborative Descent for Distributed Multitask Finetuningcited by 52
  34. 2019Scalable Model Compression by Entropy Penalized Reparameterizationcited by 31
  35. 2021Effective Model Sparsification by Scheduled Grow-and-Prune Methodscited by 26
  36. 2019Non-Gaussianity of Stochastic Gradient Noisecited by 22
  37. 2023Training trajectories, mini-batch losses and the curious role of the learning ratecited by 21
  38. 2023ComPEFT: Compression for Communicating Parameter Efficient Updates via Sparsification and Quantizationcited by 18

50 most cited of 52 All 52 (JSON)

Built on by

Cited by year

Citing papers per year: 2024: 1; 2025: 4.

5 papers.

  1. 2024Compress then Serve: Serving Thousands of LoRA Adapters with Little Overheadcited by 24
  2. 2025Lossless Compression for LLM Tensor Incremental Snapshotscited by 4
  3. 2025GPT as a Monte Carlo Language Tree: A Probabilistic Perspectivecited by 0
  4. 2025Closer to Reality: Practical Semi-Supervised Federated Learning for Foundation Model Adaptationcited by 0
  5. 2025BitSnap: Checkpoint Sparsification and Quantization in LLM Trainingcited by 0