# Lossless Compression for LLM Tensor Incremental Snapshots

arXiv: https://arxiv.org/abs/2505.09810 · HTML: https://mlsanity.com/paper/2505.09810/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 8 of its 21 references that are in the map.
- First posted on arXiv 14 May 2025.

## Builds on (most cited first, 8 of 8)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [BLOOM: A 176B-Parameter Open-Access Multilingual Language Model](https://mlsanity.com/paper/2211.05100/) (2022)
- [The MultiBERTs: BERT Reproductions for Robustness Analysis](https://mlsanity.com/paper/2106.16163/) (2021)
- [Time series compression: a survey](https://mlsanity.com/paper/2101.08784/) (2021)
- [A Comprehensive Survey of Compression Algorithms for Language Models](https://mlsanity.com/paper/2401.15347/) (2024)
- [CPR: Understanding and Improving Failure Tolerant Training for Deep Learning Recommendation with Partial Recovery](https://mlsanity.com/paper/2011.02999/) (2020)
- [FCBench: Cross-Domain Benchmarking of Lossless Compression for Floating-Point Data](https://mlsanity.com/paper/2312.10301/) (2023)
- [Lossless and Near-Lossless Compression for Foundation Models](https://mlsanity.com/paper/2404.15198/) (2024)

## Built on by (most cited first, 4 of 4)

- [ZipServ: Fast and Memory-Efficient LLM Inference with Hardware-Aware Lossless Compression](https://mlsanity.com/paper/2603.17435/) (2026)
- [ZipCCL: Efficient Lossless Data Compression of Communication Collectives for Accelerating LLM Training](https://mlsanity.com/paper/2604.27844/) (2026)
- [NeuralZip: Reusable Setup for Fast Lossless Compression](https://mlsanity.com/paper/2610.09916/) (2026)
- [ULPack: Reference-Conditioned Lossless Compression of Checkpoint Trajectories](https://mlsanity.com/paper/2610.14184/) (2026)

## Abstract (arXiv)

During the training of Large Language Models (LLMs), tensor data is periodically "checkpointed" to persistent storage to allow recovery of work done in the event of failure. The volume of data that must be copied during each checkpoint, even when using reduced-precision representations such as bfloat16, often reaches hundreds of gigabytes. Furthermore, the data must be moved across a network and written to a storage system before the next epoch occurs. With a view to ultimately building an optimized checkpointing solution, this paper presents experimental analysis of checkpoint data used to derive a design that maximizes the use of lossless compression to reduce the volume of data. We examine how tensor data and its compressibility evolve during model training and evaluate the efficacy of existing common off-the-shelf general purpose compression engines combined with known data optimization techniques such as byte-grouping and incremental delta compression. Leveraging our analysis we have built an effective compression solution, known as Language Model Compressor (LMC), which is based on byte-grouping and Huffman encoding. LMC offers more compression performance than the best alternative (BZ2) but with an order-of-magnitude reduction in the time needed to perform the compression. We show that a 16-core parallel implementation of LMC can attain compression and decompression throughput of 2.78 GiB/s and 3.76 GiB/s respectively. This increase in performance ultimately reduces the CPU resources needed and provides more time to copy the data to the storage system before the next epoch thus allowing for higher-frequency checkpoints.