# An Efficient Compression of Deep Neural Network Checkpoints Based on Prediction and Context Modeling

arXiv: https://arxiv.org/abs/2506.12000 · HTML: https://mlsanity.com/paper/2506.12000/

## Key facts

- Cited by 2 papers in the map (data through 14 Oct 2026).
- Builds on 8 of its 14 references that are in the map.
- First posted on arXiv 13 Jun 2025.

## Builds on (most cited first, 8 of 8)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [The Pile: An 800GB Dataset of Diverse Text for Language Modeling](https://mlsanity.com/paper/2101.00027/) (2020)
- [Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling](https://mlsanity.com/paper/2304.01373/) (2023)
- [A Fast Transformer-based General-Purpose Lossless Compressor](https://mlsanity.com/paper/2203.16114/) (2022)
- [On Efficient Constructions of Checkpoints](https://mlsanity.com/paper/2009.13003/) (2020)
- [DZip: improved general-purpose lossless compression based on novel neural network modeling](https://mlsanity.com/paper/1911.03572/) (2019)
- [ExCP: Extreme LLM Checkpoint Compression via Weight-Momentum Joint Shrinking](https://mlsanity.com/paper/2406.11257/) (2024)

## Built on by (most cited first, 2 of 2)

- [A Model-Driven Lossless Compression Algorithm Resistant to Mismatch](https://mlsanity.com/paper/2601.17684/) (2026)
- [ULPack: Reference-Conditioned Lossless Compression of Checkpoint Trajectories](https://mlsanity.com/paper/2610.14184/) (2026)

## Abstract (arXiv)

This paper is dedicated to an efficient compression of weights and optimizer states (called checkpoints) obtained at different stages during a neural network training process. First, we propose a prediction-based compression approach, where values from the previously saved checkpoint are used for context modeling in arithmetic coding. Second, in order to enhance the compression performance, we also propose to apply pruning and quantization of the checkpoint values. Experimental results show that our approach achieves substantial bit size reduction, while enabling near-lossless training recovery from restored checkpoints, preserving the model's performance and making it suitable for storage-limited environments.