# BitSnap: Checkpoint Sparsification and Quantization in LLM Training

arXiv: https://arxiv.org/abs/2511.12376 · HTML: https://mlsanity.com/paper/2511.12376/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 20 of its 26 references that are in the map.
- First posted on arXiv 15 Nov 2025.

## Builds on (most cited first, 20 of 20)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [PaLM: Scaling Language Modeling with Pathways](https://mlsanity.com/paper/2204.02311/) (2022)
- [OPT: Open Pre-trained Transformer Language Models](https://mlsanity.com/paper/2205.01068/) (2022)
- [Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelism](https://mlsanity.com/paper/1909.08053/) (2019)
- [AWQ: Activation-aware Weight Quantization for LLM Compression and Acceleration](https://mlsanity.com/paper/2306.00978/) (2023)
- [LLM-Pruner: On the Structural Pruning of Large Language Models](https://mlsanity.com/paper/2305.11627/) (2023)
- [To prune, or not to prune: exploring the efficacy of pruning for model compression](https://mlsanity.com/paper/1710.01878/) (2017)
- [8-bit Optimizers via Block-wise Quantization](https://mlsanity.com/paper/2110.02861/) (2021)
- [Model compression via distillation and quantization](https://mlsanity.com/paper/1802.05668/) (2018)
- [Adaptive Quantization for Deep Neural Network](https://mlsanity.com/paper/1712.01048/) (2017)
- [8-Bit Approximations for Parallelism in Deep Learning](https://mlsanity.com/paper/1511.04561/) (2015)
- [Memory Efficient Optimizers with 4-bit States](https://mlsanity.com/paper/2309.01507/) (2023)
- [Efficient Post-training Quantization with FP8 Formats](https://mlsanity.com/paper/2309.14592/) (2023)
- [TRANSOM: An Efficient Fault-Tolerant System for Training LLMs](https://mlsanity.com/paper/2310.10046/) (2023)
- [On Efficient Constructions of Checkpoints](https://mlsanity.com/paper/2009.13003/) (2020)
- [ExCP: Extreme LLM Checkpoint Compression via Weight-Momentum Joint Shrinking](https://mlsanity.com/paper/2406.11257/) (2024)
- [Lossless and Near-Lossless Compression for Foundation Models](https://mlsanity.com/paper/2404.15198/) (2024)
- [Lossy and Lossless (L2) Post-training Model Size Compression](https://mlsanity.com/paper/2308.04269/) (2023)
- [Compute, Time and Energy Characterization of Encoder-Decoder Networks with Automatic Mixed Precision Training](https://mlsanity.com/paper/2008.08062/) (2020)
- [Just-in-time Quantization with Processing-In-Memory for Efficient ML Training](https://mlsanity.com/paper/2311.05034/) (2023)

## Abstract (arXiv)

As large language models (LLMs) continue to grow in size and complexity, efficient checkpoint saving&loading has become crucial for managing storage, memory usage, and fault tolerance in LLM training. The current works do not comprehensively take into account the optimization of these several aspects. This paper proposes a novel checkpoint sparsification and quantization method that adapts dynamically to different training stages and model architectures. We present a comprehensive analysis of existing lossy and lossless compression techniques, identify current limitations, and introduce our adaptive approach that balances compression ratio, speed, and precision impact throughout the training process. Experiments on different sizes of LLMs demonstrate that our bitmask-based sparsification method achieves 16x compression ratio without compromising model accuracy. Additionally, the cluster-based quantization method achieves 2x compression ratio with little precision loss.