mlsanityData to 7 Oct 2026

Reducing the GPU Memory Bottleneck with Lossless Compression for ML -- Extended

arXiv 2605.30728 ↗First posted 29 May 2026cs.LG cs.DCcited by 0

Not on the map. Why?

Abstract (arXiv)

Machine learning (ML) training and inference often process data sets far exceeding GPU memory capacity, forcing them to rely on PCIe for on-demand tensor transfers, causing critical transfer bottlenecks. Lossy compression has been proposed to relieve bottlenecks but introduces workload-dependent accuracy loss, making it complex or even prohibitive to use in existing ML deployments. We explore lossless compression as an alternative that avoids this deployment complexity. We identify where lossless compression can be integrated into ML pipelines while minimizing interference with GPU execution. Based on our findings, we introduce Invariant Bit Packing (IBP), a novel lossless compression algorithm designed to minimize data transfer time for ML. IBP identifies and eliminates invariant bits across groups of tensors, improving throughput through GPU-optimized decompression that leverages warp parallelism, low-overhead bit operations, and asynchronous PCIe transfers. We provide easy-to-use APIs, showcasing them by adding IBP support to GNN training, as well as DLRM and LLM inference frameworks. IBP achieves, on average, 74% faster GNN training, 180% faster DLRM embedding lookup, and 24% faster LLM inference.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 44 of its 96 references that are in the map. First posted on arXiv 29 May 2026.

Builds on

44 of 96 references.

References · 44

  1. 2017Attention Is All You Needcited by 59,932
  2. 2017Inductive Representation Learning on Large Graphscited by 6,284
  3. 2023Efficient Memory Management for Large Language Model Serving with PagedAttentioncited by 5,767
  4. 2021RoFormer: Enhanced Transformer with Rotary Position Embeddingcited by 4,581
  5. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  6. 2020Open Graph Benchmark: Datasets for Machine Learning on Graphscited by 2,148
  7. 2018Graph Convolutional Neural Networks for Web-Scale Recommender Systemscited by 1,133
  8. 2020Learned Image Compression with Discretized Gaussian Mixture Likelihoods and Attention Modulescited by 570
  9. 2017Deep Gradient Compression: Reducing the Communication Bandwidth for Distributed Trainingcited by 471
  10. 2017Deep Gaussian Embedding of Graphs: Unsupervised Inductive Learning via Rankingcited by 332
  11. 2018DVC: An End-to-end Deep Video Compression Frameworkcited by 324
  12. 2023Atom: Low-bit Quantization for Efficient and Accurate LLM Servingcited by 255
32 more
  1. 2024InfiniGen: Efficient Generative Inference of Large Language Models with Dynamic KV Cache Managementcited by 242
  2. 2023CacheGen: KV Cache Compression and Streaming for Fast Large Language Model Servingcited by 199
  3. 2022Lossy Image Compression with Conditional Diffusion Modelscited by 165
  4. 2017Real-Time Adaptive Image Compressioncited by 127
  5. 2021Colossal-AI: A Unified Deep Learning System For Large-Scale Parallel Trainingcited by 96
  6. 2024AI and Memory Wallcited by 89
  7. 2020Molecular graph generation with Graph Neural Networkscited by 79
  8. 2021Software-Hardware Co-design for Fast and Scalable Training of Deep Learning Recommendation Modelscited by 78
  9. 2020DeepRecSys: A System for Optimizing End-To-End At-scale Neural Recommendation Inferencecited by 54
  10. 2021BGL: GPU-Efficient GNN Training by Optimizing Graph Data I/O and Preprocessingcited by 50
  11. 2017Flexpoint: An Adaptive Numerical Format for Efficient Training of Deep Neural Networkscited by 43
  12. 2020Understanding Training Efficiency of Deep Learning Recommendation Models at Scalecited by 43
  13. 2024Accuracy is Not All You Needcited by 27
  14. 202570% Size, 100% Accuracy: Lossless LLM Compression for Efficient GPU Inference via Dynamic-Length Float (DFloat11)cited by 23
  15. 2022MariusGNN: Resource-Efficient Out-of-Core Training of Graph Neural Networkscited by 19
  16. 2023THC: Accelerating Distributed Deep Learning Using Tensor Homomorphic Compressioncited by 15
  17. 2022GPU-Initiated On-Demand High-Throughput Storage Access in the BaM System Architecturecited by 12
  18. 2020EMOGI: Efficient Memory-access for Out-of-memory Graph-traversal In GPUscited by 10
  19. 2022BagPipe: Accelerating Deep Recommendation Model Trainingcited by 10
  20. 2024Graph Neural Network Training Systems: A Performance Comparison of Full-Graph and Mini-Batchcited by 10
  21. 2022A GPU-specialized Inference Parameter Server for Large-Scale Deep Recommendation Modelscited by 9
  22. 2026ZipServ: Fast and Memory-Efficient LLM Inference with Hardware-Aware Lossless Compressioncited by 8
  23. 2023FCBench: Cross-Domain Benchmarking of Lossless Compression for Floating-Point Datacited by 5
  24. 2022GROW: A Row-Stationary Sparse-Dense GEMM Accelerator for Memory-Efficient Graph Convolutional Neural Networkscited by 4
  25. 2022BiFeat: Supercharge GNN Training via Graph Feature Quantizationcited by 4
  26. 2022A Frequency-aware Software Cache for Large Recommendation System Embeddingscited by 4
  27. 2023SGCN: Exploiting Compressed-Sparse Features in Deep Graph Convolutional Network Acceleratorscited by 4
  28. 2024Accelerating Communication in Deep Learning Recommendation Model Training with Dual-Level Adaptive Lossy Compressioncited by 4
  29. 2017Tuple-oriented Compression for Large-scale Mini-batch Stochastic Gradient Descentcited by 3
  30. 2023Legion: Automatically Pushing the Envelope of Multi-GPU System for Billion-Scale GNN Trainingcited by 3
  31. 2023Recoil: Parallel rANS Decoding with Decoder-Adaptive Scalabilitycited by 3
  32. 2020Datasets for Benchmarking Floating-Point Compressorscited by 1

Built on by

Not cited yet.