# Compressing DMA Engine: Leveraging Activation Sparsity for Training Deep Neural Networks

arXiv: https://arxiv.org/abs/1705.01626 · HTML: https://mlsanity.com/paper/1705.01626/

## Key facts

- Cited by 37 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 3 May 2017.

## Built on by (most cited first, 30 of 37)

- [The Lazy Neuron Phenomenon: On Emergence of Activation Sparsity in Transformers](https://mlsanity.com/paper/2210.06313/) (2022)
- [NISPA: Neuro-Inspired Stability-Plasticity Adaptation for Continual Learning in Sparse Networks](https://mlsanity.com/paper/2206.09117/) (2022)
- [Fine-tuning Language Models over Slow Networks using Activation Compression with Guarantees](https://mlsanity.com/paper/2206.01299/) (2022)
- [DynO: Dynamic Onloading of Deep Neural Networks from Cloud to Device](https://mlsanity.com/paper/2104.09949/) (2021)
- [Oaken: Fast and Efficient LLM Serving with Online-Offline Hybrid KV Cache Quantization](https://mlsanity.com/paper/2503.18599/) (2025)
- [SmartSAGE: Training Large-scale Graph Neural Networks using In-Storage Processing Architectures](https://mlsanity.com/paper/2205.04711/) (2022)
- [Training Personalized Recommendation Systems from (GPU) Scratch: Look Forward not Backwards](https://mlsanity.com/paper/2205.04702/) (2022)
- [Accelerating Deep Neural Networks via Semi-Structured Activation Sparsity](https://mlsanity.com/paper/2309.06626/) (2023)
- [Resource-Efficient Deep Learning: A Survey on Model-, Arithmetic-, and Implementation-Level Techniques](https://mlsanity.com/paper/2112.15131/) (2021)
- [COMET: A Novel Memory-Efficient Deep Learning Training Framework by Using Error-Bounded Lossy Compression](https://mlsanity.com/paper/2111.09562/) (2021)
- [Enabling Unstructured Sparse Acceleration on Structured Sparse Accelerators](https://mlsanity.com/paper/2403.07953/) (2024)
- [Edge-Cloud Collaborated Object Detection via Difficult-Case Discriminator](https://mlsanity.com/paper/2108.12858/) (2021)
- [The Impact of Uniform Inputs on Activation Sparsity and Energy-Latency Attacks in Computer Vision](https://mlsanity.com/paper/2403.18587/) (2024)
- [Universal Properties of Activation Sparsity in Modern Large Language Models](https://mlsanity.com/paper/2509.00454/) (2025)
- [GROW: A Row-Stationary Sparse-Dense GEMM Accelerator for Memory-Efficient Graph Convolutional Neural Networks](https://mlsanity.com/paper/2203.00158/) (2022)
- [SGCN: Exploiting Compressed-Sparse Features in Deep Graph Convolutional Network Accelerators](https://mlsanity.com/paper/2301.10388/) (2023)
- [Compression-aware Projection with Greedy Dimension Reduction for Convolutional Neural Network Activations](https://mlsanity.com/paper/2110.08828/) (2021)
- [DiVa: An Accelerator for Differentially Private Machine Learning](https://mlsanity.com/paper/2208.12392/) (2022)
- [SPADE: Sparse Pillar-based 3D Object Detection Accelerator for Autonomous Driving](https://mlsanity.com/paper/2305.07522/) (2023)
- [TAH-QUANT: Effective Activation Quantization in Pipeline Parallelism over Slow Network](https://mlsanity.com/paper/2506.01352/) (2025)
- [The Impact of Activation Sparsity on Overfitting in Convolutional Neural Networks](https://mlsanity.com/paper/2104.06153/) (2021)
- [High Performance Convolution Using Sparsity and Patterns for Inference in Deep Convolutional Neural Networks](https://mlsanity.com/paper/2104.08314/) (2021)
- [Ultra Memory-Efficient On-FPGA Training of Transformers via Tensor-Compressed Optimization](https://mlsanity.com/paper/2501.06663/) (2025)
- [XAMBA: Enabling Efficient State Space Models on Resource-Constrained Neural Processing Units](https://mlsanity.com/paper/2502.06924/) (2025)
- [Adacc: An Adaptive Framework Unifying Compression and Activation Recomputation for LLM Training](https://mlsanity.com/paper/2508.00806/) (2025)
- [LayerPipe: Accelerating Deep Neural Network Training by Intra-Layer and Inter-Layer Gradient Pipelining and Multiprocessor Scheduling](https://mlsanity.com/paper/2108.06629/) (2021)
- [DELTA: Dynamically Optimizing GPU Memory beyond Tensor Recomputation](https://mlsanity.com/paper/2203.15980/) (2022)
- [Nesting Forward Automatic Differentiation for Memory-Efficient Deep Neural Network Training](https://mlsanity.com/paper/2209.10778/) (2022)
- [Improved Projection Learning for Lower Dimensional Feature Maps](https://mlsanity.com/paper/2210.15170/) (2022)
- [GraNNite: Enabling High-Performance Execution of Graph Neural Networks on Resource-Constrained Neural Processing Units](https://mlsanity.com/paper/2502.06921/) (2025)

## Abstract (arXiv)

Popular deep learning frameworks require users to fine-tune their memory usage so that the training data of a deep neural network (DNN) fits within the GPU physical memory. Prior work tries to address this restriction by virtualizing the memory usage of DNNs, enabling both CPU and GPU memory to be utilized for memory allocations. Despite its merits, virtualizing memory can incur significant performance overheads when the time needed to copy data back and forth from CPU memory is higher than the latency to perform the computations required for DNN forward and backward propagation. We introduce a high-performance virtualization strategy based on a "compressing DMA engine" (cDMA) that drastically reduces the size of the data structures that are targeted for CPU-side allocations. The cDMA engine offers an average 2.6x (maximum 13.8x) compression ratio by exploiting the sparsity inherent in offloaded data, improving the performance of virtualized DNNs by an average 32% (maximum 61%).