mlsanityData to 7 Oct 2026

Compressing DMA Engine: Leveraging Activation Sparsity for Training Deep Neural Networks

arXiv 1705.01626 ↗First posted 3 May 2017cs.LG cs.ARcited by 36

Not on the map. Why?

Abstract (arXiv)

Popular deep learning frameworks require users to fine-tune their memory usage so that the training data of a deep neural network (DNN) fits within the GPU physical memory. Prior work tries to address this restriction by virtualizing the memory usage of DNNs, enabling both CPU and GPU memory to be utilized for memory allocations. Despite its merits, virtualizing memory can incur significant performance overheads when the time needed to copy data back and forth from CPU memory is higher than the latency to perform the computations required for DNN forward and backward propagation. We introduce a high-performance virtualization strategy based on a "compressing DMA engine" (cDMA) that drastically reduces the size of the data structures that are targeted for CPU-side allocations. The cDMA engine offers an average 2.6x (maximum 13.8x) compression ratio by exploiting the sparsity inherent in offloaded data, improving the performance of virtualized DNNs by an average 32% (maximum 61%).

Key facts. Cited by 36 papers in the map (data through 7 Oct 2026). First posted on arXiv 3 May 2017.

Builds on

No reference list yet.

Built on by

Cited by year

Citing papers per year: 2021: 9; 2022: 11; 2023: 5; 2024: 2; 2025: 9.

36 papers.

Sort:

By area:

  1. 2022The Lazy Neuron Phenomenon: On Emergence of Activation Sparsity in Transformerscited by 121
  2. 2022NISPA: Neuro-Inspired Stability-Plasticity Adaptation for Continual Learning in Sparse Networkscited by 38
  3. 2022Fine-tuning Language Models over Slow Networks using Activation Compression with Guaranteescited by 29
  4. 2021DynO: Dynamic Onloading of Deep Neural Networks from Cloud to Devicecited by 17
  5. 2025Oaken: Fast and Efficient LLM Serving with Online-Offline Hybrid KV Cache Quantizationcited by 13
  6. 2022SmartSAGE: Training Large-scale Graph Neural Networks using In-Storage Processing Architecturescited by 12
  7. 2022Training Personalized Recommendation Systems from (GPU) Scratch: Look Forward not Backwardscited by 11
  8. 2023Accelerating Deep Neural Networks via Semi-Structured Activation Sparsitycited by 10
  9. 2021Resource-Efficient Deep Learning: A Survey on Model-, Arithmetic-, and Implementation-Level Techniquescited by 9
  10. 2021COMET: A Novel Memory-Efficient Deep Learning Training Framework by Using Error-Bounded Lossy Compressioncited by 8
  11. 2024Enabling Unstructured Sparse Acceleration on Structured Sparse Acceleratorscited by 8
  12. 2021Edge-Cloud Collaborated Object Detection via Difficult-Case Discriminatorcited by 7
24 more
  1. 2024The Impact of Uniform Inputs on Activation Sparsity and Energy-Latency Attacks in Computer Visioncited by 7
  2. 2025Universal Properties of Activation Sparsity in Modern Large Language Modelscited by 5
  3. 2022GROW: A Row-Stationary Sparse-Dense GEMM Accelerator for Memory-Efficient Graph Convolutional Neural Networkscited by 4
  4. 2023SGCN: Exploiting Compressed-Sparse Features in Deep Graph Convolutional Network Acceleratorscited by 4
  5. 2023SPADE: Sparse Pillar-based 3D Object Detection Accelerator for Autonomous Drivingcited by 4
  6. 2021The Impact of Activation Sparsity on Overfitting in Convolutional Neural Networkscited by 3
  7. 2021High Performance Convolution Using Sparsity and Patterns for Inference in Deep Convolutional Neural Networkscited by 3
  8. 2021Compression-aware Projection with Greedy Dimension Reduction for Convolutional Neural Network Activationscited by 3
  9. 2022DiVa: An Accelerator for Differentially Private Machine Learningcited by 3
  10. 2025Ultra Memory-Efficient On-FPGA Training of Transformers via Tensor-Compressed Optimizationcited by 3
  11. 2025XAMBA: Enabling Efficient State Space Models on Resource-Constrained Neural Processing Unitscited by 3
  12. 2025Adacc: An Adaptive Framework Unifying Compression and Activation Recomputation for LLM Trainingcited by 3
  13. 2021LayerPipe: Accelerating Deep Neural Network Training by Intra-Layer and Inter-Layer Gradient Pipelining and Multiprocessor Schedulingcited by 2
  14. 2022DELTA: Dynamically Optimizing GPU Memory beyond Tensor Recomputationcited by 2
  15. 2022Improved Projection Learning for Lower Dimensional Feature Mapscited by 2
  16. 2025GraNNite: Enabling High-Performance Execution of Graph Neural Networks on Resource-Constrained Neural Processing Unitscited by 2
  17. 2025TAH-QUANT: Effective Activation Quantization in Pipeline Parallelism over Slow Networkcited by 2
  18. 2021Smartphone Impostor Detection with Behavioral Data Privacy and Minimalist Hardware Supportcited by 1
  19. 2022Nesting Forward Automatic Differentiation for Memory-Efficient Deep Neural Network Trainingcited by 1
  20. 2023Bandwidth-efficient Inference for Neural Image Compressioncited by 1
  21. 2022APack: Off-Chip, Lossless Data Compression for Efficient Deep Learning Inferencecited by 0
  22. 2023Vision Transformers for Mobile Applications: A Short Surveycited by 0
  23. 2025Example-Based Concept Analysis Framework for Deep Weather Forecast Modelscited by 0
  24. 2025Distribution-Aware Tensor Decomposition for Compression of Convolutional Neural Networkscited by 0