# DASNet: Dynamic Activation Sparsity for Neural Network Efficiency Improvement

arXiv: https://arxiv.org/abs/1909.06964 · HTML: https://mlsanity.com/paper/1909.06964/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 13 Sep 2019.

## Built on by (most cited first, 5 of 5)

- [HighLight: Efficient and Flexible DNN Acceleration with Hierarchical Structured Sparsity](https://mlsanity.com/paper/2305.12718/) (2023)
- [Decomposing the Neurons: Activation Sparsity via Mixture of Experts for Continual Test Time Adaptation](https://mlsanity.com/paper/2405.16486/) (2024)
- [The Impact of Activation Sparsity on Overfitting in Convolutional Neural Networks](https://mlsanity.com/paper/2104.06153/) (2021)
- [Dual sparse training framework: inducing activation map sparsity via Transformed ℓ1 regularization](https://mlsanity.com/paper/2405.19652/) (2024)
- [MoASE++: Mixture of Activation Sparsity Experts with Domain-Adaptive On-policy Distillation for Continual Test Time Adaptation](https://mlsanity.com/paper/2605.17743/) (2026)

## Abstract (arXiv)

To improve the execution speed and efficiency of neural networks in embedded systems, it is crucial to decrease the model size and computational complexity. In addition to conventional compression techniques, e.g., weight pruning and quantization, removing unimportant activations can reduce the amount of data communication and the computation cost. Unlike weight parameters, the pattern of activations is directly related to input data and thereby changes dynamically. To regulate the dynamic activation sparsity (DAS), in this work, we propose a generic low-cost approach based on winners-take-all (WTA) dropout technique. The network enhanced by the proposed WTA dropout, namely textitDASNet, features structured activation sparsity with an improved sparsity level. Compared to the static feature map pruning methods, DASNets provide better computation cost reduction. The WTA technique can be easily applied in deep neural networks without incurring additional training variables. More importantly, DASNet can be seamlessly integrated with other compression techniques, such as weight pruning and quantization, without compromising on accuracy. Our experiments on various networks and datasets present significant run-time speedups with negligible accuracy loss.