# Attention Based Pruning for Shift Networks

arXiv: https://arxiv.org/abs/1905.12300 · HTML: https://mlsanity.com/paper/1905.12300/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 29 May 2019.

## Built on by (most cited first, 7 of 7)

- [Quantization and Deployment of Deep Neural Networks on Microcontrollers](https://mlsanity.com/paper/2105.13331/) (2021)
- [Dilated convolution with learnable spacings](https://mlsanity.com/paper/2112.03740/) (2021)
- [Complexity-Driven CNN Compression for Resource-constrained Edge AI](https://mlsanity.com/paper/2208.12816/) (2022)
- [DNN Quantization with Attention](https://mlsanity.com/paper/2103.13322/) (2021)
- [Learning Pruned Structure and Weights Simultaneously from Scratch: an Attention based Approach](https://mlsanity.com/paper/2111.02399/) (2021)
- [Inferring Graph Signal Translations as Invariant Transformations for Classification Tasks](https://mlsanity.com/paper/2102.09493/) (2021)
- [Enhancing Video Understanding: Deep Neural Networks for Spatiotemporal Analysis](https://mlsanity.com/paper/2502.07277/) (2025)

## Abstract (arXiv)

In many application domains such as computer vision, Convolutional Layers (CLs) are key to the accuracy of deep learning methods. However, it is often required to assemble a large number of CLs, each containing thousands of parameters, in order to reach state-of-the-art accuracy, thus resulting in complex and demanding systems that are poorly fitted to resource-limited devices. Recently, methods have been proposed to replace the generic convolution operator by the combination of a shift operation and a simpler 1x1 convolution. The resulting block, called Shift Layer (SL), is an efficient alternative to CLs in the sense it allows to reach similar accuracies on various tasks with faster computations and fewer parameters. In this contribution, we introduce Shift Attention Layers (SALs), which extend SLs by using an attention mechanism that learns which shifts are the best at the same time the network function is trained. We demonstrate SALs are able to outperform vanilla SLs (and CLs) on various object recognition benchmarks while significantly reducing the number of float operations and parameters for the inference.