AttentionLite: Towards Efficient Self-Attention Models for Vision
Not on the map. Why?
Abstract (arXiv)
We propose a novel framework for producing a class of parameter and compute efficient models called AttentionLitesuitable for resource-constrained applications. Prior work has primarily focused on optimizing models either via knowledge distillation or pruning. In addition to fusing these two mechanisms, our joint optimization framework also leverages recent advances in self-attention as a substitute for convolutions. We can simultaneously distill knowledge from a compute-heavy teacher while also pruning the student model in a single pass of training thereby reducing training and fine-tuning times considerably. We evaluate the merits of our proposed approach on the CIFAR-10, CIFAR-100, and Tiny-ImageNet datasets. Not only do our AttentionLite models significantly outperform their unoptimized counterparts in accuracy, we find that in some cases, that they perform almost as well as their compute-heavy teachers while consuming only a fraction of the parameters and FLOPs. Concretely, AttentionLite models can achieve upto30x parameter efficiency and 2x computation efficiency with no significant accuracy drop compared to their teacher.
Key facts. Cited by 10 papers in the map (data through 14 Oct 2026). First posted on arXiv 21 Dec 2020.
Builds on
No reference list yet.
Built on by
Citing papers per year: 2021: 3; 2022: 3; 2023: 4.
10 papers.
- 2022Separable Self-attention for Mobile Vision Transformerscited by 163
- 2021HIRE-SNN: Harnessing the Inherent Robustness of Energy-Efficient Deep Spiking Neural Networks by Training with Crafted Input Noisecited by 65
- 2023Learning to Linearize Deep Neural Networks for Secure and Efficient Private Inferencecited by 20
- 2021Pipeline Parallelism for Inference on Heterogeneous Edge Computingcited by 13
- 2022Lottery Aware Sparsity Hunting: Enabling Federated Learning on Resource-Limited Edgecited by 12
- 2023ViTA: A Vision Transformer Inference Accelerator for Edge Applicationscited by 11
- 2023Making Models Shallow Again: Jointly Learning to Reduce Non-Linearity and Depth for Latency-Efficient Private Inferencecited by 4
- 2023Recent Advances in Scalable Energy-Efficient and Trustworthy Spiking Neural networks: from Algorithms to Technologycited by 3
- 2021Sparse Distillation: Speeding Up Text Classification by Using Bigger Student Modelscited by 1
- 2022A Fast and Efficient Conditional Learning for Tunable Trade-Off between Accuracy and Robustnesscited by 1
- 2023Recent Advances in Scalable Energy-Efficient and Trustworthy Spiking Neural networks: from Algorithms to Technology2 Dec 2023
- 2023Making Models Shallow Again: Jointly Learning to Reduce Non-Linearity and Depth for Latency-Efficient Private Inference26 Apr 2023
- 2023ViTA: A Vision Transformer Inference Accelerator for Edge Applications17 Feb 2023
- 2023Learning to Linearize Deep Neural Networks for Secure and Efficient Private Inference23 Jan 2023
- 2022Lottery Aware Sparsity Hunting: Enabling Federated Learning on Resource-Limited Edge27 Aug 2022
- 2022Separable Self-attention for Mobile Vision Transformers6 Jun 2022
- 2022A Fast and Efficient Conditional Learning for Tunable Trade-Off between Accuracy and Robustness28 Mar 2022
- 2021Pipeline Parallelism for Inference on Heterogeneous Edge Computing28 Oct 2021
- 2021Sparse Distillation: Speeding Up Text Classification by Using Bigger Student Models16 Oct 2021
- 2021HIRE-SNN: Harnessing the Inherent Robustness of Energy-Efficient Deep Spiking Neural Networks by Training with Crafted Input Noise6 Oct 2021