mlsanityData to 7 Oct 2026

Improving the Efficiency of Transformers for Resource-Constrained Devices

arXiv 2106.16006 ↗First posted 30 Jun 2021cs.LG cs.CVcited by 12

Not on the map. Why?

Abstract (arXiv)

Transformers provide promising accuracy and have become popular and used in various domains such as natural language processing and computer vision. However, due to their massive number of model parameters, memory and computation requirements, they are not suitable for resource-constrained low-power devices. Even with high-performance and specialized devices, the memory bandwidth can become a performance-limiting bottleneck. In this paper, we present a performance analysis of state-of-the-art vision transformers on several devices. We propose to reduce the overall memory footprint and memory transfers by clustering the model parameters. We show that by using only 64 clusters to represent model parameters, it is possible to reduce the data transfer from the main memory by more than 4x, achieve up to 22% speedup and 39% energy savings on mobile devices with less than 0.1% accuracy loss.

Key facts. Cited by 12 papers in the map (data through 7 Oct 2026). Builds on 21 of its 36 references that are in the map. First posted on arXiv 30 Jun 2021.

Builds on

21 of 36 references.

References · 21

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  5. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  6. 2014ImageNet Large Scale Visual Recognition Challengecited by 8,441
  7. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  8. 2014Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modelingcited by 2,341
  9. 2015Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Codingcited by 2,291
  10. 2020Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformerscited by 1,368
  11. 2021Transformers in Vision: A Surveycited by 1,009
  12. 2020Efficient Transformers: A Surveycited by 784
9 more
  1. 2019Q-BERT: Hessian Based Ultra Low Precision Quantization of BERTcited by 355
  2. 2019Q8BERT: Quantized 8Bit BERTcited by 296
  3. 2016EIE: Efficient Inference Engine on Compressed Deep Neural Networkcited by 291
  4. 2018Model compression via distillation and quantizationcited by 287
  5. 2020Integer Quantization for Deep Learning Inference: Principles and Empirical Evaluationcited by 183
  6. 2020BinaryBERT: Pushing the Limit of BERT Quantizationcited by 175
  7. 2020Data Movement Is All You Need: A Case Study on Optimizing Transformerscited by 108
  8. 2019Efficient 8-Bit Quantization of Transformer Neural Machine Language Translation Modelcited by 58
  9. 2019Language Models with Transformerscited by 36

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 2; 2023: 2; 2024: 3; 2025: 4; 2026: 1.

12 papers.

  1. 2022Transformers in Medical Imaging: A Surveycited by 245
  2. 2023EfficientViT: Memory Efficient Vision Transformer with Cascaded Group Attentioncited by 172
  3. 2024SHViT: Single-Head Vision Transformer with Memory Efficient Macro Designcited by 55
  4. 2023Which Transformer to Favor: A Comparative Analysis of Efficiency in Vision Transformerscited by 9
  5. 2022Large Scale Time-Series Representation Learning via Simultaneous Low and High Frequency Feature Bootstrappingcited by 5
  6. 2025Automating Versatile Time-Series Analysis with Tiny Transformers on Embedded FPGAscited by 4
  7. 2024CoRA: Optimizing Low-Rank Adaptation with Common Subspace of Large Language Modelscited by 2
  8. 2025Image Recognition with Online Lightweight Vision Transformer: A Surveycited by 2
  9. 2025Frugal Machine Learning for Energy-efficient, and Resource-aware Artificial Intelligencecited by 1
  10. 2025MambaLite-Micro: Memory-Optimized Mamba Inference on MCUscited by 1
  11. 2024MAS-Attention: Memory-Aware Stream Processing for Attention Acceleration on Resource-Constrained Edge Devicescited by 0
  12. 2026Evolving Layer-Specific Scalar Functions for Hardware-Aware Transformer Adaptationcited by 0