# SwiftTron: An Efficient Hardware Accelerator for Quantized Transformers

arXiv: https://arxiv.org/abs/2304.03986 · HTML: https://mlsanity.com/paper/2304.03986/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- Builds on 24 of its 37 references that are in the map.
- First posted on arXiv 8 Apr 2023.

## Builds on (most cited first, 24 of 24)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018)
- [Training data-efficient image transformers & distillation through attention](https://mlsanity.com/paper/2012.12877/) (2020)
- [Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity](https://mlsanity.com/paper/2101.03961/) (2021)
- [Linformer: Self-Attention with Linear Complexity](https://mlsanity.com/paper/2006.04768/) (2020)
- [Universal Transformers](https://mlsanity.com/paper/1807.03819/) (2018)
- [I-BERT: Integer-only BERT Quantization](https://mlsanity.com/paper/2101.01321/) (2021)
- [HAT: Hardware-Aware Transformers for Efficient Natural Language Processing](https://mlsanity.com/paper/2005.14187/) (2020)
- [A3: Accelerating Attention Mechanisms in Neural Networks with Approximation](https://mlsanity.com/paper/2002.10941/) (2020)
- [I-ViT: Integer-only Quantization for Efficient Vision Transformer Inference](https://mlsanity.com/paper/2207.01405/) (2022)
- [Efficient 8-Bit Quantization of Transformer Neural Machine Language Translation Model](https://mlsanity.com/paper/1906.00532/) (2019)
- [FTRANS: Energy-Efficient Acceleration of Transformers using FPGA](https://mlsanity.com/paper/2007.08563/) (2020)
- [Hardware Accelerator for Multi-Head Attention and Position-Wise Feed-Forward in the Transformer](https://mlsanity.com/paper/2009.08605/) (2020)
- [Fully Quantized Transformer for Machine Translation](https://mlsanity.com/paper/1910.10485/) (2019)
- [Towards Fully 8-bit Integer Inference for the Transformer Model](https://mlsanity.com/paper/2009.08034/) (2020)
- [Hardware and Software Optimizations for Accelerating Deep Neural Networks: Survey of Current Trends, Challenges, and the Road Ahead](https://mlsanity.com/paper/2012.11233/) (2020)
- [Hardware Acceleration of Fully Quantized BERT for Efficient Natural Language Processing](https://mlsanity.com/paper/2103.02800/) (2021)
- [Towards Energy-Efficient and Secure Edge AI: A Cross-Layer Framework](https://mlsanity.com/paper/2109.09829/) (2021)
- [Special Session: Towards an Agile Design Methodology for Efficient, Reliable, and Secure ML Systems](https://mlsanity.com/paper/2204.09514/) (2022)

## Built on by (most cited first, 18 of 18)

- [Vision Transformers on the Edge: A Comprehensive Survey of Model Compression and Acceleration Strategies](https://mlsanity.com/paper/2503.02891/) (2025)
- [HADES: Hardware Accelerated Decoding for Efficient Speculation in Large Language Models](https://mlsanity.com/paper/2412.19925/) (2024)
- [Neural Network Quantization for Microcontrollers: A Comprehensive Survey of Methods, Platforms, and Applications](https://mlsanity.com/paper/2508.15008/) (2025)
- [BETA: Binarized Energy-Efficient Transformer Accelerator at the Edge](https://mlsanity.com/paper/2401.11851/) (2024)
- [Low-Cost FlashAttention with Fused Exponential and Multiplication Hardware Operators](https://mlsanity.com/paper/2505.14314/) (2025)
- [TATAA: Programmable Mixed-Precision Transformer Acceleration with a Transformable Arithmetic Architecture](https://mlsanity.com/paper/2411.03697/) (2024)
- [Stochastic Spiking Attention: Accelerating Attention with Stochastic Computing in Spiking Networks](https://mlsanity.com/paper/2402.09109/) (2024)
- [From Algorithm to Hardware: A Survey on Efficient and Safe Deployment of Deep Neural Networks](https://mlsanity.com/paper/2405.06038/) (2024)
- [Atleus: Accelerating Transformers on the Edge Enabled by 3D Heterogeneous Manycore Architectures](https://mlsanity.com/paper/2501.09588/) (2025)
- [Auxiliary Features-Guided Super Resolution for Monte Carlo Rendering](https://mlsanity.com/paper/2310.13235/) (2023)
- [TinyCL: An Efficient Hardware Architecture for Continual Learning on Autonomous Systems](https://mlsanity.com/paper/2402.09780/) (2024)
- [Shrinking the Giant : Quasi-Weightless Transformers for Low Energy Inference](https://mlsanity.com/paper/2411.01818/) (2024)
- [IterL2Norm: Fast Iterative L2-Normalization](https://mlsanity.com/paper/2412.04778/) (2024)
- [Vectorized FlashAttention with Low-cost Exponential Computation in RISC-V Vector Processors](https://mlsanity.com/paper/2510.06834/) (2025)
- [DAPA: Distribution Aware Piecewise Activation Functions for On-Device Transformer Inference and Training](https://mlsanity.com/paper/2603.19338/) (2026)
- [Focus Session: Hardware and Software Techniques for Accelerating Multimodal Foundation Models](https://mlsanity.com/paper/2604.21952/) (2026)
- [ELSA: An ELastic SNN Inference Architecture for Efficient Neuromorphic Computing](https://mlsanity.com/paper/2605.20802/) (2026)
- [DeVIT: Low-Power Vision Transformer Acceleration Using Delta Computation](https://mlsanity.com/paper/2608.01343/) (2026)

## Abstract (arXiv)

Transformers' compute-intensive operations pose enormous challenges for their deployment in resource-constrained EdgeAI / tinyML devices. As an established neural network compression technique, quantization reduces the hardware computational and memory resources. In particular, fixed-point quantization is desirable to ease the computations using lightweight blocks, like adders and multipliers, of the underlying hardware. However, deploying fully-quantized Transformers on existing general-purpose hardware, generic AI accelerators, or specialized architectures for Transformers with floating-point units might be infeasible and/or inefficient. Towards this, we propose SwiftTron, an efficient specialized hardware accelerator designed for Quantized Transformers. SwiftTron supports the execution of different types of Transformers' operations (like Attention, Softmax, GELU, and Layer Normalization) and accounts for diverse scaling factors to perform correct computations. We synthesize the complete SwiftTron architecture in a 65 nm CMOS technology with the ASIC design flow. Our Accelerator executes the RoBERTa-base model in 1.83 ns, while consuming 33.64 mW power, and occupying an area of 273 mm2. To ease the reproducibility, the RTL of our SwiftTron architecture is released at https://github.com/albertomarchisio/SwiftTron.