# ITA: An Energy-Efficient Attention and Softmax Accelerator for Quantized Transformers

arXiv: https://arxiv.org/abs/2307.03493 · HTML: https://mlsanity.com/paper/2307.03493/

## Key facts

- Cited by 12 papers in the map (data through 14 Oct 2026).
- Builds on 12 of its 19 references that are in the map.
- First posted on arXiv 7 Jul 2023.

## Builds on (most cited first, 12 of 12)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Is Space-Time Attention All You Need for Video Understanding?](https://mlsanity.com/paper/2102.05095/) (2021)
- [Q8BERT: Quantized 8Bit BERT](https://mlsanity.com/paper/1910.06188/) (2019)
- [Escaping the Big Data Paradigm with Compact Transformers](https://mlsanity.com/paper/2104.05704/) (2021)
- [I-BERT: Integer-only BERT Quantization](https://mlsanity.com/paper/2101.01321/) (2021)
- [SpAtten: Efficient Sparse Attention Architecture with Cascade Token and Head Pruning](https://mlsanity.com/paper/2012.09852/) (2020)
- [Efficient 8-Bit Quantization of Transformer Neural Machine Language Translation Model](https://mlsanity.com/paper/1906.00532/) (2019)
- [Softermax: Hardware/Software Co-Design of an Efficient Softmax for Transformers](https://mlsanity.com/paper/2103.09301/) (2021)
- [Audio Transformers](https://mlsanity.com/paper/2105.00335/) (2021)
- [MemPool: A Shared-L1 Memory Many-Core Cluster with a Low-Latency Interconnect](https://mlsanity.com/paper/2012.02973/) (2020)

## Built on by (most cited first, 12 of 12)

- [Ultra-Efficient On-Device Object Detection on AI-Integrated Smart Glasses with TinyissimoYOLO](https://mlsanity.com/paper/2311.01057/) (2023)
- [BETA: Binarized Energy-Efficient Transformer Accelerator at the Edge](https://mlsanity.com/paper/2401.11851/) (2024)
- [VEXP: A Low-Cost RISC-V ISA Extension for Accelerated Softmax Computation in Transformers](https://mlsanity.com/paper/2504.11227/) (2025)
- [Reusing Softmax Hardware Unit for GELU Computation in Transformers](https://mlsanity.com/paper/2402.10118/) (2024)
- [EXAQ: Exponent Aware Quantization For LLMs Acceleration](https://mlsanity.com/paper/2410.03185/) (2024)
- [Toward Attention-based TinyML: A Heterogeneous Accelerated Architecture and Automated Deployment Flow](https://mlsanity.com/paper/2408.02473/) (2024)
- [Hybrid Systolic Array Accelerator with Optimized Dataflow for Edge Large Language Model Inference](https://mlsanity.com/paper/2507.09010/) (2025)
- [SwiftKV: An Edge-Oriented Attention Algorithm and Multi-Head Accelerator for Fast, Efficient LLM Decoding](https://mlsanity.com/paper/2601.10953/) (2026)
- [Taming the Exponential: A Fast Softmax Surrogate for Integer-Native Edge Inference](https://mlsanity.com/paper/2604.02292/) (2026)
- [STEEL: Sparsity-Aware Fused Attention for Energy-Efficient Long-Sequence Inference on AMD's XDNA NPU](https://mlsanity.com/paper/2607.09385/) (2026)
- [Approximating Softmax in Pretrained LLMs: Model Sensitivity and Kernel Acceleration](https://mlsanity.com/paper/2609.33586/) (2026)
- [Pretraining Transformers with Quantized Softmax in Attention](https://mlsanity.com/paper/2609.33591/) (2026)

## Abstract (arXiv)

Transformer networks have emerged as the state-of-the-art approach for natural language processing tasks and are gaining popularity in other domains such as computer vision and audio processing. However, the efficient hardware acceleration of transformer models poses new challenges due to their high arithmetic intensities, large memory requirements, and complex dataflow dependencies. In this work, we propose ITA, a novel accelerator architecture for transformers and related models that targets efficient inference on embedded systems by exploiting 8-bit quantization and an innovative softmax implementation that operates exclusively on integer values. By computing on-the-fly in streaming mode, our softmax implementation minimizes data movement and energy consumption. ITA achieves competitive energy efficiency with respect to state-of-the-art transformer accelerators with 16.9 TOPS/W, while outperforming them in area efficiency with 5.93 TOPS/mm2 in 22 nm fully-depleted silicon-on-insulator technology at 0.8 V.