# BETA: Binarized Energy-Efficient Transformer Accelerator at the Edge

arXiv: https://arxiv.org/abs/2401.11851 · HTML: https://mlsanity.com/paper/2401.11851/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 25 of its 30 references that are in the map.
- First posted on arXiv 22 Jan 2024.

## Builds on (most cited first, 25 of 25)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018)
- [ProgPrompt: Generating Situated Robot Task Plans using Large Language Models](https://mlsanity.com/paper/2209.11302/) (2022)
- [Q8BERT: Quantized 8Bit BERT](https://mlsanity.com/paper/1910.06188/) (2019)
- [VisualGPT: Data-efficient Adaptation of Pretrained Language Models for Image Captioning](https://mlsanity.com/paper/2102.10407/) (2021)
- [BinaryBERT: Pushing the Limit of BERT Quantization](https://mlsanity.com/paper/2012.15701/) (2020)
- [Solving Challenging Math Word Problems Using GPT-4 Code Interpreter with Code-based Self-Verification](https://mlsanity.com/paper/2308.07921/) (2023)
- [TernaryBERT: Distillation-aware Ultra-low Bit BERT](https://mlsanity.com/paper/2009.12812/) (2020)
- [GOBO: Quantizing Attention-Based NLP Models for Low Latency and Energy Efficient Inference](https://mlsanity.com/paper/2005.03842/) (2020)
- [A3: Accelerating Attention Mechanisms in Neural Networks with Approximation](https://mlsanity.com/paper/2002.10941/) (2020)
- [BiBERT: Accurate Fully Binarized BERT](https://mlsanity.com/paper/2203.06390/) (2022)
- [FTRANS: Energy-Efficient Acceleration of Transformers using FPGA](https://mlsanity.com/paper/2007.08563/) (2020)
- [BiT: Robustly Binarized Multi-distilled Transformer](https://mlsanity.com/paper/2205.13016/) (2022)
- [Hardware Accelerator for Multi-Head Attention and Position-Wise Feed-Forward in the Transformer](https://mlsanity.com/paper/2009.08605/) (2020)
- [An Algorithm-Hardware Co-Optimized Framework for Accelerating N:M Sparse Transformers](https://mlsanity.com/paper/2208.06118/) (2022)
- [VAQF: Fully Automatic Software-Hardware Co-Design Framework for Low-Bit Vision Transformer](https://mlsanity.com/paper/2201.06618/) (2022)
- [Hardware Acceleration of Fully Quantized BERT for Efficient Natural Language Processing](https://mlsanity.com/paper/2103.02800/) (2021)
- [SwiftTron: An Efficient Hardware Accelerator for Quantized Transformers](https://mlsanity.com/paper/2304.03986/) (2023)
- [BinaryViT: Pushing Binary Vision Transformers Towards Convolutional Models](https://mlsanity.com/paper/2306.16678/) (2023)
- [ShiftAddViT: Mixture of Multiplication Primitives Towards Efficient Vision Transformer](https://mlsanity.com/paper/2306.06446/) (2023)
- [BEBERT: Efficient and Robust Binary Ensemble BERT](https://mlsanity.com/paper/2210.15976/) (2022)
- [ITA: An Energy-Efficient Attention and Softmax Accelerator for Quantized Transformers](https://mlsanity.com/paper/2307.03493/) (2023)
- [ViTA: A Vision Transformer Inference Accelerator for Edge Applications](https://mlsanity.com/paper/2302.09108/) (2023)

## Built on by (most cited first, 6 of 6)

- [COBRA: Algorithm-Architecture Co-optimized Binary Transformer Accelerator for Edge Inference](https://mlsanity.com/paper/2504.16269/) (2025)
- [Hardware Acceleration of LLMs: A comprehensive survey and comparison](https://mlsanity.com/paper/2409.03384/) (2024)
- [Ultra Memory-Efficient On-FPGA Training of Transformers via Tensor-Compressed Optimization](https://mlsanity.com/paper/2501.06663/) (2025)
- [A Motion-Aware Vector Quantization Framework with Centroid Reuse for Efficient VLA Inference](https://mlsanity.com/paper/2607.24148/) (2026)
- [NeuronMLP: Efficient LLM Inference via Singular Value Decomposition Compression and Tiling on AWS Trainium](https://mlsanity.com/paper/2510.25977/) (2025)
- [Recent Developments in Transformer Inference Deployment on FPGA Platforms: A Survey](https://mlsanity.com/paper/2609.01212/) (2026)

## Abstract (arXiv)

Existing binary Transformers are promising in edge deployment due to their compact model size, low computational complexity, and considerable inference accuracy. However, deploying binary Transformers faces challenges on prior processors due to inefficient execution of quantized matrix multiplication (QMM) and the energy consumption overhead caused by multi-precision activations. To tackle the challenges above, we first develop a computation flow abstraction method for binary Transformers to improve QMM execution efficiency by optimizing the computation order. Furthermore, a binarized energy-efficient Transformer accelerator, namely BETA, is proposed to boost the efficient deployment at the edge. Notably, BETA features a configurable QMM engine, accommodating diverse activation precisions of binary Transformers and offering high-parallelism and high-speed for QMMs with impressive energy efficiency. Experimental results evaluated on ZCU102 FPGA show BETA achieves an average energy efficiency of 174 GOPS/W, which is 1.76~21.92x higher than prior FPGA-based accelerators, showing BETA's good potential for edge Transformer acceleration.