# TransAxx: Efficient Transformers with Approximate Computing

arXiv: https://arxiv.org/abs/2402.07545 · HTML: https://mlsanity.com/paper/2402.07545/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 73 references that are in the map.
- First posted on arXiv 12 Feb 2024.

## Builds on (most cited first, 29 of 29)

- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Training data-efficient image transformers & distillation through attention](https://mlsanity.com/paper/2012.12877/) (2020)
- [Quantizing deep convolutional networks for efficient inference: A whitepaper](https://mlsanity.com/paper/1806.08342/) (2018)
- [A White Paper on Neural Network Quantization](https://mlsanity.com/paper/2106.08295/) (2021)
- [HAQ: Hardware-Aware Automated Quantization with Mixed Precision](https://mlsanity.com/paper/1811.08886/) (2018)
- [Post-Training Quantization for Vision Transformer](https://mlsanity.com/paper/2106.14156/) (2021)
- [Unifying PAC and Regret: Uniform PAC Bounds for Episodic Reinforcement Learning](https://mlsanity.com/paper/1703.07710/) (2017)
- [HAT: Hardware-Aware Transformers for Efficient Natural Language Processing](https://mlsanity.com/paper/2005.14187/) (2020)
- [Q-ViT: Accurate and Fully Quantized Low-bit Vision Transformer](https://mlsanity.com/paper/2210.06707/) (2022)
- [Global Context Vision Transformers](https://mlsanity.com/paper/2206.09959/) (2022)
- [Towards Accurate Post-Training Quantization for Vision Transformer](https://mlsanity.com/paper/2303.14341/) (2023)
- [AutoQ: Automated Kernel-Wise Neural Network Quantization](https://mlsanity.com/paper/1902.05690/) (2019)
- [HAWQV3: Dyadic Neural Network Quantization](https://mlsanity.com/paper/2011.10680/) (2020)
- [Mixed-Precision Neural Network Quantization via Learned Layer-wise Importance](https://mlsanity.com/paper/2203.08368/) (2022)
- [Hardware Approximate Techniques for Deep Neural Network Accelerators: A Survey](https://mlsanity.com/paper/2203.08737/) (2022)
- [ALWANN: Automatic Layer-Wise Approximation of Deep Neural Network Accelerators without Retraining](https://mlsanity.com/paper/1907.07229/) (2019)
- [SCALE-Sim: Systolic CNN Accelerator Simulator](https://mlsanity.com/paper/1811.02883/) (2018)
- [TFApprox: Towards a Fast Emulation of DNN Approximate Hardware Accelerators on GPU](https://mlsanity.com/paper/2002.09481/) (2020)
- [Control Variate Approximation for DNN Accelerators](https://mlsanity.com/paper/2102.09642/) (2021)
- [AdaPT: Fast Emulation of Approximate DNN Accelerators in PyTorch](https://mlsanity.com/paper/2203.04071/) (2022)
- [ApproxTrain: Fast Simulation of Approximate Multipliers for DNN Training and Inference](https://mlsanity.com/paper/2209.04161/) (2022)
- [AxTrain: Hardware-Oriented Neural Network Training for Approximate Inference](https://mlsanity.com/paper/1805.08309/) (2018)
- [Automated Machine Learning with Monte-Carlo Tree Search](https://mlsanity.com/paper/1906.00170/) (2019)
- [Automated Circuit Approximation Method Driven by Data Distribution](https://mlsanity.com/paper/1903.04188/) (2019)
- [Learning to design without prior data: Discovering generalizable design strategies using deep learning and tree search](https://mlsanity.com/paper/2211.15068/) (2022)
- [Beyond Monte Carlo Tree Search: Playing Go with Deep Alternative Neural Network and Long-Term Evaluation](https://mlsanity.com/paper/1706.04052/) (2017)
- [An Energy-Efficient Generic Accuracy Configurable Multiplier Based on Block-Level Voltage Overscaling](https://mlsanity.com/paper/2306.09032/) (2023)
- [Automated Design Approximation to Overcome Circuit Aging](https://mlsanity.com/paper/2203.07962/) (2022)

## Built on by (most cited first, 7 of 7)

- [AIE4ML: An End-to-End Framework for Compiling Neural Networks for the Next Generation of AMD AI Engines](https://mlsanity.com/paper/2512.15946/) (2025)
- [Hardware Acceleration of LLMs: A comprehensive survey and comparison](https://mlsanity.com/paper/2409.03384/) (2024)
- [Integrating Pruning with Quantization for Efficient Deep Neural Networks Compression](https://mlsanity.com/paper/2509.04244/) (2025)
- [Gradient Estimation Methods of Approximate Multipliers for High-Accuracy Retraining of Deep Learning Models](https://mlsanity.com/paper/2509.10519/) (2025)
- [Taming the Exponential: A Fast Softmax Surrogate for Integer-Native Edge Inference](https://mlsanity.com/paper/2604.02292/) (2026)
- [AxMoE: Characterizing the Impact of Approximate Multipliers on Mixture-of-Experts DNN Architectures](https://mlsanity.com/paper/2605.04754/) (2026)
- [EULER-ADAS: Energy-Efficient & SIMD-Unified Logarithmic-Posit Engine for Precision-Reconfigurable Approximate ADAS Acceleration](https://mlsanity.com/paper/2605.06875/) (2026)

## Abstract (arXiv)

Vision Transformer (ViT) models which were recently introduced by the transformer architecture have shown to be very competitive and often become a popular alternative to Convolutional Neural Networks (CNNs). However, the high computational requirements of these models limit their practical applicability especially on low-power devices. Current state-of-the-art employs approximate multipliers to address the highly increased compute demands of DNN accelerators but no prior research has explored their use on ViT models. In this work we propose TransAxx, a framework based on the popular PyTorch library that enables fast inherent support for approximate arithmetic to seamlessly evaluate the impact of approximate computing on DNNs such as ViT models. Using TransAxx we analyze the sensitivity of transformer models on the ImageNet dataset to approximate multiplications and perform approximate-aware finetuning to regain accuracy. Furthermore, we propose a methodology to generate approximate accelerators for ViT models. Our approach uses a Monte Carlo Tree Search (MCTS) algorithm to efficiently search the space of possible configurations using a hardware-driven hand-crafted policy. Our evaluation demonstrates the efficacy of our methodology in achieving significant trade-offs between accuracy and power, resulting in substantial gains without compromising on performance.