# PEANO-ViT: Power-Efficient Approximations of Non-Linearities in Vision Transformers

arXiv: https://arxiv.org/abs/2406.14854 · HTML: https://mlsanity.com/paper/2406.14854/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Builds on 7 of its 15 references that are in the map.
- First posted on arXiv 21 Jun 2024.

## Builds on (most cited first, 7 of 7)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Training data-efficient image transformers & distillation through attention](https://mlsanity.com/paper/2012.12877/) (2020)
- [Post-Training Quantization for Vision Transformer](https://mlsanity.com/paper/2106.14156/) (2021)
- [Softermax: Hardware/Software Co-Design of an Efficient Softmax for Transformers](https://mlsanity.com/paper/2103.09301/) (2021)
- [Memory-Efficient Vision Transformers: An Activation-Aware Mixed-Rank Compression Strategy](https://mlsanity.com/paper/2402.06004/) (2024)

## Built on by (most cited first, 13 of 13)

- [VISTA: Vision-Language Inference for Training-Free Stock Time-Series Analysis](https://mlsanity.com/paper/2505.18570/) (2025)
- [Low-Cost FlashAttention with Fused Exponential and Multiplication Hardware Operators](https://mlsanity.com/paper/2505.14314/) (2025)
- [Dynamic Co-Optimization Compiler: Leveraging Multi-Agent Reinforcement Learning for Enhanced DNN Accelerator Performance](https://mlsanity.com/paper/2407.08192/) (2024)
- [CHOSEN: Compilation to Hardware Optimization Stack for Efficient Vision Transformer Inference](https://mlsanity.com/paper/2407.12736/) (2024)
- [CADC: Encoding User-Item Interactions for Compressing Recommendation Model Training Data](https://mlsanity.com/paper/2407.08108/) (2024)
- [Efficient Noise Mitigation for Enhancing Inference Accuracy in DNNs on Mixed-Signal Accelerators](https://mlsanity.com/paper/2409.18553/) (2024)
- [QuAKE: Speeding up Model Inference Using Quick and Approximate Kernels for Exponential Non-Linearities](https://mlsanity.com/paper/2412.00408/) (2024)
- [Vectorized FlashAttention with Low-cost Exponential Computation in RISC-V Vector Processors](https://mlsanity.com/paper/2510.06834/) (2025)
- [QUILL: An Algorithm-Architecture Co-Design for Cache-Local Deformable Attention](https://mlsanity.com/paper/2511.13679/) (2025)
- [DAPA: Distribution Aware Piecewise Activation Functions for On-Device Transformer Inference and Training](https://mlsanity.com/paper/2603.19338/) (2026)
- [Integrated electro-optic attention nonlinearities for transformers](https://mlsanity.com/paper/2604.09512/) (2026)
- [GraphLeap: Decoupling Graph Construction and Convolution for Vision GNN Acceleration on FPGA](https://mlsanity.com/paper/2604.21290/) (2026)
- [Evolving Layer-Specific Scalar Functions for Hardware-Aware Transformer Adaptation](https://mlsanity.com/paper/2605.14047/) (2026)

## Abstract (arXiv)

The deployment of Vision Transformers (ViTs) on hardware platforms, specially Field-Programmable Gate Arrays (FPGAs), presents many challenges, which are mainly due to the substantial computational and power requirements of their non-linear functions, notably layer normalization, softmax, and Gaussian Error Linear Unit (GELU). These critical functions pose significant obstacles to efficient hardware implementation due to their complex mathematical operations and the inherent resource count and architectural limitations of FPGAs. PEANO-ViT offers a novel approach to streamlining the implementation of the layer normalization layer by introducing a division-free technique that simultaneously approximates the division and square root function. Additionally, PEANO-ViT provides a multi-scale division strategy to eliminate division operations in the softmax layer, aided by a Pade-based approximation for the exponential function. Finally, PEANO-ViT introduces a piece-wise linear approximation for the GELU function, carefully designed to bypass the computationally intensive operations associated with GELU. In our comprehensive evaluations, PEANO-ViT exhibits minimal accuracy degradation (<= 0.5% for DeiT-B) while significantly enhancing power efficiency, achieving improvements of 1.91x, 1.39x, 8.01x for layer normalization, softmax, and GELU, respectively. This improvement is achieved through substantial reductions in DSP, LUT, and register counts for these non-linear operations. Consequently, PEANO-ViT enables efficient deployment of Vision Transformers on resource- and power-constrained FPGA platforms.