# DAPA: Distribution Aware Piecewise Activation Functions for On-Device Transformer Inference and Training

arXiv: https://arxiv.org/abs/2603.19338 · HTML: https://mlsanity.com/paper/2603.19338/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 12 of its 24 references that are in the map.
- First posted on arXiv 19 Mar 2026.

## Builds on (most cited first, 12 of 12)

- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018)
- [Training data-efficient image transformers & distillation through attention](https://mlsanity.com/paper/2012.12877/) (2020)
- [Swin Transformer V2: Scaling Up Capacity and Resolution](https://mlsanity.com/paper/2111.09883/) (2021)
- [I-BERT: Integer-only BERT Quantization](https://mlsanity.com/paper/2101.01321/) (2021)
- [Auto-ViT-Acc: An FPGA-Aware Automatic Acceleration Framework for Vision Transformer with Mixed-Scheme Quantization](https://mlsanity.com/paper/2208.05163/) (2022)
- [SwiftTron: An Efficient Hardware Accelerator for Quantized Transformers](https://mlsanity.com/paper/2304.03986/) (2023)
- [PEANO-ViT: Power-Efficient Approximations of Non-Linearities in Vision Transformers](https://mlsanity.com/paper/2406.14854/) (2024)
- [Flex-SFU: Accelerating DNN Activation Functions by Non-Uniform Piecewise Approximation](https://mlsanity.com/paper/2305.04546/) (2023)
- [HG-PIPE: Vision Transformer Acceleration with Hybrid-Grained Pipeline](https://mlsanity.com/paper/2407.17879/) (2024)
- [Reusing Softmax Hardware Unit for GELU Computation in Transformers](https://mlsanity.com/paper/2402.10118/) (2024)

## Abstract (arXiv)

Non-linear activation functions play a pivotal role in on-device inference and training, as they not only consume substantial hardware resources but also impose a significant impact on system performance and energy efficiency. In this work, we propose Distribution-Aware Piecewise Activation (DAPA), a differentiable and hardware-friendly activation function for Transformer architectures by exploiting the distribution of pre-activation data. DAPA employs a non-uniform piecewise approximation that allocates finer segments to high-probability regions of the distribution, improving generalizability over prior piecewise linear methods. The resulting approximation is further quantized using Distribution-Weighted Mean Square Error to reduce latency and resource utilization for hardware deployment. Our HLS implementation demonstrates that DAPA speeds up GELU computation by 16× and decreases DSP utilization by 16× while maintaining comparable or better performance across vision Transformers and GPT-2 models.