mlsanityData to 14 Oct 2026

Genetic Quantization-Aware Approximation for Non-Linear Operations in Transformers

arXiv 2403.19591 ↗First posted 28 Mar 2024cs.LG cs.AR cs.NEcited by 2

Not on the map. Why?

Abstract (arXiv)

Non-linear functions are prevalent in Transformers and their lightweight variants, incurring substantial and frequently underestimated hardware costs. Previous state-of-the-art works optimize these operations by piece-wise linear approximation and store the parameters in look-up tables (LUT), but most of them require unfriendly high-precision arithmetics such as FP/INT 32 and lack consideration of integer-only INT quantization. This paper proposed a genetic LUT-Approximation algorithm namely GQA-LUT that can automatically determine the parameters with quantization awareness. The results demonstrate that GQA-LUT achieves negligible degradation on the challenging semantic segmentation task for both vanilla and linear Transformer models. Besides, proposed GQA-LUT enables the employment of INT8-based LUT-Approximation that achieves an area savings of 81.3~81.7% and a power reduction of 79.3~80.2% compared to the high-precision FP/INT 32 alternatives. Code is available at https:// github.com/PingchengDong/GQA-LUT.

Key facts. Cited by 2 papers in the map (data through 14 Oct 2026). Builds on 12 of its 23 references that are in the map. First posted on arXiv 28 Mar 2024.

Builds on

12 of 23 references.

References · 12

  1. 2017Attention Is All You Needcited by 60,361
  2. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  3. 2016The Cityscapes Dataset for Semantic Urban Scene Understandingcited by 4,444
  4. 2021SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformerscited by 2,940
  5. 2013Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computationcited by 2,079
  6. 2017Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inferencecited by 1,593
  7. 2019Learned Step Size Quantizationcited by 628
  8. 2021I-BERT: Integer-only BERT Quantizationcited by 255
  9. 2023LLM-FP4: 4-Bit Floating-Point Quantized Transformerscited by 69
  10. 2023Oscillation-free Quantization for Low-bit Vision Transformerscited by 57
  11. 2022Hydra Attention: Efficient Attention with Many Headscited by 55
  12. 2021Softermax: Hardware/Software Co-Design of an Efficient Softmax for Transformerscited by 54

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2025: 1; 2026: 1.

2 papers.

  1. 2025APSQ: Additive Partial Sum Quantization with Algorithm-Hardware Co-Designcited by 0
  2. 2026MIVE: A Minimalist Integer Vector Engine for Softmax LayerNorm and RMSNorm Accelerationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025MLoRQ: Bridging Low-Rank and Quantization for Transformer Compression
  2. 2024RepQuant: Towards Accurate Post-Training Quantization of Large Transformer Models via Scale Reparameterization
  3. 2025GPLQ: A General, Practical, and Lightning QAT Method for Vision Transformers
  4. 2025SeVeDo: A Heterogeneous Transformer Accelerator for Low-Bit Inference via Hierarchical Group Quantization and SVD-Guided Mixed Precision
  5. 2026Every Bit Counts: A Theoretical Study of Precision-Expressivity Tradeoffs in Quantized Transformers
  6. 2025TWEO: Transformers Without Extreme Outliers Enables FP8 Training And Quantization For Dummies
  7. 2024Jetfire: Efficient and Accurate Transformer Pretraining with INT8 Data Flow and Per-Block Quantization
  8. 2022Zero-Shot Dynamic Quantization for Transformer Inference