# NN-LUT: Neural Approximation of Non-Linear Operations for Efficient Transformer Inference

arXiv: https://arxiv.org/abs/2112.02191 · HTML: https://mlsanity.com/paper/2112.02191/

## Key facts

- Cited by 25 papers in the map (data through 14 Oct 2026).
- Builds on 14 of its 26 references that are in the map.
- First posted on arXiv 3 Dec 2021.

## Builds on (most cited first, 14 of 14)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [RoBERTa: A Robustly Optimized BERT Pretraining Approach](https://mlsanity.com/paper/1907.11692/) (2019)
- [GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understanding](https://mlsanity.com/paper/1804.07461/) (2018)
- [SQuAD: 100,000+ Questions for Machine Comprehension of Text](https://mlsanity.com/paper/1606.05250/) (2016)
- [MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devices](https://mlsanity.com/paper/2004.02984/) (2020)
- [The Expressive Power of Neural Networks: A View from the Width](https://mlsanity.com/paper/1709.02540/) (2017)
- [Q8BERT: Quantized 8Bit BERT](https://mlsanity.com/paper/1910.06188/) (2019)
- [I-BERT: Integer-only BERT Quantization](https://mlsanity.com/paper/2101.01321/) (2021)
- [SpAtten: Efficient Sparse Attention Architecture with Cascade Token and Head Pruning](https://mlsanity.com/paper/2012.09852/) (2020)
- [TernaryBERT: Distillation-aware Ultra-low Bit BERT](https://mlsanity.com/paper/2009.12812/) (2020)
- [Efficient 8-Bit Quantization of Transformer Neural Machine Language Translation Model](https://mlsanity.com/paper/1906.00532/) (2019)
- [Softermax: Hardware/Software Co-Design of an Efficient Softmax for Transformers](https://mlsanity.com/paper/2103.09301/) (2021)
- [AXNet: ApproXimate computing using an end-to-end trainable neural network](https://mlsanity.com/paper/1807.10458/) (2018)

## Built on by (most cited first, 25 of 25)

- [Full Stack Optimization of Transformer Inference: a Survey](https://mlsanity.com/paper/2302.14017/) (2023)
- [Squeezeformer: An Efficient Transformer for Automatic Speech Recognition](https://mlsanity.com/paper/2206.00888/) (2022)
- [ConSmax: Hardware-Friendly Alternative Softmax with Learnable Parameters](https://mlsanity.com/paper/2402.10930/) (2024)
- [QUARK: Quantization-Enabled Circuit Sharing for Transformer Acceleration by Exploiting Common Patterns in Nonlinear Operations](https://mlsanity.com/paper/2511.06767/) (2025)
- [LUT-DLA: Lookup Table as Efficient Extreme Low-Bit Deep Learning Accelerator](https://mlsanity.com/paper/2501.10658/) (2025)
- [Approximate Computing and the Efficient Machine Learning Expedition](https://mlsanity.com/paper/2210.00497/) (2022)
- [SOLE: Hardware-Software Co-design of Softmax and LayerNorm for Efficient Transformer Inference](https://mlsanity.com/paper/2510.17189/) (2025)
- [VEXP: A Low-Cost RISC-V ISA Extension for Accelerated Softmax Computation in Transformers](https://mlsanity.com/paper/2504.11227/) (2025)
- [A Transistor Operations Model for Deep Learning Energy Consumption Scaling Law](https://mlsanity.com/paper/2205.15062/) (2022)
- [From Algorithm to Hardware: A Survey on Efficient and Safe Deployment of Deep Neural Networks](https://mlsanity.com/paper/2405.06038/) (2024)
- [FQ-PETR: Fully Quantized Position Embedding Transformation for Multi-View 3D Object Detection](https://mlsanity.com/paper/2502.15488/) (2025)
- [VEDA: Efficient LLM Generation Through Voting-based KV Cache Eviction and Dataflow-flexible Accelerator](https://mlsanity.com/paper/2507.00797/) (2025)
- [Optimizing Neural Networks with Learnable Non-Linear Activation Functions via Lookup-Based FPGA Acceleration](https://mlsanity.com/paper/2508.17069/) (2025)
- [NLI:Non-uniform Linear Interpolation Approximation of Nonlinear Operations for Efficient LLMs Inference](https://mlsanity.com/paper/2602.02988/) (2026)
- [LLM Inference in a Flash!](https://mlsanity.com/paper/2609.16161/) (2026)
- [FusionFormer: Fusing Operations in Transformer for Efficient Streaming Speech Recognition](https://mlsanity.com/paper/2210.17079/) (2022)
- [NeuralMatrix: Compute the Entire Neural Networks with Linear Matrix Operations for Efficient Inference](https://mlsanity.com/paper/2305.14405/) (2023)
- [NOVA: NoC-based Vector Unit for Mapping Attention Layers on a CNN Accelerator](https://mlsanity.com/paper/2405.04206/) (2024)
- [QuAKE: Speeding up Model Inference Using Quick and Approximate Kernels for Exponential Non-Linearities](https://mlsanity.com/paper/2412.00408/) (2024)
- [IterL2Norm: Fast Iterative L2-Normalization](https://mlsanity.com/paper/2412.04778/) (2024)
- [FQ-PETR: Fully Quantized Position Embedding Transformation for Multi-View 3D Object Detection](https://mlsanity.com/paper/2511.09347/) (2025)
- [TriGen: NPU Architecture for End-to-End Acceleration of Large Language Models based on SW-HW Co-Design](https://mlsanity.com/paper/2602.12962/) (2026)
- [Integrated electro-optic attention nonlinearities for transformers](https://mlsanity.com/paper/2604.09512/) (2026)
- [Evolving Layer-Specific Scalar Functions for Hardware-Aware Transformer Adaptation](https://mlsanity.com/paper/2605.14047/) (2026)
- [MIVE: A Minimalist Integer Vector Engine for Softmax LayerNorm and RMSNorm Acceleration](https://mlsanity.com/paper/2606.17781/) (2026)

## Abstract (arXiv)

Non-linear operations such as GELU, Layer normalization, and Softmax are essential yet costly building blocks of Transformer models. Several prior works simplified these operations with look-up tables or integer computations, but such approximations suffer inferior accuracy or considerable hardware cost with long latency. This paper proposes an accurate and hardware-friendly approximation framework for efficient Transformer inference. Our framework employs a simple neural network as a universal approximator with its structure equivalently transformed into a LUT. The proposed framework called NN-LUT can accurately replace all the non-linear operations in popular BERT models with significant reductions in area, power consumption, and latency.