mlsanityData to 7 Oct 2026

NN-LUT: Neural Approximation of Non-Linear Operations for Efficient Transformer Inference

arXiv 2112.02191 ↗First posted 3 Dec 2021cs.LG cs.AIcited by 25

Not on the map. Why?

Abstract (arXiv)

Non-linear operations such as GELU, Layer normalization, and Softmax are essential yet costly building blocks of Transformer models. Several prior works simplified these operations with look-up tables or integer computations, but such approximations suffer inferior accuracy or considerable hardware cost with long latency. This paper proposes an accurate and hardware-friendly approximation framework for efficient Transformer inference. Our framework employs a simple neural network as a universal approximator with its structure equivalently transformed into a LUT. The proposed framework called NN-LUT can accurately replace all the non-linear operations in popular BERT models with significant reductions in area, power consumption, and latency.

Key facts. Cited by 25 papers in the map (data through 7 Oct 2026). Builds on 14 of its 26 references that are in the map. First posted on arXiv 3 Dec 2021.

Builds on

14 of 26 references.

References · 14

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  4. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  5. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  6. 2020MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devicescited by 476
  7. 2017The Expressive Power of Neural Networks: A View from the Widthcited by 318
  8. 2019Q8BERT: Quantized 8Bit BERTcited by 296
  9. 2021I-BERT: Integer-only BERT Quantizationcited by 250
  10. 2020SpAtten: Efficient Sparse Attention Architecture with Cascade Token and Head Pruningcited by 228
  11. 2020TernaryBERT: Distillation-aware Ultra-low Bit BERTcited by 145
  12. 2019Efficient 8-Bit Quantization of Transformer Neural Machine Language Translation Modelcited by 58
2 more
  1. 2021Softermax: Hardware/Software Co-Design of an Efficient Softmax for Transformerscited by 52
  2. 2018AXNet: ApproXimate computing using an end-to-end trainable neural networkcited by 2

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 4; 2023: 2; 2024: 5; 2025: 8; 2026: 6.

25 papers.

Sort:

By area:

  1. 2023Full Stack Optimization of Transformer Inference: a Surveycited by 79
  2. 2022Squeezeformer: An Efficient Transformer for Automatic Speech Recognitioncited by 50
  3. 2024ConSmax: Hardware-Friendly Alternative Softmax with Learnable Parameterscited by 14
  4. 2025QUARK: Quantization-Enabled Circuit Sharing for Transformer Acceleration by Exploiting Common Patterns in Nonlinear Operationscited by 10
  5. 2022Approximate Computing and the Efficient Machine Learning Expeditioncited by 8
  6. 2025LUT-DLA: Lookup Table as Efficient Extreme Low-Bit Deep Learning Acceleratorcited by 8
  7. 2025SOLE: Hardware-Software Co-design of Softmax and LayerNorm for Efficient Transformer Inferencecited by 5
  8. 2025VEXP: A Low-Cost RISC-V ISA Extension for Accelerated Softmax Computation in Transformerscited by 4
  9. 2022A Transistor Operations Model for Deep Learning Energy Consumption Scaling Lawcited by 2
  10. 2024From Algorithm to Hardware: A Survey on Efficient and Safe Deployment of Deep Neural Networkscited by 2
  11. 2025FQ-PETR: Fully Quantized Position Embedding Transformation for Multi-View 3D Object Detectioncited by 2
  12. 2025VEDA: Efficient LLM Generation Through Voting-based KV Cache Eviction and Dataflow-flexible Acceleratorcited by 1
13 more
  1. 2025Optimizing Neural Networks with Learnable Non-Linear Activation Functions via Lookup-Based FPGA Accelerationcited by 1
  2. 2026NLI:Non-uniform Linear Interpolation Approximation of Nonlinear Operations for Efficient LLMs Inferencecited by 1
  3. 2026LLM Inference in a Flash!cited by 1
  4. 2022FusionFormer: Fusing Operations in Transformer for Efficient Streaming Speech Recognitioncited by 0
  5. 2023NeuralMatrix: Compute the Entire Neural Networks with Linear Matrix Operations for Efficient Inferencecited by 0
  6. 2024NOVA: NoC-based Vector Unit for Mapping Attention Layers on a CNN Acceleratorcited by 0
  7. 2024QuAKE: Speeding up Model Inference Using Quick and Approximate Kernels for Exponential Non-Linearitiescited by 0
  8. 2024IterL2Norm: Fast Iterative L2-Normalizationcited by 0
  9. 2025FQ-PETR: Fully Quantized Position Embedding Transformation for Multi-View 3D Object Detectioncited by 0
  10. 2026TriGen: NPU Architecture for End-to-End Acceleration of Large Language Models based on SW-HW Co-Designcited by 0
  11. 2026Integrated electro-optic attention nonlinearities for transformerscited by 0
  12. 2026Evolving Layer-Specific Scalar Functions for Hardware-Aware Transformer Adaptationcited by 0
  13. 2026MIVE: A Minimalist Integer Vector Engine for Softmax LayerNorm and RMSNorm Accelerationcited by 0