# Towards Fully 8-bit Integer Inference for the Transformer Model

arXiv: https://arxiv.org/abs/2009.08034 · HTML: https://mlsanity.com/paper/2009.08034/

## Key facts

- Cited by 35 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 17 Sep 2020.

## Built on by (most cited first, 30 of 35)

- [LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scale](https://mlsanity.com/paper/2208.07339/) (2022)
- [I-ViT: Integer-only Quantization for Efficient Vision Transformer Inference](https://mlsanity.com/paper/2207.01405/) (2022)
- [Active-Dormant Attention Heads: Mechanistically Demystifying Extreme-Token Phenomena in LLMs](https://mlsanity.com/paper/2410.13835/) (2024)
- [SimA: Simple Softmax-free Attention for Vision Transformers](https://mlsanity.com/paper/2206.08898/) (2022)
- [ODE Transformer: An Ordinary Differential Equation-Inspired Model for Sequence Generation](https://mlsanity.com/paper/2203.09176/) (2022)
- [X-Former: In-Memory Acceleration of Transformers](https://mlsanity.com/paper/2303.07470/) (2023)
- [I-LLM: Efficient Integer-Only Inference for Fully-Quantized Low-Bit Large Language Models](https://mlsanity.com/paper/2405.17849/) (2024)
- [Watermarking LLMs with Weight Quantization](https://mlsanity.com/paper/2310.11237/) (2023)
- [SwiftTron: An Efficient Hardware Accelerator for Quantized Transformers](https://mlsanity.com/paper/2304.03986/) (2023)
- [Enabling High-Sparsity Foundational Llama Models with Efficient Pretraining and Deployment](https://mlsanity.com/paper/2405.03594/) (2024)
- [From Attention to Activation: Unravelling the Enigmas of Large Language Models](https://mlsanity.com/paper/2410.17174/) (2024)
- [ODE Transformer: An Ordinary Differential Equation-Inspired Model for Neural Machine Translation](https://mlsanity.com/paper/2104.02308/) (2021)
- [Low-bit Model Quantization for Deep Neural Networks: A Survey](https://mlsanity.com/paper/2505.05530/) (2025)
- [An Efficient Transformer Decoder with Compressed Sub-layers](https://mlsanity.com/paper/2101.00542/) (2021)
- [Is Finer Better? The Limits of Microscaling Formats in Large Language Models](https://mlsanity.com/paper/2601.19026/) (2026)
- [AMD-HookNet for Glacier Front Segmentation](https://mlsanity.com/paper/2302.02744/) (2023)
- [Unified Normalization for Accelerating and Stabilizing Transformers](https://mlsanity.com/paper/2208.01313/) (2022)
- [Dynamic Stashing Quantization for Efficient Transformer Training](https://mlsanity.com/paper/2303.05295/) (2023)
- [OutlierTune: Efficient Channel-Wise Quantization for Large Language Models](https://mlsanity.com/paper/2406.18832/) (2024)
- [MergeQuant: Accurate 4-bit Static Quantization of Large Language Models by Channel-wise Calibration](https://mlsanity.com/paper/2503.07654/) (2025)
- [VOGUE: Answer Verbalization through Multi-Task Learning](https://mlsanity.com/paper/2106.13316/) (2021)
- [Hybrid-Regressive Neural Machine Translation](https://mlsanity.com/paper/2210.10416/) (2022)
- [Understanding Parameter Sharing in Transformers](https://mlsanity.com/paper/2306.09380/) (2023)
- [The NiuTrans System for the WMT21 Efficiency Task](https://mlsanity.com/paper/2109.08003/) (2021)
- [Inference Optimizations for Large Language Models: Effects, Challenges, and Practical Considerations](https://mlsanity.com/paper/2408.03130/) (2024)
- [The NiuTrans System for WNGT 2020 Efficiency Task](https://mlsanity.com/paper/2109.08008/) (2021)
- [EIT: Enhanced Interactive Transformer](https://mlsanity.com/paper/2212.10197/) (2022)
- [Transfer training from smaller language model](https://mlsanity.com/paper/2104.11390/) (2021)
- [The RoyalFlush System for the WMT 2022 Efficiency Task](https://mlsanity.com/paper/2212.01543/) (2022)
- [Scaled Quantization for the Vision Transformer](https://mlsanity.com/paper/2303.13601/) (2023)

## Abstract (arXiv)

8-bit integer inference, as a promising direction in reducing both the latency and storage of deep neural networks, has made great progress recently. On the other hand, previous systems still rely on 32-bit floating point for certain functions in complex models (e.g., Softmax in Transformer), and make heavy use of quantization and de-quantization. In this work, we show that after a principled modification on the Transformer architecture, dubbed Integer Transformer, an (almost) fully 8-bit integer inference algorithm Scale Propagation could be derived. De-quantization is adopted when necessary, which makes the network more efficient. Our experiments on WMT16 En<->Ro, WMT14 En<->De and En->Fr translation tasks as well as the WikiText-103 language modelling task show that the fully 8-bit Transformer system achieves comparable performance with the floating point baseline but requires nearly 4x less memory footprint.