mlsanityData to 7 Oct 2026

Efficient 8-Bit Quantization of Transformer Neural Machine Language Translation Model

arXiv 1906.00532 ↗First posted 3 Jun 2019cs.LGcited by 58

Not on the map. Why?

Abstract (arXiv)

In this work, we quantize a trained Transformer machine language translation model leveraging INT8/VNNI instructions in the latest Intel^circledR Xeon^circledR Cascade Lake processors to improve inference performance while maintaining less than 0.5% drop in accuracy. To the best of our knowledge, this is the first attempt in the industry to quantize the Transformer model. This has high impact as it clearly demonstrates the various complexities of quantizing the language translation model. We present novel quantization techniques directly in TensorFlow to opportunistically replace 32-bit floating point (FP32) computations with 8-bit integers (INT8) and transform the FP32 computational graph. We also present a bin-packing parallel batching technique to maximize CPU utilization. Overall, our optimizations with INT8/VNNI deliver 1.5X improvement over the best FP32 performance. Furthermore, it reveals the opportunities and challenges to boost performance of quantized deep learning inference and establishes best practices to run inference with high efficiency on Intel CPUs.

Key facts. Cited by 58 papers in the map (data through 7 Oct 2026). First posted on arXiv 3 Jun 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 16; 2022: 14; 2023: 14; 2024: 8; 2025: 3; 2026: 3.

58 papers.

Sort:

By area:

  1. 2021A Survey of Quantization Methods for Efficient Neural Network Inferencecited by 798
  2. 2021LeViT: a Vision Transformer in ConvNet's Clothing for Faster Inferencecited by 475
  3. 2021I-BERT: Integer-only BERT Quantizationcited by 250
  4. 2021Understanding and Overcoming the Challenges of Efficient Transformer Quantizationcited by 139
  5. 2021Learned Token Pruning for Transformerscited by 138
  6. 2022LUT-GEMM: Quantized Matrix Multiplication based on LUTs for Efficient Inference in Large-Scale Generative Language Modelscited by 98
  7. 2023Cascade Speculative Drafting for Even Faster LLM Inferencecited by 83
  8. 2022Efficient Methods for Natural Language Processing: A Surveycited by 78
  9. 2021VS-Quant: Per-vector Scaled Quantization for Accurate Low-Precision Neural Network Inferencecited by 59
  10. 2021A Survey on Green Deep Learningcited by 52
  11. 2023Revisiting Token Pruning for Object Detection and Instance Segmentationcited by 50
  12. 2022ANT: Exploiting Adaptive Numerical Data Type for Low-bit Deep Neural Network Quantizationcited by 42
38 more
  1. 2024Outlier-Efficient Hopfield Layers for Large Transformer-Based Modelscited by 38
  2. 2023Efficient Post-training Quantization with FP8 Formatscited by 29
  3. 2021NN-LUT: Neural Approximation of Non-Linear Operations for Efficient Transformer Inferencecited by 25
  4. 2023Interpretability Illusions in the Generalization of Simplified Modelscited by 24
  5. 2022AlphaTuning: Quantization-Aware Parameter-Efficient Adaptation of Large-Scale Pre-Trained Language Modelscited by 22
  6. 2022Train Flat, Then Compress: Sharpness-Aware Minimization Learns More Compressible Modelscited by 20
  7. 2023SwiftTron: An Efficient Hardware Accelerator for Quantized Transformerscited by 17
  8. 2022Efficient Quantized Sparse Matrix Operations on Tensor Corescited by 16
  9. 2024A Comprehensive Survey of Compression Algorithms for Language Modelscited by 15
  10. 2021On the Distribution, Sparsity, and Inference-time Quantization of Attention Values in Transformerscited by 14
  11. 2022Pyramid-BERT: Reducing Complexity via Successive Core-set based Token Selectioncited by 13
  12. 2021Improving the Efficiency of Transformers for Resource-Constrained Devicescited by 12
  13. 2023Binarized Neural Machine Translationcited by 12
  14. 2023ITA: An Energy-Efficient Attention and Softmax Accelerator for Quantized Transformerscited by 12
  15. 2025ResMoE: Space-efficient Compression of Mixture of Experts LLMs via Residual Restorationcited by 12
  16. 2022Understanding and Improving Knowledge Distillation for Quantization-Aware Training of Large Transformer Encoderscited by 11
  17. 2024Optimizing Large Language Models through Quantization: A Comparative Analysis of PTQ and QAT Techniquescited by 10
  18. 2021Efficient Softmax Approximation for Deep Neural Networks with Attention Mechanismcited by 9
  19. 2021Low-Precision Hardware Architectures Meet Recommendation Model Inference at Scalecited by 7
  20. 2024Designing Large Foundation Models for Efficient Training and Inference: A Surveycited by 7
  21. 2021LEAP: Learnable Pruning for Transformer-based Modelscited by 6
  22. 2023Dynamic Stashing Quantization for Efficient Transformer Trainingcited by 6
  23. 2024Efficiently Distilling LLMs for Edge Applicationscited by 5
  24. 2022Sub-8-Bit Quantization Aware Training for 8-Bit Neural Network Accelerator with On-Device Speech Recognitioncited by 4
  25. 2023Tango: rethinking quantization for graph neural network training on GPUscited by 4
  26. 2023FP8-BERT: Post-Training Quantization for Transformercited by 4
  27. 2022Towards Fine-tuning Pre-trained Language Models with Integer Forward and Backward Propagationcited by 3
  28. 2024Token Pruning using a Lightweight Background Aware Vision Transformercited by 3
  29. 2025AnchorFormer: Differentiable Anchor Attention for Efficient Vision Transformercited by 3
  30. 2025Hawk: Leveraging Spatial Context for Faster Autoregressive Text-to-Image Generationcited by 3
  31. 2021The NiuTrans System for the WMT21 Efficiency Taskcited by 2
  32. 2022The Ecological Footprint of Neural Machine Translation Systemscited by 2
  33. 2022Legal-Tech Open Diaries: Lesson learned on how to develop and deploy light-weight models in the era of humongous Language Modelscited by 2
  34. 2022Fast DistilBERT on CPUscited by 2
  35. 2023Teacher Intervention: Improving Convergence of Quantization Aware Training for Ultra-Low Precision Transformerscited by 2
  36. 2023An Efficient Sparse Inference Software Accelerator for Transformer-based Language Models on CPUscited by 2
  37. 2026It Takes a MAESTRO To Prune Bad Expertscited by 2
  38. 2021The NiuTrans System for WNGT 2020 Efficiency Taskcited by 1

50 most cited of 58 All 58 (JSON)