mlsanityData to 7 Oct 2026

I-BERT: Integer-only BERT Quantization

arXiv 2101.01321 ↗First posted 5 Jan 2021cs.CLcited by 250

Not on the map. Why?

Cited by 21 in Efficient LLM inference and serving, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Transformer based models, like BERT and RoBERTa, have achieved state-of-the-art results in many Natural Language Processing tasks. However, their memory footprint, inference latency, and power consumption are prohibitive efficient inference at the edge, and even at the data center. While quantization can be a viable solution for this, previous work on quantizing Transformer based models use floating-point arithmetic during inference, which cannot efficiently utilize integer-only logical units such as the recent Turing Tensor Cores, or traditional integer-only ARM processors. In this work, we propose I-BERT, a novel quantization scheme for Transformer based models that quantizes the entire inference with integer-only arithmetic. Based on lightweight integer-only approximation methods for nonlinear operations, e.g., GELU, Softmax, and Layer Normalization, I-BERT performs an end-to-end integer-only BERT inference without any floating point calculation. We evaluate our approach on GLUE downstream tasks using RoBERTa-Base/Large. We show that for both cases, I-BERT achieves similar (and slightly higher) accuracy as compared to the full-precision baseline. Furthermore, our preliminary implementation of I-BERT shows a speedup of 2.4-4.0x for INT8 inference on a T4 GPU system as compared to FP32 inference. The framework has been developed in PyTorch and has been open-sourced.

Key facts. Cited by 250 papers in the map (data through 7 Oct 2026). Builds on 77 of its 102 references that are in the map. First posted on arXiv 5 Jan 2021.

Builds on

77 of 102 references.

References · 77

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  6. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  7. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  8. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
  9. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,415
  10. 2016Layer Normalizationcited by 5,579
  11. 2015Rethinking the Inception Architecture for Computer Visioncited by 5,451
  12. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
38 more
  1. 2018MobileNetV2: Inverted Residuals and Linear Bottleneckscited by 4,773
  2. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  3. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  4. 2017MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applicationscited by 3,827
  5. 2016Gaussian Error Linear Units (GELUs)cited by 3,431
  6. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,801
  7. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
  8. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,665
  9. 2015Learning both Weights and Connections for Efficient Neural Networkscited by 1,998
  10. 2019Searching for MobileNetV3cited by 1,984
  11. 2019Megatron-LM: Training Multi-Billion Parameter Language Models Using Model Parallelismcited by 1,829
  12. 2014FitNets: Hints for Thin Deep Netscited by 1,619
  13. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,601
  14. 2017Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inferencecited by 1,548
  15. 2019fairseq: A Fast, Extensible Toolkit for Sequence Modelingcited by 1,486
  16. 2020MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformerscited by 1,414
  17. 2019TinyBERT: Distilling BERT for Natural Language Understandingcited by 1,153
  18. 2018ShuffleNet V2: Practical Guidelines for Efficient CNN Architecture Designcited by 1,132
  19. 2016Pruning Filters for Efficient ConvNetscited by 1,129
  20. 2016SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <0.5MB model sizecited by 1,054
  21. 2017SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluationcited by 998
  22. 2018Neural Network Acceptability Judgmentscited by 952
  23. 2016XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networkscited by 829
  24. 2019Are Sixteen Heads Really Better than One?cited by 803
  25. 2018Universal Transformerscited by 613
  26. 2019Learned Step Size Quantizationcited by 604
  27. 2015BinaryConnect: Training Deep Neural Networks with binary weights during propagationscited by 571
  28. 2016DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradientscited by 557
  29. 2018Quantizing deep convolutional networks for efficient inference: A whitepapercited by 507
  30. 2019Patient Knowledge Distillation for BERT Model Compressioncited by 502
  31. 2018FBNet: Hardware-Aware Efficient ConvNet Design via Differentiable Neural Architecture Searchcited by 480
  32. 2020MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devicescited by 476
  33. 2018PACT: Parameterized Clipping Activation for Quantized Neural Networkscited by 456
  34. 2019Reducing Transformer Depth on Demand with Structured Dropoutcited by 434
  35. 2019Dreaming to Distill: Data-free Knowledge Transfer via DeepInversioncited by 419
  36. 2020Movement Pruning: Adaptive Sparsity by Fine-Tuningcited by 398
  37. 2019HAWQ: Hessian AWare Quantization of Neural Networks with Mixed-Precisioncited by 387
  38. 2020Fine-Tuning Pretrained Language Models: Weight Initializations, Data Orders, and Early Stoppingcited by 377

50 most cited of 77 All 77 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 20; 2022: 42; 2023: 56; 2024: 57; 2025: 40; 2026: 35.

250 papers.

Sort:

By area:

  1. 2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,338
  2. 2021A Survey of Quantization Methods for Efficient Neural Network Inferencecited by 798
  3. 2024EAGLE: Speculative Sampling Requires Rethinking Feature Uncertaintycited by 564
  4. 2022ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformerscited by 517
  5. 2024EAGLE-2: Faster Inference of Language Models with Dynamic Draft Treescited by 362
  6. 2025EAGLE-3: Scaling up Inference Acceleration of Large Language Models via Training-Time Testcited by 318
  7. 2023SqueezeLLM: Dense-and-Sparse Quantizationcited by 285
  8. 2022ZeroGen: Efficient Zero-shot Learning via Dataset Generationcited by 209
  9. 2022Outlier Suppression: Pushing the Limit of Low-bit Transformer Language Modelscited by 176
  10. 2023Speculative Decoding with Big Little Decodercited by 163
  11. 2022A Fast Post-Training Pruning Framework for Transformerscited by 161
  12. 2024SageAttention: Accurate 8-Bit Attention for Plug-and-play Inference Accelerationcited by 158
38 more
  1. 2021FQ-ViT: Post-Training Quantization for Fully Quantized Vision Transformercited by 148
  2. 2023Quantizable Transformers: Removing Outliers by Helping Attention Heads Do Nothingcited by 147
  3. 2021Understanding and Overcoming the Challenges of Efficient Transformer Quantizationcited by 139
  4. 2021Learned Token Pruning for Transformerscited by 138
  5. 2022Rethinking Vision Transformers for MobileNet Size and Speedcited by 121
  6. 2022LUT-GEMM: Quantized Matrix Multiplication based on LUTs for Efficient Inference in Large-Scale Generative Language Modelscited by 98
  7. 2021Learning Compact Metrics for MTcited by 93
  8. 2022I-ViT: Integer-only Quantization for Efficient Vision Transformer Inferencecited by 93
  9. 2023S3: Increasing GPU Utilization during Generative Inference for Higher Throughputcited by 81
  10. 2023Full Stack Optimization of Transformer Inference: a Surveycited by 79
  11. 2022Efficient Methods for Natural Language Processing: A Surveycited by 78
  12. 2023ZeroQuant-V2: Exploring Post-training Quantization in LLMs from Comprehensive Study to Low Rank Compensationcited by 78
  13. 2023Stable and low-precision training for large-scale vision-language modelscited by 74
  14. 2023Towards Accurate Post-Training Quantization for Vision Transformercited by 70
  15. 2021Prune Once for All: Sparse Pre-Trained Language Modelscited by 66
  16. 2023QLLM: Accurate and Efficient Low-Bitwidth Quantization for Large Language Modelscited by 63
  17. 2023Outlier Suppression+: Accurate quantization of large language models by equivalent and optimal shifting and scalingcited by 62
  18. 2024Model Compression and Efficient Inference for Large Language Models: A Surveycited by 62
  19. 2021Compute and Energy Consumption Trends in Deep Learning Inferencecited by 61
  20. 2023Compressing LLMs: The Truth is Rarely Pure and Never Simplecited by 59
  21. 2023Training Transformers with 4-bit Integerscited by 58
  22. 2022Squeezeformer: An Efficient Transformer for Automatic Speech Recognitioncited by 50
  23. 2023Revisiting Token Pruning for Object Detection and Instance Segmentationcited by 50
  24. 2022HeatViT: Hardware-Efficient Adaptive Token Pruning for Vision Transformerscited by 46
  25. 2022A Survey on Model Compression and Acceleration for Pretrained Language Modelscited by 45
  26. 2024A Survey on Transformer Compressioncited by 44
  27. 2023Towards the Law of Capacity Gap in Distilling Language Modelscited by 41
  28. 2023Predictive Pipelined Decoding: A Compute-Latency Trade-off for Exact LLM Decodingcited by 40
  29. 2021FLAT: An Optimized Dataflow for Mitigating Attention Bottleneckscited by 38
  30. 2023A Survey of Techniques for Optimizing Transformer Inferencecited by 35
  31. 2021NLP From Scratch Without Large-Scale Pretraining: A Simple and Efficient Frameworkcited by 34
  32. 2021Applications and Techniques for Fast Machine Learning in Sciencecited by 32
  33. 2022Enable Deep Learning on Mobile Devices: Methods, Systems, and Applicationscited by 32
  34. 2024Jetfire: Efficient and Accurate Transformer Pretraining with INT8 Data Flow and Per-Block Quantizationcited by 30
  35. 2023Efficient Post-training Quantization with FP8 Formatscited by 29
  36. 2023Understanding the Potential of FPGA-Based Spatial Acceleration for Large Language Model Inferencecited by 29
  37. 2023A Review of Deep Learning for Video Captioningcited by 28
  38. 2022F8Net: Fixed-Point 8-bit Only Multiplication for Network Quantizationcited by 26

50 most cited of 250 All 250 (JSON)