Data to 7 Oct 2026

Q-BERT: Hessian Based Ultra Low Precision Quantization of BERT

arXiv 1909.05840 ↗First posted 12 Sep 2019cs.CL cs.LGcited by 355

Not on the map. Why?

Cited by 28 in Efficient LLM inference and serving, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Transformer based architectures have become de-facto models used for a range of Natural Language Processing tasks. In particular, the BERT based models achieved significant accuracy gain for GLUE tasks, CoNLL-03 and SQuAD. However, BERT based models have a prohibitive memory footprint and latency. As a result, deploying BERT based models in resource constrained environments has become a challenging task. In this work, we perform an extensive analysis of fine-tuned BERT models using second order Hessian information, and we use our results to propose a novel method for quantizing BERT models to ultra low precision. In particular, we propose a new group-wise quantization scheme, and we use a Hessian based mix-precision method to compress the model further. We extensively test our proposed method on BERT downstream tasks of SST-2, MNLI, CoNLL-03, and SQuAD. We can achieve comparable performance to baseline with at most 2.3% performance degradation, even with ultra-low precision quantization down to 2 bits, corresponding up to 13× compression of the model parameters, and up to 4× compression of the embedding table as well as activations. Among all tasks, we observed the highest performance loss for BERT fine-tuned on SQuAD. By probing into the Hessian based analysis as well as visualization, we show that this is related to the fact that current training/fine-tuning strategy of BERT does not converge for SQuAD.

Key facts. Cited by 355 papers in the map (data through 7 Oct 2026). First posted on arXiv 12 Sep 2019.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 73; 2022: 76; 2023: 77; 2024: 68; 2025: 37; 2026: 24.

355 papers.

Sort:

By area:

2022SmoothQuant: Accurate and Efficient Post-Training Quantization for Large Language Modelscited by 1,338
2022LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scalecited by 881
2022GLM-130B: An Open Bilingual Pre-trained Modelcited by 837
2021A Survey of Quantization Methods for Efficient Neural Network Inferencecited by 798
2023FlexGen: High-Throughput Generative Inference of Large Language Models with a Single GPUcited by 567
2024EAGLE: Speculative Sampling Requires Rethinking Feature Uncertaintycited by 564
2022ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformerscited by 517
2022DeepSpeed-MoE: Advancing Mixture-of-Experts Inference and Training to Power Next-Generation AI Scalecited by 400
2024EAGLE-2: Faster Inference of Language Models with Dynamic Draft Treescited by 362
2021Random Feature Attentioncited by 350
2023LLM-QAT: Data-Free Quantization Aware Training for Large Language Modelscited by 320
2021Pre-Trained Models: Past, Present and Futurecited by 319

38 more

2025EAGLE-3: Scaling up Inference Acceleration of Large Language Models via Training-Time Testcited by 318
2022Confident Adaptive Language Modelingcited by 297
2023SqueezeLLM: Dense-and-Sparse Quantizationcited by 285
2021Post-Training Quantization for Vision Transformercited by 274
2022Post-training Quantization on Diffusion Modelscited by 262
2021I-BERT: Integer-only BERT Quantizationcited by 250
2022ZeroGen: Efficient Zero-shot Learning via Dataset Generationcited by 209
2023LoftQ: LoRA-Fine-Tuning-Aware Quantization for Large Language Modelscited by 195
2022Outlier Suppression: Pushing the Limit of Low-bit Transformer Language Modelscited by 176
2021PTQ4ViT: Post-training quantization for vision transformers with twin uniform quantizationcited by 173
2022Structured Pruning Learns Compact and Accurate Modelscited by 167
2023Speculative Decoding with Big Little Decodercited by 163
2022A Fast Post-Training Pruning Framework for Transformerscited by 161
2023QA-LoRA: Quantization-Aware Low-Rank Adaptation of Large Language Modelscited by 154
2021FQ-ViT: Post-Training Quantization for Fully Quantized Vision Transformercited by 148
2023Quantizable Transformers: Removing Outliers by Helping Attention Heads Do Nothingcited by 147
2021Understanding and Overcoming the Challenges of Efficient Transformer Quantizationcited by 139
2021Learned Token Pruning for Transformerscited by 138
2021AMMUS : A Survey of Transformer-based Pretrained Models in Natural Language Processingcited by 131
2024FlatQuant: Flatness Matters for LLM Quantizationcited by 116
2022The Optimal BERT Surgeon: Scalable and Accurate Second-Order Pruning for Large Language Modelscited by 115
2021Luna: Linear Unified Nested Attentioncited by 96
2021Finetuning Pretrained Transformers into RNNscited by 94
2021Sparse is Enough in Scaling Transformerscited by 86
2024BitDistiller: Unleashing the Potential of Sub-4-Bit LLMs via Self-Distillationcited by 80
2022Compression of Generative Pre-trained Language Models via Quantizationcited by 79
2023Full Stack Optimization of Transformer Inference: a Surveycited by 79
2022Efficient Methods for Natural Language Processing: A Surveycited by 78
2023ZeroQuant-V2: Exploring Post-training Quantization in LLMs from Comprehensive Study to Low Rank Compensationcited by 78
2021Vision Transformer Pruningcited by 77
2023Stable and low-precision training for large-scale vision-language modelscited by 74
2024A Complete Survey on LLM-based AI Chatbotscited by 71
2022BiBERT: Accurate Fully Binarized BERTcited by 70
2021Prune Once for All: Sparse Pre-Trained Language Modelscited by 66
2022Video Transformers: A Surveycited by 63
2023Outlier Suppression+: Accurate quantization of large language models by equivalent and optimal shifting and scalingcited by 62
2024Model Compression and Efficient Inference for Large Language Models: A Surveycited by 62
2021VS-Quant: Per-vector Scaled Quantization for Accurate Low-Precision Neural Network Inferencecited by 59

50 most cited of 355 All 355 (JSON)