mlsanityData to 13 Oct 2026

BinaryBERT: Pushing the Limit of BERT Quantization

arXiv 2012.15701 ↗First posted 31 Dec 2020cs.CLcited by 177

Not on the map. Why?

Abstract (arXiv)

The rapid development of large pre-trained language models has greatly increased the demand for model compression techniques, among which quantization is a popular solution. In this paper, we propose BinaryBERT, which pushes BERT quantization to the limit by weight binarization. We find that a binary BERT is hard to be trained directly than a ternary counterpart due to its complex and irregular loss landscape. Therefore, we propose ternary weight splitting, which initializes BinaryBERT by equivalently splitting from a half-sized ternary network. The binary model thus inherits the good performance of the ternary one, and can be further enhanced by fine-tuning the new architecture after splitting. Empirical results show that our BinaryBERT has only a slight performance drop compared with the full-precision model while being 24x smaller, achieving the state-of-the-art compression results on the GLUE and SQuAD benchmarks.

Key facts. Cited by 177 papers in the map (data through 13 Oct 2026). First posted on arXiv 31 Dec 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 13 Oct)

Citing papers per year: 2021: 21; 2022: 38; 2023: 44; 2024: 36; 2025: 27; 2026: 11.

177 papers.

Sort:

By area:

  1. 2022LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scalecited by 896
  2. 2021A Survey of Quantization Methods for Efficient Neural Network Inferencecited by 814
  3. 2023LLM-Pruner: On the Structural Pruning of Large Language Modelscited by 646
  4. 2022ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformerscited by 522
  5. 2023LLM-QAT: Data-Free Quantization Aware Training for Large Language Modelscited by 322
  6. 2022Confident Adaptive Language Modelingcited by 302
  7. 2023SqueezeLLM: Dense-and-Sparse Quantizationcited by 292
  8. 2021Post-Training Quantization for Vision Transformercited by 278
  9. 2021I-BERT: Integer-only BERT Quantizationcited by 254
  10. 2023LoftQ: LoRA-Fine-Tuning-Aware Quantization for Large Language Modelscited by 196
  11. 2022QDrop: Randomly Dropping Quantization for Extremely Low-bit Post-Training Quantizationcited by 180
  12. 2022Outlier Suppression: Pushing the Limit of Low-bit Transformer Language Modelscited by 179
38 more
  1. 2024DuQuant: Distributing Outliers via Dual Transformation Makes Stronger Quantized LLMscited by 167
  2. 2021MoEfication: Transformer Feed-forward Layers are Mixtures of Expertscited by 150
  3. 2021Learned Token Pruning for Transformerscited by 139
  4. 2024FlatQuant: Flatness Matters for LLM Quantizationcited by 123
  5. 2021Pre-trained Language Models in Biomedical Domain: A Systematic Surveycited by 87
  6. 2024A Survey of Resource-efficient LLM and Multimodal Foundation Modelscited by 86
  7. 2023PB-LLM: Partially Binarized Large Language Modelscited by 81
  8. 2022Unified Visual Transformer Compressioncited by 79
  9. 2022Compression of Generative Pre-trained Language Models via Quantizationcited by 79
  10. 2023ZeroQuant-V2: Exploring Post-training Quantization in LLMs from Comprehensive Study to Low Rank Compensationcited by 79
  11. 2022Efficient Methods for Natural Language Processing: A Surveycited by 78
  12. 2023Stable and low-precision training for large-scale vision-language modelscited by 74
  13. 2022BiBERT: Accurate Fully Binarized BERTcited by 72
  14. 2024A Complete Survey on LLM-based AI Chatbotscited by 71
  15. 2024Retrieval-Augmented Generation for Natural Language Processing: A Surveycited by 71
  16. 2025Quantization Hurts Reasoning? An Empirical Study on Quantized Reasoning Modelscited by 70
  17. 2024Model Compression and Efficient Inference for Large Language Models: A Surveycited by 64
  18. 2023Outlier Suppression+: Accurate quantization of large language models by equivalent and optimal shifting and scalingcited by 61
  19. 2023Training Transformers with 4-bit Integerscited by 60
  20. 2024Understanding LLMs: A Comprehensive Overview from Training to Inferencecited by 58
  21. 2021A Survey on Green Deep Learningcited by 52
  22. 2022BiT: Robustly Binarized Multi-distilled Transformercited by 52
  23. 2024MoDeGPT: Modular Decomposition for Large Language Model Compressioncited by 48
  24. 2021The Low-Resource Double Bind: An Empirical Study of Pruning for Low-Resource Machine Translationcited by 46
  25. 2024A Survey on Transformer Compressioncited by 46
  26. 2021A comprehensive review of Binary Neural Networkcited by 45
  27. 2022A Survey on Model Compression and Acceleration for Pretrained Language Modelscited by 45
  28. 2024CATS: Contextually-Aware Thresholding for Sparsity in Large Language Modelscited by 41
  29. 2021Towards Efficient Post-training Quantization of Pre-trained Language Modelscited by 39
  30. 2022NusaCrowd: Open Source Initiative for Indonesian NLP Resourcescited by 38
  31. 2023A Survey of Techniques for Optimizing Transformer Inferencecited by 35
  32. 2022Enable Deep Learning on Mobile Devices: Methods, Systems, and Applicationscited by 32
  33. 2024Jetfire: Efficient and Accurate Transformer Pretraining with INT8 Data Flow and Per-Block Quantizationcited by 31
  34. 2024SpikeLM: Towards General Spike-Driven Language Modeling via Elastic Bi-Spiking Mechanismscited by 31
  35. 2024A Survey of Low-bit Large Language Models: Basics, Systems, and Algorithmscited by 31
  36. 2022Auto-ViT-Acc: An FPGA-Aware Automatic Acceleration Framework for Vision Transformer with Mixed-Scheme Quantizationcited by 29
  37. 2022BiViT: Extremely Compressed Binary Vision Transformercited by 28
  38. 2022VAQF: Fully Automatic Software-Hardware Co-Design Framework for Low-Bit Vision Transformercited by 27

50 most cited of 177 All 177 (JSON)