mlsanityData to 7 Oct 2026

TernaryBERT: Distillation-aware Ultra-low Bit BERT

arXiv 2009.12812 ↗First posted 27 Sep 2020cs.CL cs.LG cs.SDcited by 145

Not on the map. Why?

Abstract (arXiv)

Transformer-based pre-training models like BERT have achieved remarkable performance in many natural language processing tasks.However, these models are both computation and memory expensive, hindering their deployment to resource-constrained devices. In this work, we propose TernaryBERT, which ternarizes the weights in a fine-tuned BERT model. Specifically, we use both approximation-based and loss-aware ternarization methods and empirically investigate the ternarization granularity of different parts of BERT. Moreover, to reduce the accuracy degradation caused by the lower capacity of low bits, we leverage the knowledge distillation technique in the training process. Experiments on the GLUE benchmark and SQuAD show that our proposed TernaryBERT outperforms the other BERT quantization methods, and even achieves comparable performance as the full-precision model while being 14.9x smaller.

Key facts. Cited by 145 papers in the map (data through 7 Oct 2026). First posted on arXiv 27 Sep 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 27; 2022: 39; 2023: 31; 2024: 28; 2025: 13; 2026: 7.

145 papers.

Sort:

By area:

  1. 2022LLM.int8(): 8-bit Matrix Multiplication for Transformers at Scalecited by 881
  2. 2021A Survey of Quantization Methods for Efficient Neural Network Inferencecited by 798
  3. 2022ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformerscited by 517
  4. 2023LLM-QAT: Data-Free Quantization Aware Training for Large Language Modelscited by 320
  5. 2021Pre-Trained Models: Past, Present and Futurecited by 319
  6. 2023SqueezeLLM: Dense-and-Sparse Quantizationcited by 285
  7. 2021Post-Training Quantization for Vision Transformercited by 274
  8. 2021I-BERT: Integer-only BERT Quantizationcited by 250
  9. 2022Outlier Suppression: Pushing the Limit of Low-bit Transformer Language Modelscited by 176
  10. 2024DuQuant: Distributing Outliers via Dual Transformation Makes Stronger Quantized LLMscited by 165
  11. 2021MoEfication: Transformer Feed-forward Layers are Mixtures of Expertscited by 149
  12. 2021Learned Token Pruning for Transformerscited by 138
38 more
  1. 2021AMMUS : A Survey of Transformer-based Pretrained Models in Natural Language Processingcited by 131
  2. 2022The Optimal BERT Surgeon: Scalable and Accurate Second-Order Pruning for Large Language Modelscited by 115
  3. 2021Pre-trained Language Models in Biomedical Domain: A Systematic Surveycited by 87
  4. 2021Scalable Vision Transformers with Hierarchical Poolingcited by 84
  5. 2024BitDistiller: Unleashing the Potential of Sub-4-Bit LLMs via Self-Distillationcited by 80
  6. 2022Unified Visual Transformer Compressioncited by 79
  7. 2022Compression of Generative Pre-trained Language Models via Quantizationcited by 79
  8. 2022Efficient Methods for Natural Language Processing: A Surveycited by 78
  9. 2023ZeroQuant-V2: Exploring Post-training Quantization in LLMs from Comprehensive Study to Low Rank Compensationcited by 78
  10. 2023Stable and low-precision training for large-scale vision-language modelscited by 74
  11. 2022BiBERT: Accurate Fully Binarized BERTcited by 70
  12. 2021Prune Once for All: Sparse Pre-Trained Language Modelscited by 66
  13. 2023Outlier Suppression+: Accurate quantization of large language models by equivalent and optimal shifting and scalingcited by 62
  14. 2024Model Compression and Efficient Inference for Large Language Models: A Surveycited by 62
  15. 2022NoisyQuant: Noisy Bias-Enhanced Post-Training Activation Quantization for Vision Transformerscited by 59
  16. 2023Training Transformers with 4-bit Integerscited by 58
  17. 2021A Survey on Green Deep Learningcited by 52
  18. 2022BiT: Robustly Binarized Multi-distilled Transformercited by 51
  19. 2023FlexRound: Learnable Rounding based on Element-wise Division for Post-Training Quantizationcited by 46
  20. 2022A Survey on Model Compression and Acceleration for Pretrained Language Modelscited by 45
  21. 2024A Survey on Transformer Compressioncited by 44
  22. 2025ParetoQ: Improving Scaling Laws in Extremely Low-bit LLM Quantizationcited by 44
  23. 2022Model Cascading: Towards Jointly Improving Efficiency and Accuracy of NLP Systemscited by 43
  24. 2021Selective Knowledge Distillation for Neural Machine Translationcited by 42
  25. 2021TR-BERT: Dynamic Token Reduction for Accelerating BERT Inferencecited by 40
  26. 2021Knowledge Inheritance for Pre-trained Language Modelscited by 38
  27. 2021FLAT: An Optimized Dataflow for Mitigating Attention Bottleneckscited by 38
  28. 2022NusaCrowd: Open Source Initiative for Indonesian NLP Resourcescited by 38
  29. 2021Towards Efficient Post-training Quantization of Pre-trained Language Modelscited by 37
  30. 2022Enable Deep Learning on Mobile Devices: Methods, Systems, and Applicationscited by 32
  31. 2021AutoTinyBERT: Automatic Hyper-parameter Optimization for Efficient Pre-trained Language Modelscited by 31
  32. 2024SpikeLM: Towards General Spike-Driven Language Modeling via Elastic Bi-Spiking Mechanismscited by 31
  33. 2024MoPE-CLIP: Structured Pruning for Efficient Vision-Language Models with Module-wise Pruning Error Metriccited by 30
  34. 2024Jetfire: Efficient and Accurate Transformer Pretraining with INT8 Data Flow and Per-Block Quantizationcited by 30
  35. 2024A Survey of Low-bit Large Language Models: Basics, Systems, and Algorithmscited by 30
  36. 2022Auto-ViT-Acc: An FPGA-Aware Automatic Acceleration Framework for Vision Transformer with Mixed-Scheme Quantizationcited by 28
  37. 2021Kronecker Decomposition for GPT Compressioncited by 27
  38. 2022VAQF: Fully Automatic Software-Hardware Co-Design Framework for Low-Bit Vision Transformercited by 27

50 most cited of 145 All 145 (JSON)