mlsanityData to 7 Oct 2026

BEBERT: Efficient and Robust Binary Ensemble BERT

arXiv 2210.15976 ↗First posted 28 Oct 2022cs.CL cs.LGcited by 13

Not on the map. Why?

Abstract (arXiv)

Pre-trained BERT models have achieved impressive accuracy on natural language processing (NLP) tasks. However, their excessive amount of parameters hinders them from efficient deployment on edge devices. Binarization of the BERT models can significantly alleviate this issue but comes with a severe accuracy drop compared with their full-precision counterparts. In this paper, we propose an efficient and robust binary ensemble BERT (BEBERT) to bridge the accuracy gap. To the best of our knowledge, this is the first work employing ensemble techniques on binary BERTs, yielding BEBERT, which achieves superior accuracy while retaining computational efficiency. Furthermore, we remove the knowledge distillation procedures during ensemble to speed up the training process without compromising accuracy. Experimental results on the GLUE benchmark show that the proposed BEBERT significantly outperforms the existing binary BERT models in accuracy and robustness with a 2x speedup on training time. Moreover, our BEBERT has only a negligible accuracy loss of 0.3% compared to the full-precision baseline while saving 15x and 13x in FLOPs and model size, respectively. In addition, BEBERT also outperforms other compressed BERTs in accuracy by up to 6.7%.

Key facts. Cited by 13 papers in the map (data through 7 Oct 2026). Builds on 21 of its 33 references that are in the map. First posted on arXiv 28 Oct 2022.

Builds on

21 of 33 references.

References · 21

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  3. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  4. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  5. 2020MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformerscited by 1,414
  6. 2019TinyBERT: Distilling BERT for Natural Language Understandingcited by 1,153
  7. 2020MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devicescited by 476
  8. 2019Q-BERT: Hessian Based Ultra Low Precision Quantization of BERTcited by 355
  9. 2019Q8BERT: Quantized 8Bit BERTcited by 296
  10. 2020Towards Understanding Ensemble, Knowledge Distillation and Self-Distillation in Deep Learningcited by 292
  11. 2021I-BERT: Integer-only BERT Quantizationcited by 250
  12. 2020DynaBERT: Dynamic BERT with Adaptive Width and Depthcited by 246
9 more
  1. 2020BinaryBERT: Pushing the Limit of BERT Quantizationcited by 175
  2. 2020TernaryBERT: Distillation-aware Ultra-low Bit BERTcited by 145
  3. 2020EarlyBERT: Efficient BERT Training via Early-bird Lottery Ticketscited by 79
  4. 2022BiBERT: Accurate Fully Binarized BERTcited by 70
  5. 2018Binary Ensemble Neural Network: More Bits per Network or More Networks per Bit?cited by 38
  6. 2021Noise Stability Regularization for Improving BERT Fine-tuningcited by 29
  7. 2022An Algorithm-Hardware Co-Optimized Framework for Accelerating N:M Sparse Transformerscited by 28
  8. 2021Elbert: Fast Albert with Confidence-Window Based Early Exitcited by 14
  9. 2021Dynamic Binary Neural Network by learning channel-wise thresholdscited by 13

Built on by

Cited by year

Citing papers per year: 2023: 3; 2024: 5; 2025: 5.

13 papers.

Sort:

By area:

  1. 2023A Survey of Techniques for Optimizing Transformer Inferencecited by 35
  2. 2025Wanda++: Pruning Large Language Models via Regional Gradientscited by 26
  3. 2024AdaZeta: Adaptive Zeroth-Order Tensor-Train Adaption for Memory-Efficient Large Language Models Fine-Tuningcited by 25
  4. 2023Quantization Variation: A New Perspective on Training Transformers with Low-Bit Precisioncited by 17
  5. 2024LoRETTA: Low-Rank Economic Tensor-Train Adaptation for Ultra-Low-Parameter Fine-Tuning of Large Language Modelscited by 12
  6. 2023A Precision-Scalable RISC-V DNN Processor with On-Device Learning Capability at the Extreme Edgecited by 10
  7. 2024Anda: Unlocking Efficient LLM Inference with a Variable-Length Grouped Activation Data Formatcited by 8
  8. 2024BETA: Binarized Energy-Efficient Transformer Accelerator at the Edgecited by 6
  9. 2025FLAT-LLM: Fine-grained Low-rank Activation Space Transformation for Large Language Model Compressioncited by 5
  10. 2025Ultra Memory-Efficient On-FPGA Training of Transformers via Tensor-Compressed Optimizationcited by 3
  11. 2024Co-Designing Binarized Transformer and Hardware Accelerator for Efficient End-to-End Edge Deploymentcited by 2
  12. 2025FedPaI: Achieving Extreme Sparsity in Federated Learning via Pruning at Initializationcited by 1
1 more
  1. 2025APT-LLM: Exploiting Arbitrary-Precision Tensor Core Computing for LLM Accelerationcited by 1