mlsanityData to 14 Oct 2026

BEExformer: A Fast Inferencing Binarized Transformer with Early Exits

arXiv 2412.05225 ↗First posted 6 Dec 2024cs.CL cs.AI cs.NEcited by 0

On the map: Efficient LLM inference and serving → Early exit and edge inference

Abstract (arXiv)

Large Language Models (LLMs) based on transformers achieve cutting-edge results on a variety of applications. However, their enormous size and processing requirements hinder deployment on constrained resources. To enhance efficiency, binarization and Early Exit (EE) have proved to be effective solutions. However, binarization may lead to performance loss as reduced precision affects gradient estimation and parameter updates. Besides, research on EE mechanisms is still in its early stages. To address these challenges, we introduce Binarized Early Exit Transformer (BEExformer), a first-of-its-kind selective learning-based transformer integrating Binarization-Aware Training (BAT) with EE for efficient and fast textual inference. Each transformer block has an integrated Selective-Learn Forget Network (SLFN) to enhance contextual retention while eliminating irrelevant information. The BAT employs a differentiable second-order approximation to the sign function, enabling gradient computation that captures both the sign and magnitude of the weights. This aids in 21.30 times reduction in model size. The EE mechanism hinges on fractional reduction in entropy among intermediate transformer blocks with soft-routing loss estimation. This accelerates inference by reducing FLOPs by 52.27% and even improves accuracy by 3.22% by resolving the "overthinking" problem inherent in deep networks. Extensive evaluation through comparison with the SOTA methods and various ablations across nine datasets covering multiple NLP tasks demonstrates its Pareto-optimal performance-efficiency trade-off.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Early exit and edge inference, one of 12 directions in Efficient LLM inference and serving. Builds on 35 of its 52 references that are in the map. First posted on arXiv 6 Dec 2024.

Builds on

35 of 52 references.

References · 35

  1. 2017Attention Is All You Needcited by 60,361
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,399
  3. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,372
  4. 2020Scaling Laws for Neural Language Modelscited by 6,586
  5. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,899
  6. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,307
  7. 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,790
  8. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,682
  9. 2020Linformer: Self-Attention with Linear Complexitycited by 1,474
  10. 2018Neural Network Acceptability Judgmentscited by 958
  11. 2019Are Sixteen Heads Really Better than One?cited by 816
  12. 2022ZeroQuant: Efficient and Affordable Post-Training Quantization for Large-Scale Transformerscited by 526
23 more
  1. 2019Reducing Transformer Depth on Demand with Structured Dropoutcited by 439
  2. 2021Dynamic Neural Networks: A Surveycited by 408
  3. 2019Q-BERT: Hessian Based Ultra Low Precision Quantization of BERTcited by 359
  4. 2020DeeBERT: Dynamic Early Exiting for Accelerating BERT Inferencecited by 332
  5. 2018SentEval: An Evaluation Toolkit for Universal Sentence Representationscited by 294
  6. 2020BERT Loses Patience: Fast and Robust Inference with Early Exitcited by 273
  7. 2021Does Knowledge Distillation Really Work?cited by 190
  8. 2020BinaryBERT: Pushing the Limit of BERT Quantizationcited by 178
  9. 2021Block Pruning For Faster Transformerscited by 177
  10. 2020TernaryBERT: Distillation-aware Ultra-low Bit BERTcited by 148
  11. 2024EfficientQAT: Efficient Quantization-Aware Training for Large Language Modelscited by 127
  12. 2020A Survey on Text Classification: From Shallow to Deep Learningcited by 80
  13. 2022BiBERT: Accurate Fully Binarized BERTcited by 73
  14. 2022A Comprehensive Survey on Model Quantization for Deep Neural Networks in Image Classificationcited by 65
  15. 2022BiT: Robustly Binarized Multi-distilled Transformercited by 52
  16. 2021Towards Efficient NLP: A Standard Evaluation and A Strong Baselinecited by 44
  17. 2018Bi-Real Net: Binarizing Deep Network Towards Real-Network Performancecited by 42
  18. 2023Efficient Post-training Quantization with FP8 Formatscited by 29
  19. 2023BiBench: Benchmarking and Analyzing Network Binarizationcited by 22
  20. 2021Post-Training Sparsity-Aware Quantizationcited by 17
  21. 2023Quantized Transformer Language Model Implementations on Edge Devicescited by 9
  22. 2024A Survey of Transformer-based Language Models with Focus on Efficiencycited by 6
  23. 2024Training Green AI Models Using Elite Samplescited by 4

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2023BiPFT: Binary Pre-trained Foundation Transformer with Low-rank Estimation of Binarization Residual Polynomials
  2. 2024Normalized Narrow Jump To Conclusions: Normalized Narrow Shortcuts for Parameter Efficient Early Exit Transformer Prediction
  3. 2024Early Exit Is a Natural Capability in Transformer-based Models: An Empirical Study on Early Exit without Joint Optimization
  4. 2024Mechanistic Interpretability of Binary and Ternary Transformers
  5. 2023BitNet: Scaling 1-bit Transformers for Large Language Models
  6. 2026LEAP: Layer-wise Exit-Aware Pretraining for Efficient Transformer Inference
  7. 2025One Jump Is All You Need: Short-Cutting Transformers for Early Exit Prediction with One Jump to Fit All Exit Levels
  8. 2026Trust, but Verify: Peeling Low-Bit Transformer Networks for Training Monitoring