mlsanityData to 7 Oct 2026

BiT: Robustly Binarized Multi-distilled Transformer

arXiv 2205.13016 ↗First posted 25 May 2022cs.LG cs.CLcited by 51

Not on the map. Why?

Abstract (arXiv)

Modern pre-trained transformers have rapidly advanced the state-of-the-art in machine learning, but have also grown in parameters and computational complexity, making them increasingly difficult to deploy in resource-constrained environments. Binarization of the weights and activations of the network can significantly alleviate these issues, however, is technically challenging from an optimization perspective. In this work, we identify a series of improvements that enables binary transformers at a much higher accuracy than what was possible previously. These include a two-set binarization scheme, a novel elastic binary activation function with learned parameters, and a method to quantize a network to its limit by successively distilling higher precision models into lower precision students. These approaches allow for the first time, fully binarized transformer models that are at a practical level of accuracy, approaching a full-precision BERT baseline on the GLUE language understanding benchmark within as little as 5.9%. Code and models are available at: https://github.com/facebookresearch/bit.

Key facts. Cited by 51 papers in the map (data through 7 Oct 2026). Builds on 42 of its 67 references that are in the map. First posted on arXiv 25 May 2022.

Builds on

42 of 67 references.

References · 42

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  6. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  7. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  8. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  9. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  10. 2016Gaussian Error Linear Units (GELUs)cited by 3,431
  11. 2013Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computationcited by 2,031
  12. 2017SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluationcited by 998
30 more
  1. 2016XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networkscited by 829
  2. 2021A Survey of Quantization Methods for Efficient Neural Network Inferencecited by 798
  3. 2019Learned Step Size Quantizationcited by 604
  4. 2016DoReFa-Net: Training Low Bitwidth Convolutional Neural Networks with Low Bitwidth Gradientscited by 557
  5. 2018PACT: Parameterized Clipping Activation for Quantized Neural Networkscited by 456
  6. 2016Quantized Neural Networks: Training Neural Networks with Low Precision Weights and Activationscited by 413
  7. 2019Q-BERT: Hessian Based Ultra Low Precision Quantization of BERTcited by 355
  8. 2019Q8BERT: Quantized 8Bit BERTcited by 296
  9. 2018LQ-Nets: Learned Quantization for Highly Accurate and Compact Deep Neural Networkscited by 272
  10. 2019Differentiable Soft Quantization: Bridging Full-Precision and Low-Bit Neural Networkscited by 234
  11. 2018Bi-Real Net: Enhancing the Performance of 1-bit CNNs With Improved Representational Capability and Advanced Training Algorithmcited by 233
  12. 2020ReActNet: Towards Precise Binary Neural Network with Generalized Activation Functionscited by 183
  13. 2020BinaryBERT: Pushing the Limit of BERT Quantizationcited by 175
  14. 2019Forward and Backward Information Retention for Accurate Binary Neural Networkscited by 169
  15. 2018Training Deep Neural Networks with 8-bit Floating Point Numberscited by 163
  16. 2018Learning to Quantize Deep Networks by Optimizing Quantization Intervals with Task Losscited by 157
  17. 2020Training with Quantization Noise for Extreme Model Compressioncited by 145
  18. 2020TernaryBERT: Distillation-aware Ultra-low Bit BERTcited by 145
  19. 2019Additive Powers-of-Two Quantization: An Efficient Non-uniform Discretization for Neural Networkscited by 143
  20. 2021Understanding and Overcoming the Challenges of Efficient Transformer Quantizationcited by 139
  21. 2020Training Binary Neural Networks with Real-to-Binary Convolutionscited by 111
  22. 2020GOBO: Quantizing Attention-Based NLP Models for Low Latency and Energy Efficient Inferencecited by 103
  23. 2016Convolutional Neural Networks using Logarithmic Data Representationcited by 97
  24. 2019Knowledge Distillation from Internal Representationscited by 87
  25. 2021Nonuniform-to-Uniform Quantization: Towards Accurate Quantization via Generalized Straight-Through Estimationcited by 78
  26. 2019Towards Unified INT8 Training for Convolutional Neural Networkcited by 77
  27. 2017Towards Effective Low-bitwidth Convolutional Neural Networkscited by 76
  28. 2022BiBERT: Accurate Fully Binarized BERTcited by 70
  29. 2019QKD: Quantization-aware Knowledge Distillationcited by 42
  30. 2018Synetgy: Algorithm-hardware Co-design for ConvNet Accelerators on Embedded FPGAscited by 18

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 3; 2023: 15; 2024: 12; 2025: 13; 2026: 8.

51 papers.

Sort:

By area:

  1. 2023LLM-QAT: Data-Free Quantization Aware Training for Large Language Modelscited by 320
  2. 2023BitNet: Scaling 1-bit Transformers for Large Language Modelscited by 175
  3. 2023PB-LLM: Partially Binarized Large Language Modelscited by 80
  4. 2025ParetoQ: Improving Scaling Laws in Extremely Low-bit LLM Quantizationcited by 44
  5. 2023A Survey of Techniques for Optimizing Transformer Inferencecited by 35
  6. 2024SpikeLM: Towards General Spike-Driven Language Modeling via Elastic Bi-Spiking Mechanismscited by 31
  7. 2022BiViT: Extremely Compressed Binary Vision Transformercited by 28
  8. 2023BiBench: Benchmarking and Analyzing Network Binarizationcited by 22
  9. 2024Scalable MatMul-free Language Modelingcited by 21
  10. 2023BinaryViT: Pushing Binary Vision Transformers Towards Convolutional Modelscited by 17
  11. 2023Quantization Variation: A New Perspective on Training Transformers with Low-Bit Precisioncited by 17
  12. 2023ShiftAddViT: Mixture of Multiplication Primitives Towards Efficient Vision Transformercited by 15
38 more
  1. 2024A Comprehensive Survey of Compression Algorithms for Language Modelscited by 15
  2. 2023Binarized Neural Machine Translationcited by 12
  3. 2023BinaryViT: Towards Efficient and Accurate Binary Vision Transformerscited by 7
  4. 2023Binary and Ternary Natural Language Generationcited by 7
  5. 2025Binarized Mamba-Transformer for Lightweight Quad Bayer HybridEVS Demosaicingcited by 7
  6. 2024BETA: Binarized Energy-Efficient Transformer Accelerator at the Edgecited by 6
  7. 2024Sorbet: A Neuromorphic Hardware-Compatible Transformer-Based Spiking Language Modelcited by 6
  8. 2025BHViT: Binarized Hybrid Vision Transformercited by 5
  9. 2025Otters: An Energy-Efficient SpikingTransformer via Optical Time-to-First-Spike Encodingcited by 5
  10. 2022Efficient Speech Representation Learning with Low-Bit Quantizationcited by 4
  11. 2023A Comprehensive Review and a Taxonomy of Edge Machine Learning: Requirements, Paradigms, and Techniquescited by 4
  12. 2023GSB: Group Superposition Binarization for Vision Transformer with Limited Training Samplescited by 4
  13. 2025COBRA: Algorithm-Architecture Co-optimized Binary Transformer Accelerator for Edge Inferencecited by 4
  14. 2025Compute-Optimal Quantization-Aware Trainingcited by 4
  15. 2024BOLD: Boolean Logic Deep Learningcited by 3
  16. 2024Dynamical Mean-Field Theory of Self-Attention Neural Networkscited by 3
  17. 2024Towards Narrowing the Generalization Gap in Deep Boolean Networkscited by 3
  18. 2026FAAR: Format-Aware Adaptive Rounding for NVFP4cited by 3
  19. 2026WinQ: Accelerating Quantization-Aware Training of Language Models Around Saddle Pointscited by 3
  20. 2024Co-Designing Binarized Transformer and Hardware Accelerator for Efficient End-to-End Edge Deploymentcited by 2
  21. 2025Highly Efficient and Effective LLMs with Multi-Boolean Architecturescited by 2
  22. 2025FraQAT: Quantization Aware Training with Fractional bitscited by 2
  23. 2026HBVLA: Pushing 1-Bit Post-Training Quantization for Vision-Language-Action Modelscited by 2
  24. 2023BPT: Binary Point Cloud Transformer for Place Recognitioncited by 1
  25. 2024FBPT: A Fully Binary Point Transformercited by 1
  26. 2024BiDense: Binarization for Dense Predictioncited by 1
  27. 2025Binary Neural Networks for Large Language Model: A Surveycited by 1
  28. 2025BiVM: Accurate Binarized Neural Network for Efficient Video Mattingcited by 1
  29. 2022Boosting Binary Neural Networks via Dynamic Thresholds Learningcited by 0
  30. 2023Attention or Convolution: Transformer Encoders in Audio Language Models for Inference Efficiencycited by 0
  31. 2024BEExformer: A Fast Inferencing Binarized Transformer with Early Exitscited by 0
  32. 2025HadamRNN: Binary and Sparse Ternary Orthogonal RNNscited by 0
  33. 2025High-Fidelity Differential-information Driven Binary Vision Transformercited by 0
  34. 2025BiHDTrans: binary hyperdimensional transformer for efficient multivariate time series classificationcited by 0
  35. 2026SpikySpace: A Spiking State Space Model for Energy-Efficient Time Series Forecastingcited by 0
  36. 2026Matterhorn: Masked Time-to-First-Spike Encoding by Reassigning the Silent State for Sparse and Energy-Efficient Spiking Transformerscited by 0
  37. 2026pQuant: Towards Effective Low-Bit Language Models via Decoupled Linear Quantization-Aware Trainingcited by 0
  38. 2026Otters++: A Time-to-first-spike Based Energy Efficient Optical Spiking Transformercited by 0

50 most cited of 51 All 51 (JSON)