mlsanityData to 7 Oct 2026

Binarized Neural Machine Translation

arXiv 2302.04907 ↗First posted 9 Feb 2023cs.CL cs.LGcited by 12

Not on the map. Why?

Abstract (arXiv)

The rapid scaling of language models is motivating research using low-bitwidth quantization. In this work, we propose a novel binarization technique for Transformers applied to machine translation (BMT), the first of its kind. We identify and address the problem of inflated dot-product variance when using one-bit weights and activations. Specifically, BMT leverages additional LayerNorms and residual connections to improve binarization quality. Experiments on the WMT dataset show that a one-bit weight-only Transformer can achieve the same quality as a float one, while being 16x smaller in size. One-bit activations incur varying degrees of quality drop, but mitigated by the proposed architectural changes. We further conduct a scaling law study using production-scale translation datasets, which shows that one-bit weight Transformers scale and generalize well in both in-domain and out-of-domain settings. Implementation in JAX/Flax will be open sourced.

Key facts. Cited by 12 papers in the map (data through 7 Oct 2026). Builds on 28 of its 41 references that are in the map. First posted on arXiv 9 Feb 2023.

Builds on

28 of 41 references.

References · 28

  1. 2017Attention Is All You Needcited by 59,932
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2020Scaling Laws for Neural Language Modelscited by 6,411
  4. 2016Layer Normalizationcited by 5,579
  5. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  6. 2022Training Compute-Optimal Large Language Modelscited by 2,927
  7. 2018A Call for Clarity in Reporting BLEU Scorescited by 1,790
  8. 2020GShard: Scaling Giant Models with Conditional Computation and Automatic Shardingcited by 1,601
  9. 2022Solving Quantitative Reasoning Problems with Language Modelscited by 1,359
  10. 2020BLEURT: Learning Robust Metrics for Text Generationcited by 1,196
  11. 2016XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networkscited by 829
  12. 2016TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systemscited by 660
16 more
  1. 2019Q8BERT: Quantized 8Bit BERTcited by 296
  2. 2022The case for 4-bit precision: k-bit Inference Scaling Lawscited by 247
  3. 2018Bi-Real Net: Enhancing the Performance of 1-bit CNNs With Improved Representational Capability and Advanced Training Algorithmcited by 233
  4. 2020BinaryBERT: Pushing the Limit of BERT Quantizationcited by 175
  5. 2021Scaling Laws for Neural Machine Translationcited by 110
  6. 2020BLEU might be Guilty but References are not Innocentcited by 109
  7. 2021High Quality Rather than High Model Probability: Minimum Bayes Risk Decoding with Neural Metricscited by 104
  8. 2021Characterizing signal propagation to close the performance gap in unnormalized ResNetscited by 87
  9. 2022BiBERT: Accurate Fully Binarized BERTcited by 70
  10. 2019Efficient 8-Bit Quantization of Transformer Neural Machine Language Translation Modelcited by 58
  11. 2022BiT: Robustly Binarized Multi-distilled Transformercited by 51
  12. 2021PokeBNN: A Binary Pursuit of Lightweight Accuracycited by 48
  13. 2021How Do Adam and Training Strategies Help BNNs Optimization?cited by 46
  14. 2021An Evaluation of Edge TPU Accelerators for Convolutional Neural Networkscited by 39
  15. 2019Fully Quantized Transformer for Machine Translationcited by 36
  16. 2019Neural Machine Translation with 4-Bit Precision and Beyondcited by 2

Built on by

Cited by year

Citing papers per year: 2023: 2; 2024: 5; 2025: 5.

12 papers.

  1. 2023BitNet: Scaling 1-bit Transformers for Large Language Modelscited by 175
  2. 2023Understanding the Potential of FPGA-Based Spatial Acceleration for Large Language Model Inferencecited by 29
  3. 2024Scalable MatMul-free Language Modelingcited by 21
  4. 2024Learning from Students: Applying t-Distributions to Explore Accurate and Efficient Formats for LLMscited by 18
  5. 2024Q-S5: Towards Quantized State Space Modelscited by 9
  6. 2025Neuromorphic Principles for Efficient Large Language Models on Intel Loihi 2cited by 6
  7. 2025Neural Network Quantization for Microcontrollers: A Comprehensive Survey of Methods, Platforms, and Applicationscited by 6
  8. 2024Dynamical Mean-Field Theory of Self-Attention Neural Networkscited by 3
  9. 2024Robust Training of Neural Networks at Arbitrary Precision and Sparsitycited by 3
  10. 2025PROM: Prioritize Reduction of Multiplications Over Lower Bit-Widths for Efficient CNNscited by 1
  11. 2025Efficient Certified Reasoning for Binarized Neural Networkscited by 0
  12. 2025From Scarcity to Efficiency: Investigating the Effects of Data Augmentation on African Machine Translationcited by 0