# Binarized Neural Machine Translation

arXiv: https://arxiv.org/abs/2302.04907 · HTML: https://mlsanity.com/paper/2302.04907/

## Key facts

- Cited by 13 papers in the map (data through 14 Oct 2026).
- Builds on 28 of its 41 references that are in the map.
- First posted on arXiv 9 Feb 2023.

## Builds on (most cited first, 28 of 28)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Scaling Laws for Neural Language Models](https://mlsanity.com/paper/2001.08361/) (2020)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [PaLM: Scaling Language Modeling with Pathways](https://mlsanity.com/paper/2204.02311/) (2022)
- [Training Compute-Optimal Large Language Models](https://mlsanity.com/paper/2203.15556/) (2022)
- [A Call for Clarity in Reporting BLEU Scores](https://mlsanity.com/paper/1804.08771/) (2018)
- [GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding](https://mlsanity.com/paper/2006.16668/) (2020)
- [Solving Quantitative Reasoning Problems with Language Models](https://mlsanity.com/paper/2206.14858/) (2022)
- [BLEURT: Learning Robust Metrics for Text Generation](https://mlsanity.com/paper/2004.04696/) (2020)
- [XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networks](https://mlsanity.com/paper/1603.05279/) (2016)
- [TensorFlow: Large-Scale Machine Learning on Heterogeneous Distributed Systems](https://mlsanity.com/paper/1603.04467/) (2016)
- [Q8BERT: Quantized 8Bit BERT](https://mlsanity.com/paper/1910.06188/) (2019)
- [The case for 4-bit precision: k-bit Inference Scaling Laws](https://mlsanity.com/paper/2212.09720/) (2022)
- [Bi-Real Net: Enhancing the Performance of 1-bit CNNs With Improved Representational Capability and Advanced Training Algorithm](https://mlsanity.com/paper/1808.00278/) (2018)
- [BinaryBERT: Pushing the Limit of BERT Quantization](https://mlsanity.com/paper/2012.15701/) (2020)
- [Scaling Laws for Neural Machine Translation](https://mlsanity.com/paper/2109.07740/) (2021)
- [BLEU might be Guilty but References are not Innocent](https://mlsanity.com/paper/2004.06063/) (2020)
- [High Quality Rather than High Model Probability: Minimum Bayes Risk Decoding with Neural Metrics](https://mlsanity.com/paper/2111.09388/) (2021)
- [Characterizing signal propagation to close the performance gap in unnormalized ResNets](https://mlsanity.com/paper/2101.08692/) (2021)
- [BiBERT: Accurate Fully Binarized BERT](https://mlsanity.com/paper/2203.06390/) (2022)
- [Efficient 8-Bit Quantization of Transformer Neural Machine Language Translation Model](https://mlsanity.com/paper/1906.00532/) (2019)
- [BiT: Robustly Binarized Multi-distilled Transformer](https://mlsanity.com/paper/2205.13016/) (2022)
- [PokeBNN: A Binary Pursuit of Lightweight Accuracy](https://mlsanity.com/paper/2112.00133/) (2021)
- [How Do Adam and Training Strategies Help BNNs Optimization?](https://mlsanity.com/paper/2106.11309/) (2021)
- [An Evaluation of Edge TPU Accelerators for Convolutional Neural Networks](https://mlsanity.com/paper/2102.10423/) (2021)
- [Fully Quantized Transformer for Machine Translation](https://mlsanity.com/paper/1910.10485/) (2019)
- [Neural Machine Translation with 4-Bit Precision and Beyond](https://mlsanity.com/paper/1909.06091/) (2019)

## Built on by (most cited first, 13 of 13)

- [BitNet: Scaling 1-bit Transformers for Large Language Models](https://mlsanity.com/paper/2310.11453/) (2023)
- [Understanding the Potential of FPGA-Based Spatial Acceleration for Large Language Model Inference](https://mlsanity.com/paper/2312.15159/) (2023)
- [Scalable MatMul-free Language Modeling](https://mlsanity.com/paper/2406.02528/) (2024)
- [Learning from Students: Applying t-Distributions to Explore Accurate and Efficient Formats for LLMs](https://mlsanity.com/paper/2405.03103/) (2024)
- [Q-S5: Towards Quantized State Space Models](https://mlsanity.com/paper/2406.09477/) (2024)
- [Neuromorphic Principles for Efficient Large Language Models on Intel Loihi 2](https://mlsanity.com/paper/2503.18002/) (2025)
- [Neural Network Quantization for Microcontrollers: A Comprehensive Survey of Methods, Platforms, and Applications](https://mlsanity.com/paper/2508.15008/) (2025)
- [Dynamical Mean-Field Theory of Self-Attention Neural Networks](https://mlsanity.com/paper/2406.07247/) (2024)
- [Robust Training of Neural Networks at Arbitrary Precision and Sparsity](https://mlsanity.com/paper/2409.09245/) (2024)
- [PROM: Prioritize Reduction of Multiplications Over Lower Bit-Widths for Efficient CNNs](https://mlsanity.com/paper/2505.03254/) (2025)
- [Comprehensive Benchmarking of Binary Neural Networks on NVM Crossbar Architectures](https://mlsanity.com/paper/2308.06227/) (2023)
- [Efficient Certified Reasoning for Binarized Neural Networks](https://mlsanity.com/paper/2507.02916/) (2025)
- [From Scarcity to Efficiency: Investigating the Effects of Data Augmentation on African Machine Translation](https://mlsanity.com/paper/2509.07471/) (2025)

## Abstract (arXiv)

The rapid scaling of language models is motivating research using low-bitwidth quantization. In this work, we propose a novel binarization technique for Transformers applied to machine translation (BMT), the first of its kind. We identify and address the problem of inflated dot-product variance when using one-bit weights and activations. Specifically, BMT leverages additional LayerNorms and residual connections to improve binarization quality. Experiments on the WMT dataset show that a one-bit weight-only Transformer can achieve the same quality as a float one, while being 16x smaller in size. One-bit activations incur varying degrees of quality drop, but mitigated by the proposed architectural changes. We further conduct a scaling law study using production-scale translation datasets, which shows that one-bit weight Transformers scale and generalize well in both in-domain and out-of-domain settings. Implementation in JAX/Flax will be open sourced.