# Compression of Neural Machine Translation Models via Pruning

arXiv: https://arxiv.org/abs/1606.09274 · HTML: https://mlsanity.com/paper/1606.09274/

## Key facts

- Cited by 44 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 29 Jun 2016.

## Built on by (most cited first, 30 of 44)

- [Scaling Language Models: Methods, Analysis & Insights from Training Gopher](https://mlsanity.com/paper/2112.11446/) (2021)
- [The Unreasonable Ineffectiveness of the Deeper Layers](https://mlsanity.com/paper/2403.17887/) (2024)
- [A Survey on Deep Neural Network Pruning-Taxonomy, Comparison, Analysis, and Recommendations](https://mlsanity.com/paper/2308.06767/) (2023)
- [Block Pruning For Faster Transformers](https://mlsanity.com/paper/2109.04838/) (2021)
- [MATE: Multi-view Attention for Table Transformer Efficiency](https://mlsanity.com/paper/2109.04312/) (2021)
- [A Survey on Green Deep Learning](https://mlsanity.com/paper/2111.05193/) (2021)
- [Intriguing Properties of Quantization at Scale](https://mlsanity.com/paper/2305.19268/) (2023)
- [The Low-Resource Double Bind: An Empirical Study of Pruning for Low-Resource Machine Translation](https://mlsanity.com/paper/2110.03036/) (2021)
- [A Survey on Model Compression and Acceleration for Pretrained Language Models](https://mlsanity.com/paper/2202.07105/) (2022)
- [Importance-based Neuron Allocation for Multilingual Neural Machine Translation](https://mlsanity.com/paper/2107.06569/) (2021)
- [Keep the Gradients Flowing: Using Gradient Flow to Study Sparse Network Optimization](https://mlsanity.com/paper/2102.01670/) (2021)
- [Rethinking White-Box Watermarks on Deep Learning Models under Neural Structural Obfuscation](https://mlsanity.com/paper/2303.09732/) (2023)
- [Layer-wise Model Pruning based on Mutual Information](https://mlsanity.com/paper/2108.12594/) (2021)
- [Zero-Knowledge Proof-based Verifiable Decentralized Machine Learning in Communication Network: A Comprehensive Survey](https://mlsanity.com/paper/2310.14848/) (2023)
- [On the Limitations of Compute Thresholds as a Governance Strategy](https://mlsanity.com/paper/2407.05694/) (2024)
- [Pruning-then-Expanding Model for Domain Adaptation of Neural Machine Translation](https://mlsanity.com/paper/2103.13678/) (2021)
- [Don't Complete It! Preventing Unhelpful Code Completion for Productive and Sustainable Neural Code Completion Systems](https://mlsanity.com/paper/2209.05948/) (2022)
- [Personalized Algorithm Generation: A Case Study in Learning ODE Integrators](https://mlsanity.com/paper/2105.01303/) (2021)
- [FedZKP: Federated Model Ownership Verification with Zero-knowledge Proof](https://mlsanity.com/paper/2305.04507/) (2023)
- [The Cost of Compression: Investigating the Impact of Compression on Parametric Knowledge in Language Models](https://mlsanity.com/paper/2312.00960/) (2023)
- [Pruning Attention Heads of Transformer Models Using A* Search: A Novel Approach to Compress Big NLP Architectures](https://mlsanity.com/paper/2110.15225/) (2021)
- [Cracking White-box DNN Watermarks via Invariant Neuron Transforms](https://mlsanity.com/paper/2205.00199/) (2022)
- [Continual Learning of Neural Machine Translation within Low Forgetting Risk Regions](https://mlsanity.com/paper/2211.01542/) (2022)
- [Baseline Pruning-Based Approach to Trojan Detection in Neural Networks](https://mlsanity.com/paper/2101.12016/) (2021)
- [Bridging Fairness and Environmental Sustainability in Natural Language Processing](https://mlsanity.com/paper/2211.04256/) (2022)
- [FedSOV: Federated Model Secure Ownership Verification with Unforgeable Signature](https://mlsanity.com/paper/2305.06085/) (2023)
- [RobWE: Robust Watermark Embedding for Personalized Federated Learning Model Ownership Protection](https://mlsanity.com/paper/2402.19054/) (2024)
- [LCS: Learning Compressible Subspaces for Adaptive Network Compression at Inference Time](https://mlsanity.com/paper/2110.04252/) (2021)
- [Logic Shrinkage: Learned FPGA Netlist Sparsity for Efficient Neural Network Inference](https://mlsanity.com/paper/2112.02346/) (2021)
- [End-to-End Supermask Pruning: Learning to Prune Image Captioning Models](https://mlsanity.com/paper/2110.03298/) (2021)

## Abstract (arXiv)

Neural Machine Translation (NMT), like many other deep learning domains, typically suffers from over-parameterization, resulting in large storage sizes. This paper examines three simple magnitude-based pruning schemes to compress NMT models, namely class-blind, class-uniform, and class-distribution, which differ in terms of how pruning thresholds are computed for the different classes of weights in the NMT architecture. We demonstrate the efficacy of weight pruning as a compression technique for a state-of-the-art NMT system. We show that an NMT model with over 200 million parameters can be pruned by 40% with very little performance loss as measured on the WMT'14 English-German translation task. This sheds light on the distribution of redundancy in the NMT architecture. Our main result is that with retraining, we can recover and even surpass the original performance with an 80%-pruned model.