mlsanityData to 7 Oct 2026

Compression of Neural Machine Translation Models via Pruning

arXiv 1606.09274 ↗First posted 29 Jun 2016cs.AI cs.CL cs.NEcited by 43

Not on the map. Why?

Abstract (arXiv)

Neural Machine Translation (NMT), like many other deep learning domains, typically suffers from over-parameterization, resulting in large storage sizes. This paper examines three simple magnitude-based pruning schemes to compress NMT models, namely class-blind, class-uniform, and class-distribution, which differ in terms of how pruning thresholds are computed for the different classes of weights in the NMT architecture. We demonstrate the efficacy of weight pruning as a compression technique for a state-of-the-art NMT system. We show that an NMT model with over 200 million parameters can be pruned by 40% with very little performance loss as measured on the WMT'14 English-German translation task. This sheds light on the distribution of redundancy in the NMT architecture. Our main result is that with retraining, we can recover and even surpass the original performance with an 80%-pruned model.

Key facts. Cited by 43 papers in the map (data through 7 Oct 2026). First posted on arXiv 29 Jun 2016.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 18; 2022: 7; 2023: 9; 2024: 5; 2025: 1; 2026: 3.

43 papers.

Sort:

By area:

  1. 2021Scaling Language Models: Methods, Analysis & Insights from Training Gophercited by 1,199
  2. 2024The Unreasonable Ineffectiveness of the Deeper Layerscited by 219
  3. 2023A Survey on Deep Neural Network Pruning-Taxonomy, Comparison, Analysis, and Recommendationscited by 200
  4. 2021Block Pruning For Faster Transformerscited by 176
  5. 2021MATE: Multi-view Attention for Table Transformer Efficiencycited by 54
  6. 2021A Survey on Green Deep Learningcited by 52
  7. 2023Intriguing Properties of Quantization at Scalecited by 48
  8. 2021The Low-Resource Double Bind: An Empirical Study of Pruning for Low-Resource Machine Translationcited by 46
  9. 2022A Survey on Model Compression and Acceleration for Pretrained Language Modelscited by 45
  10. 2021Importance-based Neuron Allocation for Multilingual Neural Machine Translationcited by 23
  11. 2021Keep the Gradients Flowing: Using Gradient Flow to Study Sparse Network Optimizationcited by 22
  12. 2023Rethinking White-Box Watermarks on Deep Learning Models under Neural Structural Obfuscationcited by 20
31 more
  1. 2021Layer-wise Model Pruning based on Mutual Informationcited by 19
  2. 2023Zero-Knowledge Proof-based Verifiable Decentralized Machine Learning in Communication Network: A Comprehensive Surveycited by 18
  3. 2024On the Limitations of Compute Thresholds as a Governance Strategycited by 17
  4. 2021Pruning-then-Expanding Model for Domain Adaptation of Neural Machine Translationcited by 15
  5. 2022Don't Complete It! Preventing Unhelpful Code Completion for Productive and Sustainable Neural Code Completion Systemscited by 12
  6. 2023FedZKP: Federated Model Ownership Verification with Zero-knowledge Proofcited by 9
  7. 2023The Cost of Compression: Investigating the Impact of Compression on Parametric Knowledge in Language Modelscited by 9
  8. 2021Personalized Algorithm Generation: A Case Study in Learning ODE Integratorscited by 8
  9. 2021Pruning Attention Heads of Transformer Models Using A* Search: A Novel Approach to Compress Big NLP Architecturescited by 6
  10. 2022Continual Learning of Neural Machine Translation within Low Forgetting Risk Regionscited by 6
  11. 2021Baseline Pruning-Based Approach to Trojan Detection in Neural Networkscited by 5
  12. 2022Cracking White-box DNN Watermarks via Invariant Neuron Transformscited by 5
  13. 2022Bridging Fairness and Environmental Sustainability in Natural Language Processingcited by 5
  14. 2021LCS: Learning Compressible Subspaces for Adaptive Network Compression at Inference Timecited by 3
  15. 2021Logic Shrinkage: Learned FPGA Netlist Sparsity for Efficient Neural Network Inferencecited by 3
  16. 2023FedSOV: Federated Model Secure Ownership Verification with Unforgeable Signaturecited by 3
  17. 2024RobWE: Robust Watermark Embedding for Personalized Federated Learning Model Ownership Protectioncited by 3
  18. 2021End-to-End Supermask Pruning: Learning to Prune Image Captioning Modelscited by 2
  19. 2025TELL-TALE: Task Efficient LLMs with Task Aware Layer Eliminationcited by 2
  20. 2021Recurrent Stacking of Layers in Neural Networks: An Application to Neural Machine Translationcited by 1
  21. 2022An Embarrassingly Simple Approach for Intellectual Property Rights Protection on Recurrent Neural Networkscited by 1
  22. 2023Revisiting Offline Compression: Going Beyond Factorization-based Methods for Transformer Language Modelscited by 1
  23. 2024Protecting Deep Learning Model Copyrights with Adversarial Example-Free Reuse Detectioncited by 1
  24. 2024HAPM -- Hardware Aware Pruning Method for CNN hardware accelerators in resource constrained devicescited by 1
  25. 2021ES-ENAS: Efficient Evolutionary Optimization for Large Hybrid Search Spacescited by 0
  26. 2021A Statistician Teaches Deep Learningcited by 0
  27. 2022Sharp asymptotics on the compression of two-layer neural networkscited by 0
  28. 2023Adapter Pruning using Tropical Characterizationcited by 0
  29. 2026Are the High-weight Neurons the Important Ones in Image Classification Neural Networks?cited by 0
  30. 2026Efficient Multilingual Neural Machine Translation via Corpus-Driven Vocabulary Pruning: An English-Arabic Case Studycited by 0
  31. 2026FedASAP: Activation Statistics-driven Structured Adaptive Pruning for Efficient Personalized Federated Learning for Lesion Segmentation on brain MRIcited by 0