mlsanityData to 7 Oct 2026

What Do Compressed Multilingual Machine Translation Models Forget?

arXiv 2205.10828 ↗First posted 22 May 2022cs.CL cs.AI cs.LGcited by 9

Not on the map. Why?

Abstract (arXiv)

Recently, very large pre-trained models achieve state-of-the-art results in various natural language processing (NLP) tasks, but their size makes it more challenging to apply them in resource-constrained environments. Compression techniques allow to drastically reduce the size of the models and therefore their inference time with negligible impact on top-tier metrics. However, the general performance averaged across multiple tasks and/or languages may hide a drastic performance drop on under-represented features, which could result in the amplification of biases encoded by the models. In this work, we assess the impact of compression methods on Multilingual Neural Machine Translation models (MNMT) for various language groups, gender, and semantic biases by extensive analysis of compressed models on different machine translation benchmarks, i.e. FLORES-101, MT-Gender, and DiBiMT. We show that the performance of under-represented languages drops significantly, while the average BLEU metric only slightly decreases. Interestingly, the removal of noisy memorization with compression leads to a significant improvement for some medium-resource languages. Finally, we demonstrate that compression amplifies intrinsic gender and semantic biases, even in high-resource languages. Code: https://github.com/alirezamshi/bias-compressedMT

Key facts. Cited by 9 papers in the map (data through 7 Oct 2026). Builds on 52 of its 62 references that are in the map. First posted on arXiv 22 May 2022.

Builds on

52 of 62 references.

References · 52

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  5. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  6. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,801
  7. 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,250
  8. 2019Transformer-XL: Attentive Language Models Beyond a Fixed-Length Contextcited by 1,830
  9. 2018A Call for Clarity in Reporting BLEU Scorescited by 1,790
  10. 2019Energy and Policy Considerations for Deep Learning in NLPcited by 1,242
  11. 2016Bag of Tricks for Efficient Text Classificationcited by 869
  12. 2016Sequence-Level Knowledge Distillationcited by 761
38 more
  1. 2017To prune, or not to prune: exploring the efficacy of pruning for model compressioncited by 596
  2. 2021Carbon Emissions and Large Neural Network Trainingcited by 561
  3. 2020Beyond English-Centric Multilingual Machine Translationcited by 559
  4. 2021The FLORES-101 Evaluation Benchmark for Low-Resource and Multilingual Machine Translationcited by 556
  5. 2019The State of Sparsity in Deep Neural Networkscited by 471
  6. 2016FastText.zip: Compressing text classification modelscited by 331
  7. 2020Multilingual Translation with Extensible Multilingual Pretraining and Finetuningcited by 287
  8. 2020Improving Massively Multilingual Neural Machine Translation and Zero-Shot Translationcited by 277
  9. 2021Pruning and Quantization for Deep Neural Network Acceleration: A Surveycited by 259
  10. 2021I-BERT: Integer-only BERT Quantizationcited by 250
  11. 2019Evaluating Gender Bias in Machine Translationcited by 247
  12. 2019Massively Multilingual Neural Machine Translationcited by 246
  13. 2019Analyzing the Structure of Attention in a Transformer Language Modelcited by 241
  14. 2020Integer Quantization for Deep Learning Inference: Principles and Empirical Evaluationcited by 183
  15. 2019CCMatrix: Mining Billions of High-Quality Parallel Sentences on the WEBcited by 177
  16. 2022DeepNet: Scaling Transformers to 1,000 Layerscited by 176
  17. 2022QDrop: Randomly Dropping Quantization for Extremely Low-bit Post-Training Quantizationcited by 175
  18. 2019What Do Compressed Deep Neural Networks Forget?cited by 164
  19. 2020Characterising Bias in Compressed Modelscited by 162
  20. 2021The Curious Case of Hallucinations in Neural Machine Translationcited by 161
  21. 2021Efficient Deep Learning: A Survey on Making Deep Learning Models Smaller, Faster, and Bettercited by 153
  22. 2021Understanding and Overcoming the Challenges of Efficient Transformer Quantizationcited by 139
  23. 2019CCAligned: A Massive Collection of Cross-Lingual Web-Document Pairscited by 136
  24. 2021Sustainable Modular Debiasing of Language Modelscited by 98
  25. 2021Larger-Scale Transformers for Multilingual Masked Language Modelingcited by 86
  26. 2022Compression of Generative Pre-trained Language Models via Quantizationcited by 79
  27. 2020Gender in Danger? Evaluating Speech Translation Technology on the MuST-SHE Corpuscited by 57
  28. 2021Stereotype and Skew: Quantifying Gender Bias in Pre-trained and Fine-tuned Language Modelscited by 57
  29. 2021The Low-Resource Double Bind: An Empirical Study of Pruning for Low-Resource Machine Translationcited by 46
  30. 2021Selective Knowledge Distillation for Neural Machine Translationcited by 42
  31. 2021Beyond Preserved Accuracy: Evaluating Loyalty and Robustness of BERT Compressioncited by 38
  32. 2018Alternating Multi-bit Quantization for Recurrent Neural Networkscited by 30
  33. 2021Gender Bias Amplification During Speed-Quality Optimization in Neural Machine Translationcited by 28
  34. 2022Under the Morphosyntactic Lens: A Multifaceted Evaluation of Gender Bias in Speech Translationcited by 23
  35. 2020Learning Light-Weight Translation Models from Deep Transformercited by 22
  36. 2020Gender Coreference and Bias Evaluation at WMT 2020cited by 21
  37. 2022A Study of Gender Impact in Self-supervised Models for Speech-to-Text Systemscited by 14
  38. 2022TextPruner: A Model Pruning Toolkit for Pre-Trained Language Modelscited by 10

50 most cited of 52 All 52 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 4; 2023: 2; 2024: 0; 2025: 1; 2026: 2.

9 papers.

  1. 2022Efficient Methods for Natural Language Processing: A Surveycited by 78
  2. 2023Mitigating Hallucinations and Off-target Machine Translation with Source-Contrastive and Language-Contrastive Decodingcited by 42
  3. 2022RQUGE: Reference-Free Metric for Evaluating Question Generation by Answering the Questioncited by 17
  4. 2022SMaLL-100: Introducing Shallow Multilingual Machine Translation Model for Low-Resource Languagescited by 16
  5. 2022Too Brittle To Touch: Comparing the Stability of Quantization and Distillation Towards Developing Lightweight Low-Resource MT Modelscited by 5
  6. 2023Investigating Multi-Pivot Ensembling with Massively Multilingual Machine Translation Modelscited by 5
  7. 2025Fair-GPTQ: Bias-Aware Quantization for Large Language Modelscited by 0
  8. 2026Efficient Multilingual Neural Machine Translation via Corpus-Driven Vocabulary Pruning: An English-Arabic Case Studycited by 0
  9. 2026Unknown-Traffic Detection, Calibration and Shortcut Reliance in Distilled Encrypted-Traffic Classifiers over One Yearcited by 0