mlsanityData to 7 Oct 2026

Intriguing Properties of Compression on Multilingual Models

arXiv 2211.02738 ↗First posted 4 Nov 2022cs.CL cs.AIcited by 15

Not on the map. Why?

Abstract (arXiv)

Multilingual models are often particularly dependent on scaling to generalize to a growing number of languages. Compression techniques are widely relied upon to reconcile the growth in model size with real world resource constraints, but compression can have a disparate effect on model performance for low-resource languages. It is thus crucial to understand the trade-offs between scale, multilingualism, and compression. In this work, we propose an experimental framework to characterize the impact of sparsifying multilingual pre-trained language models during fine-tuning. Applying this framework to mBERT named entity recognition models across 40 languages, we find that compression confers several intriguing and previously unknown generalization properties. In contrast to prior findings, we find that compression may improve model robustness over dense models. We additionally observe that under certain sparsification regimes compression may aid, rather than disproportionately impact the performance of low-resource languages.

Key facts. Cited by 15 papers in the map (data through 7 Oct 2026). Builds on 51 of its 71 references that are in the map. First posted on arXiv 4 Nov 2022.

Builds on

51 of 71 references.

References · 51

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  4. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  5. 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,907
  6. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  7. 2015Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Codingcited by 2,291
  8. 2015Learning both Weights and Connections for Efficient Neural Networkscited by 1,998
  9. 2019Cross-lingual Language Model Pretrainingcited by 1,088
  10. 2019Are Sixteen Heads Really Better than One?cited by 803
  11. 2020XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalizationcited by 738
  12. 2019How multilingual is Multilingual BERT?cited by 642
38 more
  1. 2019The State of Sparsity in Deep Neural Networkscited by 471
  2. 2018Massively Multilingual Sentence Embeddings for Zero-Shot Cross-Lingual Transfer and Beyondcited by 434
  3. 2020Movement Pruning: Adaptive Sparsity by Fine-Tuningcited by 398
  4. 2019Q-BERT: Hessian Based Ultra Low Precision Quantization of BERTcited by 355
  5. 2019Beto, Bentz, Becas: The Surprising Cross-Lingual Effectiveness of BERTcited by 301
  6. 2020DynaBERT: Dynamic BERT with Adaptive Width and Depthcited by 246
  7. 2020Are All Languages Created Equal in Multilingual BERT?cited by 206
  8. 2020Compressing BERT: Studying the Effects of Weight Pruning on Transfer Learningcited by 184
  9. 2021Block Pruning For Faster Transformerscited by 176
  10. 2020BinaryBERT: Pushing the Limit of BERT Quantizationcited by 175
  11. 2019What Do Compressed Deep Neural Networks Forget?cited by 164
  12. 2020Characterising Bias in Compressed Modelscited by 162
  13. 2020Participatory Research for Low-resourced Machine Translation: A Case Study in African Languagescited by 159
  14. 2021MasakhaNER: Named Entity Recognition for African Languagescited by 155
  15. 2021Composable Sparse Fine-Tuning for Cross-Lingual Transfercited by 144
  16. 2020When Being Unseen from mBERT is just the Beginning: Handling New Languages With Multilingual Language Modelscited by 131
  17. 2020Finding Universal Grammatical Relations in Multilingual BERTcited by 108
  18. 2020Compressing Large-Scale Transformer-Based Models: A Case Study on BERTcited by 95
  19. 2021Learning Compact Metrics for MTcited by 93
  20. 2020Extending Multilingual BERT to Low-Resource Languagescited by 88
  21. 2020EarlyBERT: Efficient BERT Training via Early-bird Lottery Ticketscited by 79
  22. 2022Efficient Methods for Natural Language Processing: A Surveycited by 78
  23. 2020An Analysis of Simple Data Augmentation for Named Entity Recognitioncited by 75
  24. 2021NL-Augmenter: A Framework for Task-Sensitive Natural Language Augmentationcited by 72
  25. 2021The Low-Resource Double Bind: An Empirical Study of Pruning for Low-Resource Machine Translationcited by 46
  26. 2021When is BERT Multilingual? Isolating Crucial Ingredients for Cross-lingual Transfercited by 41
  27. 2019Small and Practical BERT Models for Sequence Labelingcited by 40
  28. 2020XtremeDistil: Multi-stage Distillation for Massive Multilingual Modelscited by 38
  29. 2021Beyond Preserved Accuracy: Evaluating Loyalty and Robustness of BERT Compressioncited by 38
  30. 2020Load What You Need: Smaller Versions of Multilingual BERTcited by 32
  31. 2019Is Multilingual BERT Fluent in Language Generation?cited by 28
  32. 2019Towards Compact and Robust Deep Neural Networkscited by 23
  33. 2020Learning Light-Weight Translation Models from Deep Transformercited by 22
  34. 2021Keep the Gradients Flowing: Using Gradient Flow to Study Sparse Network Optimizationcited by 22
  35. 2021Contributions of Transformer Attention Heads in Multi- and Cross-lingual Taskscited by 16
  36. 2022Toward More Meaningful Resources for Lower-resourced Languagescited by 14
  37. 2021Data Augmentation for Low-Resource Named Entity Recognition Using Backtranslationcited by 10
  38. 2020Towards Supervised and Unsupervised Neural Machine Translation Baselines for Nigerian Pidgincited by 8

50 most cited of 51 All 51 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 6; 2024: 3; 2025: 4; 2026: 2.

15 papers.

Sort:

By area:

  1. 2023Evaluating the Social Impact of Generative AI Systems in Systems and Societycited by 91
  2. 2023Intriguing Properties of Quantization at Scalecited by 48
  3. 2024How Does Quantization Affect Multilingual LLMs?cited by 41
  4. 2023Model Compression in Practice: Lessons Learned from Practitioners Creating On-device Machine Learning Experiencescited by 21
  5. 2023Understanding the Effect of Model Compression on Social Bias in Large Language Modelscited by 19
  6. 2023FAIR-Ensemble: When Fairness Naturally Emerges From Deep Ensemblingcited by 11
  7. 2025Why Do Some Inputs Break Low-Bit LLM Quantization?cited by 8
  8. 2023Dynamic ASR Pathways: An Adaptive Masking Approach Towards Efficient Pruning of A Multilingual ASR Modelcited by 3
  9. 2024DMin: Scalable Training Data Influence Estimation for Diffusion Modelscited by 3
  10. 2024You Never Know: Quantization Induces Inconsistent Biases in Vision-Language Foundation Modelscited by 2
  11. 2025M-Wanda: Improving One-Shot Pruning for Multilingual LLMscited by 2
  12. 2026Multilinguality at the Edge: Developing Language Models for the Global Southcited by 1
3 more
  1. 2025As easy as PIE: understanding when pruning causes language models to disagreecited by 0
  2. 2025The Uneven Impact of Post-Training Quantization in Machine Translationcited by 0
  3. 2026Temporal Taxation Compounds Under Post-Training Compression of Whisper Modelscited by 0