Probing Structured Pruning on Multilingual Pre-trained Models: Settings, Algorithms, and Efficiency
Not on the map. Why?
Abstract (arXiv)
Structured pruning has been extensively studied on monolingual pre-trained language models and is yet to be fully evaluated on their multilingual counterparts. This work investigates three aspects of structured pruning on multilingual pre-trained language models: settings, algorithms, and efficiency. Experiments on nine downstream tasks show several counter-intuitive phenomena: for settings, individually pruning for each language does not induce a better result; for algorithms, the simplest method performs the best; for efficiency, a fast model does not imply that it is also small. To facilitate the comparison on all sparsity levels, we present Dynamic Sparsification, a simple approach that allows training the model once and adapting to different model sizes at inference. We hope this work fills the gap in the study of structured pruning on multilingual pre-trained models and sheds light on future research.
Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 39 of its 48 references that are in the map. First posted on arXiv 6 Apr 2022.
Builds on
39 of 48 references.
References · 39
- 2017Attention Is All You Needcited by 59,932
- 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
- 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
- 2015Distilling the Knowledge in a Neural Networkcited by 10,076
- 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
- 2019Unsupervised Cross-lingual Representation Learning at Scalecited by 3,907
- 2019ALBERT: A Lite BERT for Self-supervised Learning of Language Representationscited by 2,772
- 2020mT5: A massively multilingual pre-trained text-to-text transformercited by 1,813
- 2020MiniLM: Deep Self-Attention Distillation for Task-Agnostic Compression of Pre-Trained Transformerscited by 1,414
- 2019Cross-lingual Language Model Pretrainingcited by 1,088
- 2018XNLI: Evaluating Cross-lingual Sentence Representationscited by 913
- 2019Analyzing Multi-Head Self-Attention: Specialized Heads Do the Heavy Lifting, the Rest Can Be Prunedcited by 844
27 more
- 2019Are Sixteen Heads Really Better than One?cited by 803
- 2020XTREME: A Massively Multilingual Multi-task Benchmark for Evaluating Cross-lingual Generalizationcited by 738
- 2019On the Cross-lingual Transferability of Monolingual Representationscited by 575
- 2017Learning Sparse Neural Networks through L0 Regularizationcited by 571
- 2019CCNet: Extracting High Quality Monolingual Datasets from Web Crawl Datacited by 544
- 2021Sparsity in Deep Learning: Pruning and growth for efficient inference and training in neural networkscited by 521
- 2020TyDi QA: A Benchmark for Information-Seeking Question Answering in Typologically Diverse Languagescited by 486
- 2018Massively Multilingual Sentence Embeddings for Zero-Shot Cross-Lingual Transfer and Beyondcited by 434
- 2019MLQA: Evaluating Cross-lingual Extractive Question Answeringcited by 339
- 2020DeeBERT: Dynamic Early Exiting for Accelerating BERT Inferencecited by 322
- 2019PAWS-X: A Cross-lingual Adversarial Dataset for Paraphrase Identificationcited by 262
- 2019Massively Multilingual Neural Machine Translationcited by 246
- 2020DynaBERT: Dynamic BERT with Adaptive Width and Depthcited by 246
- 2019Structured Pruning of Large Language Modelscited by 201
- 2020HAT: Hardware-Aware Transformers for Efficient Natural Language Processingcited by 136
- 2020The Right Tool for the Job: Matching Model and Instance Complexitiescited by 125
- 2020On the Effect of Dropping Layers of Pre-trained Transformer Modelscited by 118
- 2020When BERT Plays the Lottery, All Tickets Are Winningcited by 114
- 2021Larger-Scale Transformers for Multilingual Masked Language Modelingcited by 86
- 2021Super Tickets in Pre-Trained Language Models: From Model Compression to Improving Generalizationcited by 56
- 2021Learning Language Specific Sub-network for Multilingual Machine Translationcited by 48
- 2020VECO: Variable and Flexible Cross-lingual Pre-training for Language Understanding and Generationcited by 34
- 2021Importance-based Neuron Allocation for Multilingual Neural Machine Translationcited by 23
- 2020Deep Transformers with Latent Depthcited by 17
- 2021An Efficient Transformer Decoder with Compressed Sub-layerscited by 10
- 2021Adaptive Sparse Transformer for Multilingual Translationcited by 8
- 2021On the Prunability of Attention Heads in Multilingual BERTcited by 4
Built on by
Citing papers per year: 2022: 1; 2023: 2.
3 papers.