mlsanityData to 13 Oct 2026

Pruning Pretrained Encoders with a Multitask Objective

arXiv 2112.05705 ↗First posted 10 Dec 2021cs.CL cs.LGcited by 0

Not on the map. Why?

Abstract (arXiv)

The sizes of pretrained language models make them challenging and expensive to use when there are multiple desired downstream tasks. In this work, we adopt recent strategies for model pruning during finetuning to explore the question of whether it is possible to prune a single encoder so that it can be used for multiple tasks. We allocate a fixed parameter budget and compare pruning a single model with a multitask objective against the best ensemble of single-task models. We find that under two pruning strategies (element-wise and rank pruning), the approach with the multitask objective outperforms training models separately when averaged across all tasks, and it is competitive on each individual one. Additional analysis finds that using a multitask objective during pruning can also be an effective method for reducing model sizes for low-resource tasks.

Key facts. No citing papers in the map yet (data through 13 Oct 2026). Builds on 27 of its 35 references that are in the map. First posted on arXiv 10 Dec 2021.

Builds on

27 of 35 references.

References · 27

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,304
  2. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,549
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,231
  4. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,281
  5. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,213
  6. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,133
  7. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,898
  8. 2017A Broad-Coverage Challenge Corpus for Sentence Understanding through Inferencecited by 2,676
  9. 2013Estimating or Propagating Gradients Through Stochastic Neurons for Conditional Computationcited by 2,056
  10. 2015Learning both Weights and Connections for Efficient Neural Networkscited by 2,017
  11. 2019TinyBERT: Distilling BERT for Natural Language Understandingcited by 1,157
  12. 2017SemEval-2017 Task 1: Semantic Textual Similarity - Multilingual and Cross-lingual Focused Evaluationcited by 998
15 more
  1. 2018Neural Network Acceptability Judgmentscited by 957
  2. 2019CTRL: A Conditional Transformer Language Model for Controllable Generationcited by 757
  3. 2017To prune, or not to prune: exploring the efficacy of pruning for model compressioncited by 602
  4. 2020MobileBERT: a Compact Task-Agnostic BERT for Resource-Limited Devicescited by 480
  5. 2021Compacter: Efficient Low-Rank Hypercomplex Adapter Layerscited by 459
  6. 2020Movement Pruning: Adaptive Sparsity by Fine-Tuningcited by 399
  7. 2020Parameter-Efficient Transfer Learning with Diff Pruningcited by 352
  8. 2019Multi-Task Deep Neural Networks for Natural Language Understandingcited by 350
  9. 2021Parameter-efficient Multi-task Fine-tuning for Transformers via Shared Hypernetworkscited by 281
  10. 2021Muppet: Massive Multi-task Representations with Pre-Finetuningcited by 192
  11. 2021Block Pruning For Faster Transformerscited by 176
  12. 2020Sparse GPU Kernels for Deep Learningcited by 141
  13. 2021Super Tickets in Pre-Trained Language Models: From Model Compression to Improving Generalizationcited by 56
  14. 2020Learning Low-rank Deep Neural Networks via Singular Vector Orthogonality Regularization and Singular Value Sparsificationcited by 52
  15. 2021Layer-wise Model Pruning based on Mutual Informationcited by 19

Built on by

Not cited yet.