Data to 7 Oct 2026

Parameter-Efficient Transfer Learning with Diff Pruning

arXiv 2012.07463 ↗First posted 14 Dec 2020cs.CL cs.LGcited by 351

Not on the map. Why?

Abstract (arXiv)

While task-specific finetuning of pretrained networks has led to significant empirical advances in NLP, the large size of networks makes finetuning difficult to deploy in multi-task, memory-constrained settings. We propose diff pruning as a simple approach to enable parameter-efficient transfer learning within the pretrain-finetune framework. This approach views finetuning as learning a task-specific diff vector that is applied on top of the pretrained parameter vector, which remains fixed and is shared across different tasks. The diff vector is adaptively pruned during training with a differentiable approximation to the L0-norm penalty to encourage sparsity. Diff pruning becomes parameter-efficient as the number of tasks increases, as it requires storing only the nonzero positions and weights of the diff vector for each task, while the cost of storing the shared pretrained model remains constant. It further does not require access to all tasks during training, which makes it attractive in settings where tasks arrive in stream or the set of tasks is unknown. We find that models finetuned with diff pruning can match the performance of fully finetuned baselines on the GLUE benchmark while only modifying 0.5% of the pretrained model's parameters per task.

Key facts. Cited by 351 papers in the map (data through 7 Oct 2026). First posted on arXiv 14 Dec 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 22; 2022: 50; 2023: 83; 2024: 93; 2025: 73; 2026: 30.

351 papers.

Sort:

By area:

2022Visual Prompt Tuningcited by 1,674
2021BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-modelscited by 1,234
2022Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learningcited by 886
2021Towards a Unified View of Parameter-Efficient Transfer Learningcited by 863
2022AdaptFormer: Adapting Vision Transformers for Scalable Visual Recognitioncited by 714
2024Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Surveycited by 641
2023Bias and Fairness in Large Language Models: A Surveycited by 626
2021Fast Model Editing at Scalecited by 491
2021Recent Advances in Natural Language Processing via Large Pre-Trained Language Models: A Surveycited by 457
2023AdaLoRA: Adaptive Budget Allocation for Parameter-Efficient Fine-Tuningcited by 337
2021VL-Adapter: Parameter-Efficient Transfer Learning for Vision-and-Language Taskscited by 319
2024PiSSA: Principal Singular Values and Singular Vectors Adaptation of Large Language Modelscited by 261

38 more

2022Self-Supervised Speech Representation Learning: A Reviewcited by 258
2022LST: Ladder Side-Tuning for Parameter and Memory Efficient Transfer Learningcited by 228
2023Parameter-Efficient Fine-Tuning Methods for Pretrained Language Models: A Critical Review and Assessmentcited by 227
2024Assessing the Brittleness of Safety Alignment via Pruning and Low-Rank Modificationscited by 220
2021Training Neural Networks with Fixed Sparse Maskscited by 212
2022Frozen CLIP Models are Efficient Video Learnerscited by 181
2022Delta Tuning: A Comprehensive Study of Parameter Efficient Methods for Pre-trained Language Modelscited by 179
2021Block Pruning For Faster Transformerscited by 176
2023Scaling Down to Scale Up: A Guide to Parameter-Efficient Fine-Tuningcited by 171
2021UniPELT: A Unified Framework for Parameter-Efficient Language Model Tuningcited by 162
2021Raise a Child in Large Language Model: Towards Effective and Generalizable Fine-tuningcited by 145
2021Composable Sparse Fine-Tuning for Cross-Lingual Transfercited by 144
2022On the Effectiveness of Parameter-Efficient Fine-Tuningcited by 140
2024Mobile Edge Intelligence for Large Language Models: A Contemporary Surveycited by 125
2023Multitask Prompt Tuning Enables Parameter-Efficient Transfer Learningcited by 115
2023Parameter-Efficient Orthogonal Finetuning via Butterfly Factorizationcited by 112
2021BERT Busters: Outlier Dimensions that Disrupt Transformerscited by 110
2024Low-Rank Few-Shot Adaptation of Vision-Language Modelscited by 103
2021The Efficiency Misnomercited by 98
2023Sparse Low-rank Adaptation of Pre-trained Language Modelscited by 96
2023SLoRA: Federated Parameter Efficient Fine-Tuning of Language Modelscited by 92
2023LQ-LoRA: Low-rank Plus Quantized Matrix Decomposition for Efficient Language Model Finetuningcited by 92
2022BLOOM+1: Adding Language Support to BLOOM for Zero-Shot Promptingcited by 90
2022Petals: Collaborative Inference and Fine-tuning of Large Modelscited by 84
2022Head2Toe: Utilizing Intermediate Representations for Better Transfer Learningcited by 83
2023Cascade Speculative Drafting for Even Faster LLM Inferencecited by 83
2023Modular Deep Learningcited by 82
2023Punica: Multi-Tenant LoRA Servingcited by 80
2023Bactrian-X: Multilingual Replicable Instruction-Following Models with Low-Rank Adaptationcited by 79
2022Efficient Methods for Natural Language Processing: A Surveycited by 78
2022Revisiting Parameter-Efficient Tuning: Are We Really There Yet?cited by 76
2022StoryDALL-E: Adapting Pretrained Text-to-Image Transformers for Story Continuationcited by 74
2023Sensitivity-Aware Visual Parameter-Efficient Fine-Tuningcited by 74
2022Vision Transformers are Parameter-Efficient Audio-Visual Learnerscited by 72
2023Delta-LoRA: Fine-Tuning High-Rank Parameters with the Delta of Low-Rank Matricescited by 72
20233DSAM-adapter: Holistic adaptation of SAM from 2D to 3D for promptable tumor segmentationcited by 70
2023E^2VPT: An Effective and Efficient Approach for Visual Prompt Tuningcited by 70
2023Parameter-Efficient Fine-Tuning Design Spacescited by 67

50 most cited of 351 All 351 (JSON)