mlsanityData to 14 Oct 2026

Masking as an Efficient Alternative to Finetuning for Pretrained Language Models

arXiv 2004.12406 ↗First posted 26 Apr 2020cs.CLcited by 93

Not on the map. Why?

Abstract (arXiv)

We present an efficient method of utilizing pretrained language models, where we learn selective binary masks for pretrained weights in lieu of modifying them through finetuning. Extensive evaluations of masking BERT and RoBERTa on a series of NLP tasks show that our masking scheme yields performance comparable to finetuning, yet has a much smaller memory footprint when several tasks need to be inferred simultaneously. Through intrinsic evaluations, we show that representations computed by masked language models encode information necessary for solving downstream tasks. Analyzing the loss landscape, we show that masking and finetuning produce models that reside in minima that can be connected by a line segment with nearly constant test accuracy. This confirms that masking can be utilized as an efficient alternative to finetuning.

Key facts. Cited by 93 papers in the map (data through 14 Oct 2026). First posted on arXiv 26 Apr 2020.

Builds on

No reference list yet.

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 15; 2022: 18; 2023: 21; 2024: 23; 2025: 9; 2026: 7.

93 papers.

Sort:

By area:

  1. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,915
  2. 2021BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-modelscited by 1,245
  3. 2021Recent Advances in Natural Language Processing via Large Pre-Trained Language Models: A Surveycited by 463
  4. 2024PiSSA: Principal Singular Values and Singular Vectors Adaptation of Large Language Modelscited by 269
  5. 2024Assessing the Brittleness of Safety Alignment via Pruning and Low-Rank Modificationscited by 231
  6. 2023Parameter-Efficient Fine-Tuning Methods for Pretrained Language Models: A Critical Review and Assessmentcited by 230
  7. 2021Training Neural Networks with Fixed Sparse Maskscited by 214
  8. 2022Delta Tuning: A Comprehensive Study of Parameter Efficient Methods for Pre-trained Language Modelscited by 180
  9. 2021Learned Token Pruning for Transformerscited by 139
  10. 2022Toward Transparent AI: A Survey on Interpreting the Inner Structures of Deep Neural Networkscited by 129
  11. 2024Let Your Graph Do the Talking: Encoding Structured Data for LLMscited by 118
  12. 2023Glot500: Scaling Multilingual Corpora and Language Models to 500 Languagescited by 116
38 more
  1. 2023Sparse Low-rank Adaptation of Pre-trained Language Modelscited by 96
  2. 2023Modular Deep Learningcited by 83
  3. 2023Punica: Multi-Tenant LoRA Servingcited by 83
  4. 2023Sensitivity-Aware Visual Parameter-Efficient Fine-Tuningcited by 74
  5. 2023Parameter-Efficient Fine-Tuning Design Spacescited by 67
  6. 2021Low-Complexity Probing via Finding Subnetworkscited by 64
  7. 2024BLoB: Bayesian Low-Rank Adaptation by Backpropagation for Large Language Modelscited by 50
  8. 2021Discrete and Soft Prompting for Multilingual Modelscited by 49
  9. 2023AutoPEFT: Automatic Configuration Search for Parameter-Efficient Fine-Tuningcited by 49
  10. 2024Parameter-Efficient Fine-Tuning in Large Models: A Survey of Methodologiescited by 47
  11. 2021Time Masking for Temporal Language Modelscited by 46
  12. 2025Parameter-Efficient Fine-Tuning for Foundation Modelscited by 45
  13. 2023Gradient-based Parameter Selection for Efficient Fine-Tuningcited by 40
  14. 2024Advancing Parameter Efficiency in Fine-tuning via Representation Editingcited by 37
  15. 2024ALoRA: Allocating Low-Rank Adaptation for Fine-tuning Large Language Modelscited by 36
  16. 2022Modular and Parameter-Efficient Multimodal Fusion with Promptingcited by 35
  17. 2021Enabling Lightweight Fine-tuning for Pre-trained Language Model Compression based on Matrix Product Operatorscited by 29
  18. 2024Goldfish: Monolingual Language Models for 350 Languagescited by 29
  19. 2022Parameter-Efficient Sparsity for Large Language Models Fine-Tuningcited by 27
  20. 2024On Behalf of the Stakeholders: Trends in NLP Model Interpretability in the Era of LLMscited by 25
  21. 2024Cross-Problem Learning for Solving Vehicle Routing Problemscited by 24
  22. 2024Understanding Layer Significance in LLM Alignmentcited by 19
  23. 2022Modular and On-demand Bias Mitigation with Attribute-Removal Subnetworkscited by 18
  24. 2022Sparse Structure Search for Parameter-Efficient Tuningcited by 17
  25. 2023ConPET: Continual Parameter-Efficient Tuning for Large Language Modelscited by 17
  26. 2024Random Masking Finds Winning Tickets for Parameter Efficient Fine-tuningcited by 17
  27. 2024Sparse High Rank Adapterscited by 17
  28. 2022Debiasing Masks: A New Framework for Shortcut Mitigation in NLUcited by 16
  29. 2023Discovering Knowledge-Critical Subnetworks in Pretrained Language Modelscited by 16
  30. 2021Disentangling Representations of Text by Masking Transformerscited by 15
  31. 2022Meta-Learning the Difference: Preparing Large Language Models for Efficient Adaptationcited by 14
  32. 2023Less is More: Selective Layer Finetuning with SubTuningcited by 14
  33. 2023AUTO: Adaptive Outlier Optimization for Test-Time OOD Detectioncited by 13
  34. 2023MacLaSa: Multi-Aspect Controllable Text Generation via Efficient Sampling from Compact Latent Spacecited by 12
  35. 2023mLoRA: Fine-Tuning LoRA Adapters via Highly-Efficient Pipeline Parallelism in Multiple GPUscited by 12
  36. 2025Parameter-Efficient Continual Fine-Tuning: A Surveycited by 11
  37. 2022Learning to Win Lottery Tickets in BERT Transfer via Task-agnostic Mask Trainingcited by 10
  38. 2024Expanding Sparse Tuning for Low Memory Usagecited by 10

50 most cited of 93 All 93 (JSON)