mlsanityData to 7 Oct 2026

Random Masking Finds Winning Tickets for Parameter Efficient Fine-tuning

arXiv 2405.02596 ↗First posted 4 May 2024cs.LG cs.AI cs.CLcited by 16

Not on the map. Why?

Abstract (arXiv)

Fine-tuning large language models (LLM) can be costly. Parameter-efficient fine-tuning (PEFT) addresses the problems by training a fraction of the parameters, whose success reveals the expressiveness and flexibility of pretrained models. This paper studies the limit of PEFT, by further simplifying its design and reducing the number of trainable parameters beyond standard setups. To this end, we use Random Masking to fine-tune the pretrained model. Despite its simplicity, we show that Random Masking is surprisingly effective: with a larger-than-expected learning rate, Random Masking can match the performance of standard PEFT algorithms such as LoRA on various tasks, using fewer trainable parameters. We provide both empirical and theoretical explorations into the success of Random Masking. We show that masking induces a flatter loss landscape and more distant solutions, which allows for and necessitates large learning rates.

Key facts. Cited by 16 papers in the map (data through 7 Oct 2026). Builds on 64 of its 85 references that are in the map. First posted on arXiv 4 May 2024.

Builds on

64 of 85 references.

References · 64

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2020Language Models are Few-Shot Learnerscited by 32,199
  3. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  4. 2023Llama 2: Open Foundation and Fine-Tuned Chat Modelscited by 12,865
  5. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,304
  6. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  7. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  8. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  9. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  10. 2022OPT: Open Pre-trained Transformer Language Modelscited by 3,464
  11. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  12. 2015Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Codingcited by 2,291
38 more
  1. 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,250
  2. 2019BoolQ: Exploring the Surprising Difficulty of Natural Yes/No Questionscited by 2,140
  3. 2019SuperGLUE: A Stickier Benchmark for General-Purpose Language Understanding Systemscited by 1,718
  4. 2022Visual Prompt Tuningcited by 1,674
  5. 2021BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-modelscited by 1,234
  6. 2022Editing Models with Task Arithmeticcited by 1,076
  7. 2022Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learningcited by 886
  8. 2019DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphscited by 883
  9. 2021Towards a Unified View of Parameter-Efficient Transfer Learningcited by 863
  10. 2017Remote Sensing Image Scene Classification: Benchmark and State of the Artcited by 791
  11. 2020AdapterFusion: Non-Destructive Task Composition for Transfer Learningcited by 780
  12. 2023LLaMA-Adapter: Efficient Fine-tuning of Language Models with Zero-init Attentioncited by 776
  13. 2021GPT Understands, Toocited by 715
  14. 2017Learning Efficient Convolutional Networks through Network Slimmingcited by 673
  15. 2020Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuningcited by 658
  16. 2023LLaMA-Adapter V2: Parameter-Efficient Visual Instruction Modelcited by 602
  17. 2014Convex Optimization: Algorithms and Complexitycited by 569
  18. 2021Compacter: Efficient Low-Rank Hypercomplex Adapter Layerscited by 458
  19. 2018WiC: the Word-in-Context Dataset for Evaluating Context-Sensitive Meaning Representationscited by 370
  20. 2020Parameter-Efficient Transfer Learning with Diff Pruningcited by 351
  21. 2023Fine-Tuning Language Models with Just Forward Passescited by 337
  22. 2021VL-Adapter: Parameter-Efficient Transfer Learning for Vision-and-Language Taskscited by 319
  23. 2017Variational Dropout Sparsifies Deep Neural Networkscited by 265
  24. 2020AdapterDrop: On the Efficiency of Adapters in Transformerscited by 223
  25. 2023Task Arithmetic in the Tangent Space: Improved Editing of Pre-Trained Modelscited by 218
  26. 2021Training Neural Networks with Fixed Sparse Maskscited by 212
  27. 2018ReCoRD: Bridging the Gap between Human and Machine Commonsense Reading Comprehensioncited by 198
  28. 2022Delta Tuning: A Comprehensive Study of Parameter Efficient Methods for Pre-trained Language Modelscited by 179
  29. 2023Scaling Down to Scale Up: A Guide to Parameter-Efficient Fine-Tuningcited by 171
  30. 2021UniPELT: A Unified Framework for Parameter-Efficient Language Model Tuningcited by 162
  31. 2021On the Effectiveness of Adapter-based Tuning for Pretrained Language Model Adaptationcited by 147
  32. 2021Raise a Child in Large Language Model: Towards Effective and Generalizable Fine-tuningcited by 145
  33. 2020Sparse GPU Kernels for Deep Learningcited by 138
  34. 2023Full Parameter Fine-tuning for Large Language Models with Limited Resourcescited by 134
  35. 2023LoRA-FA: Efficient and Effective Low Rank Representation Fine-tuningcited by 132
  36. 2022Convolutional Bypasses Are Better Vision Transformer Adapterscited by 120
  37. 2022A Kernel-Based View of Language Model Fine-Tuningcited by 118
  38. 2023Sparse Low-rank Adaptation of Pre-trained Language Modelscited by 96

50 most cited of 64 All 64 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 6; 2025: 6; 2026: 4.

16 papers.

Sort:

By area:

  1. 2024Lottery Ticket Adaptation: Mitigating Destructive Interference in LLMscited by 31
  2. 2024Understanding Layer Significance in LLM Alignmentcited by 19
  3. 2024MoFO: Momentum-Filtered Optimizer for Mitigating Forgetting in LLM Fine-Tuningcited by 13
  4. 2024FREE-Merging: Fourier Transform for Efficient Model Mergingcited by 12
  5. 2024MoreauPruner: Robust Pruning of Large Language Models against Weight Perturbationscited by 5
  6. 2025DREAM: Scalable Red Teaming for Text-to-Image Generative Systems via Distribution Modelingcited by 5
  7. 2025Efficient Orthogonal Fine-Tuning with Principal Subspace Adaptationcited by 4
  8. 2025DiaBlo: Diagonal Blocks Are Sufficient For Finetuningcited by 2
  9. 2026The Multiple Ticket Hypothesis: Random Sparse Subnetworks Suffice for RLVRcited by 2
  10. 2026Model-Dowser: Data-Free Importance Probing to Mitigate Catastrophic Forgetting in Multimodal Large Language Modelscited by 2
  11. 2026FedMosaic: Federated Retrieval-Augmented Generation via Parametric Adapterscited by 2
  12. 2024Dynamic Subset Tuning: Expanding the Operational Range of Parameter-Efficient Training for Large Language Modelscited by 1
4 more
  1. 2025GEM: A Scale-Aware and Distribution-Sensitive Sparse Fine-Tuning Framework for Effective Downstream Adaptationcited by 0
  2. 2025Revisiting Mixout: An Overlooked Path to Robust Finetuningcited by 0
  3. 2025The Quest for Winning Tickets in Low-Rank Adapterscited by 0
  4. 2026TerMeZO: Ternary Sparse Zeroth-Order Optimization for Fine-tuning BitNet Models at the Edgecited by 0