mlsanityData to 7 Oct 2026

Training Neural Networks with Fixed Sparse Masks

arXiv 2111.09839 ↗First posted 18 Nov 2021cs.LGcited by 212

Not on the map. Why?

Abstract (arXiv)

During typical gradient-based training of deep neural networks, all of the model's parameters are updated at each iteration. Recent work has shown that it is possible to update only a small subset of the model's parameters during training, which can alleviate storage and communication requirements. In this paper, we show that it is possible to induce a fixed sparse mask on the model's parameters that selects a subset to update over many iterations. Our method constructs the mask out of the k parameters with the largest Fisher information as a simple approximation as to which parameters are most important for the task at hand. In experiments on parameter-efficient transfer learning and distributed training, we show that our approach matches or exceeds the performance of other methods for training with sparse updates while being more efficient in terms of memory usage and communication costs. We release our code publicly to promote further applications of our approach.

Key facts. Cited by 212 papers in the map (data through 7 Oct 2026). Builds on 37 of its 49 references that are in the map. First posted on arXiv 18 Nov 2021.

Builds on

37 of 49 references.

References · 37

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2020Language Models are Few-Shot Learnerscited by 32,199
  4. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  5. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  6. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,592
  7. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  8. 2016Overcoming catastrophic forgetting in neural networkscited by 4,718
  9. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  10. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  11. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  12. 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,250
25 more
  1. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,201
  2. 2016Federated Learning: Strategies for Improving Communication Efficiencycited by 1,461
  3. 2020Big Self-Supervised Models are Strong Semi-Supervised Learnerscited by 1,375
  4. 2021BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-modelscited by 1,234
  5. 2017Learning Sparse Neural Networks through L0 Regularizationcited by 571
  6. 2018Do Better ImageNet Models Transfer Better?cited by 560
  7. 2016Federated Optimization: Distributed Machine Learning for On-Device Intelligencecited by 558
  8. 2017Learning multiple visual domains with residual adapterscited by 553
  9. 2015Train faster, generalize better: Stability of stochastic gradient descentcited by 547
  10. 2021Compacter: Efficient Low-Rank Hypercomplex Adapter Layerscited by 458
  11. 2020AdapterHub: A Framework for Adapting Transformerscited by 441
  12. 2014New insights and perspectives on the natural gradient methodcited by 435
  13. 2020Parameter-Efficient Transfer Learning with Diff Pruningcited by 351
  14. 2018Learning and Generalization in Overparameterized Neural Networks, Going Beyond Two Layerscited by 340
  15. 2021Parameter-efficient Multi-task Fine-tuning for Transformers via Shared Hypernetworkscited by 278
  16. 2017Sparse Communication for Distributed Gradient Descentcited by 243
  17. 2019Simple, Scalable Adaptation for Neural Machine Translationcited by 239
  18. 2019Limitations of the Empirical Fisher Approximation for Natural Gradient Descentcited by 213
  19. 2019To Tune or Not to Tune? Adapting Pretrained Representations to Diverse Taskscited by 167
  20. 2013Revisiting Natural Gradient for Deep Networkscited by 157
  21. 2021Group Fisher Pruning for Practical Network Compressioncited by 130
  22. 2018Faster gaze prediction with dense networks and Fisher pruningcited by 92
  23. 2020Masking as an Efficient Alternative to Finetuning for Pretrained Language Modelscited by 92
  24. 2019Where is the Information in a Deep Neural Network?cited by 40
  25. 2018A Closer Look at Structured Pruning for Neural Network Compressioncited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 1; 2022: 23; 2023: 44; 2024: 68; 2025: 53; 2026: 23.

212 papers.

Sort:

By area:

  1. 2023QLoRA: Efficient Finetuning of Quantized LLMscited by 3,355
  2. 2022Editing Models with Task Arithmeticcited by 1,076
  3. 2022Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learningcited by 886
  4. 2024Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Surveycited by 641
  5. 2022Git Re-Basin: Merging Models modulo Permutation Symmetriescited by 460
  6. 2021VL-Adapter: Parameter-Efficient Transfer Learning for Vision-and-Language Taskscited by 319
  7. 2024PiSSA: Principal Singular Values and Singular Vectors Adaptation of Large Language Modelscited by 261
  8. 2023LongLoRA: Efficient Fine-tuning of Long-Context Large Language Modelscited by 243
  9. 2022LST: Ladder Side-Tuning for Parameter and Memory Efficient Transfer Learningcited by 228
  10. 2023Parameter-Efficient Fine-Tuning Methods for Pretrained Language Models: A Critical Review and Assessmentcited by 227
  11. 2022Patching open-vocabulary models by interpolating weightscited by 187
  12. 2023Scaling Down to Scale Up: A Guide to Parameter-Efficient Fine-Tuningcited by 171
38 more
  1. 2023AIM: Adapting Image Models for Efficient Video Action Recognitioncited by 145
  2. 2022On the Effectiveness of Parameter-Efficient Fine-Tuningcited by 140
  3. 2024Safe LoRA: the Silver Lining of Reducing Safety Risks when Fine-tuning Large Language Modelscited by 122
  4. 2023Multitask Prompt Tuning Enables Parameter-Efficient Transfer Learningcited by 115
  5. 2023Parameter-Efficient Orthogonal Finetuning via Butterfly Factorizationcited by 112
  6. 2023Task-Specific Skill Localization in Fine-tuned Language Modelscited by 104
  7. 2023SimDA: Simple Diffusion Adapter for Efficient Video Generationcited by 101
  8. 2023LQ-LoRA: Low-rank Plus Quantized Matrix Decomposition for Efficient Language Model Finetuningcited by 92
  9. 2022BLOOM+1: Adding Language Support to BLOOM for Zero-Shot Promptingcited by 90
  10. 2022Petals: Collaborative Inference and Fine-tuning of Large Modelscited by 84
  11. 2023Modular Deep Learningcited by 82
  12. 2022Efficient Methods for Natural Language Processing: A Surveycited by 78
  13. 2022Vision Transformers are Parameter-Efficient Audio-Visual Learnerscited by 72
  14. 2022Make Sharpness-Aware Minimization Stronger: A Sparsified Perturbation Approachcited by 70
  15. 2023Parameter-Efficient Fine-Tuning Design Spacescited by 67
  16. 2022Learning to Compose Soft Prompts for Compositional Zero-Shot Learningcited by 66
  17. 2023Parameter-Efficient Fine-Tuning without Introducing New Latencycited by 59
  18. 2022Visual Query Tuning: Towards Effective Usage of Intermediate Representations for Parameter and Memory Efficient Transfer Learningcited by 57
  19. 2023AutoPEFT: Automatic Configuration Search for Parameter-Efficient Fine-Tuningcited by 49
  20. 2024Automated Federated Pipeline for Parameter-Efficient Fine-Tuning of Large Language Modelscited by 49
  21. 2024RoSA: Accurate Parameter-Efficient Fine-Tuning via Robust Adaptationcited by 48
  22. 2023IncreLoRA: Incremental Parameter Allocation Method for Parameter-Efficient Fine-tuningcited by 45
  23. 2024SLTrain: a sparse plus low-rank approach for parameter and memory efficient pretrainingcited by 44
  24. 2024Parameter-Efficient Fine-Tuning in Large Models: A Survey of Methodologiescited by 44
  25. 2025Parameter-Efficient Fine-Tuning for Foundation Modelscited by 43
  26. 2023Distributed Inference and Fine-tuning of Large Language Models Over The Internetcited by 42
  27. 2023Incorporating Structured Representations into Pretrained Vision & Language Models Using Scene Graphscited by 39
  28. 2022Compute-Efficient Deep Learning: Algorithmic Trends and Opportunitiescited by 38
  29. 2023DePT: Decomposed Prompt Tuning for Parameter-Efficient Fine-tuningcited by 34
  30. 2022Multi-Head Adapter Routing for Cross-Task Generalizationcited by 33
  31. 2023Batched Low-Rank Adaptation of Foundation Modelscited by 32
  32. 2023Benchmarking Robustness of Adaptation Methods on Pre-trained Vision-Language Modelscited by 30
  33. 2023Towards Trustworthy and Aligned Machine Learning: A Data-centric Survey with Causality Perspectivescited by 30
  34. 2023Elastic Weight Removal for Faithful and Abstractive Dialogue Generationcited by 29
  35. 2024APT: Adaptive Pruning and Tuning Pretrained Language Models for Efficient Training and Inferencecited by 29
  36. 2022Discovering Language-neutral Sub-networks in Multilingual Language Modelscited by 28
  37. 2024Gradient Routing: Masking Gradients to Localize Computation in Neural Networkscited by 28
  38. 2024Zeroth-Order Fine-Tuning of LLMs with Extreme Sparsitycited by 27

50 most cited of 212 All 212 (JSON)