mlsanityData to 7 Oct 2026

Not All Adapters Matter: Selective Adapter Freezing for Memory-Efficient Fine-Tuning of Language Models

arXiv 2412.03587 ↗First posted 26 Nov 2024cs.CL cs.AI cs.LGcited by 5

Not on the map. Why?

Abstract (arXiv)

Transformer-based large-scale pre-trained models achieve great success. Fine-tuning is the standard practice for leveraging these models in downstream tasks. Among the fine-tuning methods, adapter-tuning provides a parameter-efficient fine-tuning by introducing lightweight trainable modules while keeping most pre-trained parameters frozen. However, existing adapter-tuning methods still impose substantial resource usage. Through our investigation, we show that each adapter unequally contributes to both task performance and resource usage. Motivated by this insight, we propose Selective Adapter FrEezing (SAFE), which gradually freezes less important adapters early to reduce unnecessary resource usage while maintaining performance. In our experiments, SAFE reduces memory usage, computation amount, and training time by 42.85%, 34.59%, and 11.82%, respectively, while achieving comparable or better task performance compared to the baseline. We also demonstrate that SAFE induces regularization effect, thereby smoothing the loss landscape, which enables the model to generalize better by avoiding sharp minima.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Builds on 42 of its 55 references that are in the map. First posted on arXiv 26 Nov 2024.

Builds on

42 of 55 references.

References · 42

  1. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  2. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  4. 2023LLaMA: Open and Efficient Foundation Language Modelscited by 13,958
  5. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  6. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  7. 2017Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithmscited by 4,283
  8. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  9. 2016SQuAD: 100,000+ Questions for Machine Comprehension of Textcited by 4,115
  10. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  11. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  12. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,801
30 more
  1. 2015Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Codingcited by 2,291
  2. 2019HuggingFace's Transformers: State-of-the-art Natural Language Processingcited by 2,201
  3. 2019Similarity of Neural Network Representations Revisitedcited by 1,695
  4. 2021BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-modelscited by 1,234
  5. 2023SparseGPT: Massive Language Models Can Be Accurately Pruned in One-Shotcited by 1,090
  6. 2022Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learningcited by 886
  7. 2016Training Deep Nets with Sublinear Memory Costcited by 825
  8. 2020AdapterFusion: Non-Destructive Task Composition for Transfer Learningcited by 780
  9. 2021Do Vision Transformers See Like Convolutional Neural Networks?cited by 671
  10. 2021Compacter: Efficient Low-Rank Hypercomplex Adapter Layerscited by 458
  11. 2020AdapterHub: A Framework for Adapting Transformerscited by 441
  12. 2021SWAD: Domain Generalization by Seeking Flat Minimacited by 431
  13. 2020What is being transferred in transfer learning?cited by 365
  14. 2022How Do Vision Transformers Work?cited by 305
  15. 2019An Investigation into Neural Net Optimization via Hessian Eigenvalue Densitycited by 289
  16. 2014The Loss Surfaces of Multilayer Networkscited by 278
  17. 2021Pruning and Quantization for Deep Neural Network Acceleration: A Surveycited by 259
  18. 2017The E2E Dataset: New Challenges For End-to-End Generationcited by 252
  19. 2020Revisiting Few-sample BERT Fine-tuningcited by 237
  20. 2020AdapterDrop: On the Efficiency of Adapters in Transformerscited by 223
  21. 2022On the Effectiveness of Parameter-Efficient Fine-Tuningcited by 140
  22. 2023LaMini-LM: A Diverse Herd of Distilled Models from Large-Scale Instructionscited by 124
  23. 2022SparseAdapter: An Easy Approach for Improving the Parameter-Efficiency of Adapterscited by 83
  24. 2022Parameter-efficient Model Adaptation for Vision Transformerscited by 81
  25. 2018The Description Length of Deep Learning Modelscited by 71
  26. 2023LoRAPrune: Structured Pruning Meets Low-Rank Parameter-Efficient Fine-Tuningcited by 53
  27. 2020Artificial Neural Variability for Deep Learning: On Overfitting, Noise Memorization, and Catastrophic Forgettingcited by 28
  28. 2022Parameter-Efficient Sparsity for Large Language Models Fine-Tuningcited by 27
  29. 2023Make Pre-trained Model Reversible: From Parameter to Memory Efficient Fine-Tuningcited by 26
  30. 2024SmartFRZ: An Efficient Training Framework using Attention-Based Layer Freezingcited by 12

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 2; 2026: 3.

5 papers.

  1. 2026BaRA: Bayesian Adaptive Rank Allocation for Parameter-Efficient Fine-Tuningcited by 1
  2. 2025First Attentions Last: Better Exploiting First Attentions for Efficient Transformer Trainingcited by 0
  3. 2025CorGi: Contribution-Guided Block-Wise Interval Caching for Training-Free Acceleration of Diffusion Transformerscited by 0
  4. 2026CRAMER: Control via Request-Aware Masking for Editing Recommenderscited by 0
  5. 2026Thread-Efficient Decoding for Neural Texture Compressioncited by 0