mlsanityData to 7 Oct 2026

Less is More: Selective Layer Finetuning with SubTuning

arXiv 2302.06354 ↗First posted 13 Feb 2023cs.LG cs.AIcited by 13

Not on the map. Why?

Abstract (arXiv)

Finetuning a pretrained model has become a standard approach for training neural networks on novel tasks, resulting in fast convergence and improved performance. In this work, we study an alternative finetuning method, where instead of finetuning all the weights of the network, we only train a carefully chosen subset of layers, keeping the rest of the weights frozen at their initial (pretrained) values. We demonstrate that subset finetuning (or SubTuning) often achieves accuracy comparable to full finetuning of the model, and even surpasses the performance of full finetuning when training data is scarce. Therefore, SubTuning allows deploying new tasks at minimal computational cost, while enjoying the benefits of finetuning the entire model. This yields a simple and effective method for multi-task learning, where different tasks do not interfere with one another, and yet share most of the resources at inference time. We demonstrate the efficiency of SubTuning across multiple tasks, using different network architectures and pretraining methods.

Key facts. Cited by 13 papers in the map (data through 7 Oct 2026). Builds on 44 of its 64 references that are in the map. First posted on arXiv 13 Feb 2023.

Builds on

44 of 64 references.

References · 44

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  6. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  7. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
  8. 2016Overcoming catastrophic forgetting in neural networkscited by 4,718
  9. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  10. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  11. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  12. 2019Benchmarking Neural Network Robustness to Common Corruptions and Perturbationscited by 2,857
32 more
  1. 2016iCaRL: Incremental Classifier and Representation Learningcited by 2,313
  2. 2019Do ImageNet Classifiers Generalize to ImageNet?cited by 1,507
  3. 2020WILDS: A Benchmark of in-the-Wild Distribution Shiftscited by 1,366
  4. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,265
  5. 2021BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-modelscited by 1,234
  6. 2016Pruning Filters for Efficient ConvNetscited by 1,129
  7. 2021Towards a Unified View of Parameter-Efficient Transfer Learningcited by 863
  8. 2017GradNorm: Gradient Normalization for Adaptive Loss Balancing in Deep Multitask Networkscited by 816
  9. 2018Multi-Task Learning as Multi-Objective Optimizationcited by 809
  10. 2017An Overview of Multi-Task Learning in Deep Neural Networkscited by 791
  11. 2020RobustBench: a standardized adversarial robustness benchmarkcited by 659
  12. 2021ImageNet-21K Pretraining for the Massescited by 607
  13. 2017Learning multiple visual domains with residual adapterscited by 553
  14. 2022Image as a Foreign Language: BEiT Pretraining for All Vision and Vision-Language Taskscited by 510
  15. 2018Not All Samples Are Created Equal: Deep Learning with Importance Samplingcited by 323
  16. 2022LST: Ladder Side-Tuning for Parameter and Memory Efficient Transfer Learningcited by 228
  17. 2022Surgical Fine-Tuning Improves Adaptation to Distribution Shiftscited by 205
  18. 2018Efficient parametrization of multi-domain deep neural networkscited by 168
  19. 2019AdaShare: Learning What To Share For Efficient Deep Multi-Task Learningcited by 159
  20. 2022On the Effectiveness of Parameter-Efficient Fine-Tuningcited by 140
  21. 2022Class-Incremental Learning by Knowledge Distillation with Adaptive Feature Consolidationcited by 113
  22. 2020Masking as an Efficient Alternative to Finetuning for Pretrained Language Modelscited by 92
  23. 2022Head2Toe: Utilizing Intermediate Representations for Better Transfer Learningcited by 83
  24. 2022Parameter-efficient Model Adaptation for Vision Transformerscited by 81
  25. 2022Conv-Adapter: Exploring Parameter Efficient Transfer Learning for ConvNetscited by 62
  26. 2020How fine can fine-tuning be? Learning efficient language modelscited by 41
  27. 2022The Challenges of Continuous Self-Supervised Learningcited by 36
  28. 2020Multi-task learning for natural language processing in the 2020s: where are we going?cited by 27
  29. 2021Searching for Fast Model Families on Datacenter Acceleratorscited by 22
  30. 2019Side-Tuning: A Baseline for Network Adaptation via Additive Side Networkscited by 16
  31. 2018Piggyback: Adapting a Single Network to Multiple Tasks by Learning to Mask Weightscited by 11
  32. 2021Balancing Average and Worst-case Accuracy in Multitask Learningcited by 10

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 4; 2025: 5; 2026: 4.

13 papers.

Sort:

By area:

  1. 2024Gradient Routing: Masking Gradients to Localize Computation in Neural Networkscited by 28
  2. 2025LENSLLM: Unveiling Fine-Tuning Dynamics for LLM Selectioncited by 14
  3. 2025PRO-VPT: Distribution-Adaptive Visual Prompt Tuning via Prompt Relocationcited by 9
  4. 2024Layer-wise Importance Matters: Less Memory for Better Performance in Parameter-efficient Fine-tuning of Large Language Modelscited by 7
  5. 2025AIM-Fair: Advancing Algorithmic Fairness via Selectively Fine-Tuning Biased Models with Contextual Synthetic Datacited by 5
  6. 2024F3OCUS -- Federated Finetuning of Vision-Language Foundation Models with Optimal Client Layer Updating Strategy via Multi-objective Meta-Heuristicscited by 4
  7. 2025FLoE: Fisher-Based Layer Selection for Efficient Sparse Adaptation of Low-Rank Expertscited by 3
  8. 2026Multi-Modal Data-Enhanced Foundation Models for Prediction and Control in Wireless Networks: A Surveycited by 3
  9. 2024Exploring Selective Layer Fine-Tuning in Federated Learningcited by 2
  10. 2025Study of Training Dynamics for Memory-Constrained Fine-Tuningcited by 1
  11. 2026Backdooring Masked Diffusion Language Modelscited by 0
  12. 2026Mixture-of-Control: State-Aware Fine-Tuning for Transformer-based Modelscited by 0
1 more
  1. 2026Open-Vocabulary Gaze Object Prediction: Benchmark and Methodcited by 0