mlsanityData to 14 Oct 2026

S2A: A Unified Framework for Parameter and Memory Efficient Transfer Learning

arXiv 2503.08154 ↗First posted 11 Mar 2025cs.CVcited by 0

Not on the map. Why?

Abstract (arXiv)

Parameter-efficient transfer learning (PETL) aims to reduce the scales of pretrained models for multiple downstream tasks. However, as the models keep scaling up, the memory footprint of existing PETL methods is not significantly reduced compared to the reduction of learnable parameters. This limitation hinders the practical deployment of PETL methods on memory-constrained devices. To this end, we proposed a new PETL framework, called Structure to Activation (S2A), to reduce the memory footprint of activation during fine-tuning. Specifically, our framework consists of: 1) Activation modules design(i.e., bias, prompt and side modules) in the parametric model structure, which results in a significant reduction of adjustable parameters and activation memory; 2) 4-bit quantization of activations based on their derivatives for non-parametric structures (e.g., nonlinear functions), which maintains accuracy while significantly reducing memory usage. Our S2A method consequently offers a lightweight solution in terms of both parameters and memory footprint. We evaluated S2A with different backbones and performed extensive experiments on various datasets to evaluate the effectiveness. The results show that our methods not only outperform existing PETL techniques, achieving a fourfold reduction in GPU memory footprint on average, but also shows competitive performance in accuracy with fewer tunable parameters. These demonstrate that our method is highly suitable for practical transfer learning on hardware-constrained devices.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 50 of its 72 references that are in the map. First posted on arXiv 11 Mar 2025.

Builds on

50 of 72 references.

References · 50

  1. 2015Deep Residual Learning for Image Recognitioncited by 52,096
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  3. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,305
  4. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  5. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  6. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 5,045
  7. 2020Training data-efficient image transformers & distillation through attentioncited by 4,586
  8. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,236
  9. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,915
  10. 2017MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applicationscited by 3,845
  11. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,795
  12. 2022A ConvNet for the 2020scited by 3,579
38 more
  1. 2016Gaussian Error Linear Units (GELUs)cited by 3,502
  2. 2015Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Codingcited by 2,329
  3. 2021Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processingcited by 2,234
  4. 2013Describing Textures in the Wildcited by 2,125
  5. 2019Searching for MobileNetV3cited by 1,994
  6. 2017Decoupled Weight Decay Regularizationcited by 1,831
  7. 2022Visual Prompt Tuningcited by 1,681
  8. 2017Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inferencecited by 1,593
  9. 2017EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classificationcited by 1,496
  10. 2021BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-modelscited by 1,245
  11. 2021Swin Transformer V2: Scaling Up Capacity and Resolutioncited by 1,228
  12. 2017ShuffleNet: An Extremely Efficient Convolutional Neural Network for Mobile Devicescited by 1,200
  13. 2018ShuffleNet V2: Practical Guidelines for Efficient CNN Architecture Designcited by 1,136
  14. 2016SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and <0.5MB model sizecited by 1,060
  15. 2021Scaling Vision Transformerscited by 999
  16. 2016Training Deep Nets with Sublinear Memory Costcited by 848
  17. 2022AdaptFormer: Adapting Vision Transformers for Scalable Visual Recognitioncited by 719
  18. 2021P-Tuning v2: Prompt Tuning Can Be Comparable to Fine-tuning Universally Across Scales and Taskscited by 689
  19. 2017Learning multiple visual domains with residual adapterscited by 557
  20. 2018PACT: Parameterized Clipping Activation for Quantized Neural Networkscited by 467
  21. 2018HAQ: Hardware-Aware Automated Quantization with Mixed Precisioncited by 443
  22. 2019A Large-scale Study of Representation Learning with the Visual Task Adaptation Benchmarkcited by 425
  23. 2019Understanding Straight-Through Estimator in Training Activation Quantized Neural Netscited by 231
  24. 2022LST: Ladder Side-Tuning for Parameter and Memory Efficient Transfer Learningcited by 230
  25. 2018Efficient parametrization of multi-domain deep neural networkscited by 168
  26. 2019Additive Powers-of-Two Quantization: An Efficient Non-uniform Discretization for Neural Networkscited by 148
  27. 2015Fixed Point Quantization of Deep Convolutional Networkscited by 143
  28. 2022Head2Toe: Utilizing Intermediate Representations for Better Transfer Learningcited by 83
  29. 2022Conv-Adapter: Exploring Parameter Efficient Transfer Learning for ConvNetscited by 62
  30. 2019Low-Memory Neural Network Training: A Technical Reportcited by 57
  31. 2024Dynamic Adapter Meets Prompt Tuning: Parameter-Efficient Transfer Learning for Point Cloud Analysiscited by 53
  32. 2025Parameter-Efficient Fine-Tuning for Foundation Modelscited by 45
  33. 2023UniPT: Universal Parallel Tuning for Transfer Learning with Efficient Parameter and Memorycited by 26
  34. 2024Scaling Sparse Fine-Tuning to Large Language Modelscited by 26
  35. 2023G-Adapter: Towards Structure-Aware Parameter-Efficient Transfer Learning for Graph Transformer Networkscited by 25
  36. 2022Few-Bit Backward: Quantized Gradients of Activation Functions for Memory Footprint Reductioncited by 14
  37. 2023Towards Efficient Verification of Quantized Neural Networkscited by 11
  38. 2020DA3:Dynamic Additive Attention Adaption for Memory-EfficientOn-Device Multi-Domain Learningcited by 6

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026Parameter Efficiency Is Not Memory Efficiency: Rethinking Fine-Tuning for On-Device LLM Adaptation
  2. 2023Parameter and Computation Efficient Transfer Learning for Vision-Language Pre-trained Models
  3. 2024Multiple-Exit Tuning: Towards Inference-Efficient Adaptation for Vision Transformer
  4. 2026S0 Tuning: Zero-Overhead Adaptation of Hybrid Recurrent-Attention Models
  5. 2025Faster Parameter-Efficient Tuning with Token Redundancy Reduction
  6. 2026S2D: Selective Spectral Decay for Quantization-Friendly Conditioning of Neural Activations
  7. 2022PANDA: Prompt Transfer Meets Knowledge Distillation for Efficient Model Adaptation
  8. 2026LOFT: Low-Rank Orthogonal Fine-Tuning via Task-Aware Support Selection