# Parameter-efficient is not sufficient: Exploring Parameter, Memory, and Time Efficient Adapter Tuning for Dense Predictions

arXiv: https://arxiv.org/abs/2306.09729 · HTML: https://mlsanity.com/paper/2306.09729/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- Builds on 44 of its 67 references that are in the map.
- First posted on arXiv 16 Jun 2023.

## Builds on (most cited first, 30 of 44)

- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer](https://mlsanity.com/paper/1910.10683/) (2019)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Feature Pyramid Networks for Object Detection](https://mlsanity.com/paper/1612.03144/) (2016)
- [Parameter-Efficient Transfer Learning for NLP](https://mlsanity.com/paper/1902.00751/) (2019)
- [Pre-train, Prompt, and Predict: A Systematic Survey of Prompting Methods in Natural Language Processing](https://mlsanity.com/paper/2107.13586/) (2021)
- [Visual Prompt Tuning](https://mlsanity.com/paper/2203.12119/) (2022)
- [Unified Perceptual Parsing for Scene Understanding](https://mlsanity.com/paper/1807.10221/) (2018)
- [Energy and Policy Considerations for Deep Learning in NLP](https://mlsanity.com/paper/1906.02243/) (2019)
- [BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-models](https://mlsanity.com/paper/2106.10199/) (2021)
- [Scaling Vision Transformers](https://mlsanity.com/paper/2106.04560/) (2021)
- [Few-Shot Parameter-Efficient Fine-Tuning is Better and Cheaper than In-Context Learning](https://mlsanity.com/paper/2205.05638/) (2022)
- [Training Deep Nets with Sublinear Memory Cost](https://mlsanity.com/paper/1604.06174/) (2016)
- [AdapterFusion: Non-Destructive Task Composition for Transfer Learning](https://mlsanity.com/paper/2005.00247/) (2020)
- [AdaptFormer: Adapting Vision Transformers for Scalable Visual Recognition](https://mlsanity.com/paper/2205.13535/) (2022)
- [EVA: Exploring the Limits of Masked Visual Representation Learning at Scale](https://mlsanity.com/paper/2211.07636/) (2022)
- [Intrinsic Dimensionality Explains the Effectiveness of Language Model Fine-Tuning](https://mlsanity.com/paper/2012.13255/) (2020)
- [Vision Transformer Adapter for Dense Predictions](https://mlsanity.com/paper/2205.08534/) (2022)
- [Using DeepSpeed and Megatron to Train Megatron-Turing NLG 530B, A Large-Scale Generative Language Model](https://mlsanity.com/paper/2201.11990/) (2022)
- [Image as a Foreign Language: BEiT Pretraining for All Vision and Vision-Language Tasks](https://mlsanity.com/paper/2208.10442/) (2022)
- [InternImage: Exploring Large-Scale Vision Foundation Models with Deformable Convolutions](https://mlsanity.com/paper/2211.05778/) (2022)
- [Compacter: Efficient Low-Rank Hypercomplex Adapter Layers](https://mlsanity.com/paper/2106.04647/) (2021)
- [Fine-Tuning Pretrained Language Models: Weight Initializations, Data Orders, and Early Stopping](https://mlsanity.com/paper/2002.06305/) (2020)
- [Prompting Visual-Language Models for Efficient Video Understanding](https://mlsanity.com/paper/2112.04478/) (2021)
- [Scaling & Shifting Your Features: A New Baseline for Efficient Model Tuning](https://mlsanity.com/paper/2210.08823/) (2022)
- [DETRs with Collaborative Hybrid Assignments Training](https://mlsanity.com/paper/2211.12860/) (2022)
- [LST: Ladder Side-Tuning for Parameter and Memory Efficient Transfer Learning](https://mlsanity.com/paper/2206.06522/) (2022)
- [AdapterDrop: On the Efficiency of Adapters in Transformers](https://mlsanity.com/paper/2010.11918/) (2020)

## Built on by (most cited first, 24 of 24)

- [End-to-End Temporal Action Detection with 1B Parameters Across 1000 Frames](https://mlsanity.com/paper/2311.17241/) (2023)
- [Parameter-Efficient Fine-Tuning for Foundation Models](https://mlsanity.com/paper/2501.13787/) (2025)
- [R2-Tuning: Efficient Image-to-Video Transfer Learning for Video Temporal Grounding](https://mlsanity.com/paper/2404.00801/) (2024)
- [5%>100%: Breaking Performance Shackles of Full Fine-Tuning on Visual Recognition Tasks](https://mlsanity.com/paper/2408.08345/) (2024)
- [UNIC: Universal Classification Models via Multi-teacher Distillation](https://mlsanity.com/paper/2408.05088/) (2024)
- [Parameter-Efficient Fine-Tuning for Pre-Trained Vision Models: A Survey and Benchmark](https://mlsanity.com/paper/2402.02242/) (2024)
- [Adapter is All You Need for Tuning Visual Tasks](https://mlsanity.com/paper/2311.15010/) (2023)
- [Learning to Adapt Frozen CLIP for Few-Shot Test-Time Domain Adaptation](https://mlsanity.com/paper/2506.17307/) (2025)
- [M2IST: Multi-Modal Interactive Side-Tuning for Efficient Referring Expression Comprehension](https://mlsanity.com/paper/2407.01131/) (2024)
- [LoSA: Long-Short-range Adapter for Scaling End-to-End Temporal Action Localization](https://mlsanity.com/paper/2404.01282/) (2024)
- [ALPS: An Auto-Labeling and Pre-training Scheme for Remote Sensing Segmentation With Segment Anything Model](https://mlsanity.com/paper/2406.10855/) (2024)
- [Multiple-Exit Tuning: Towards Inference-Efficient Adaptation for Vision Transformer](https://mlsanity.com/paper/2409.13999/) (2024)
- [Token Adaptation via Side Graph Convolution for Efficient Fine-tuning of 3D Point Cloud Transformers](https://mlsanity.com/paper/2502.14142/) (2025)
- [Sparse-Dense Side-Tuner for efficient Video Temporal Grounding](https://mlsanity.com/paper/2507.07744/) (2025)
- [Resource-Efficient Personal Large Language Models Fine-Tuning with Collaborative Edge Computing](https://mlsanity.com/paper/2408.10746/) (2024)
- [Pre-training Everywhere: Parameter-Efficient Fine-Tuning for Medical Image Analysis via Target Parameter Pre-training](https://mlsanity.com/paper/2408.15011/) (2024)
- [Memory-Efficient Transfer Learning with Fading Side Networks via Masked Dual Path Distillation](https://mlsanity.com/paper/2604.09088/) (2026)
- [LSP-ST: Ladder Shape-Biased Side-Tuning for Robust Infrared Small Target Detection](https://mlsanity.com/paper/2504.14481/) (2025)
- [EMLoC: Emulator-based Memory-efficient Fine-tuning with LoRA Correction](https://mlsanity.com/paper/2506.12015/) (2025)
- [MSLoRA: Multi-Scale Low-Rank Adaptation via Attention Reweighting](https://mlsanity.com/paper/2511.12400/) (2025)
- [VFM-ISRefiner: Towards Better Adapting Vision Foundation Models for Interactive Segmentation of Remote Sensing Images](https://mlsanity.com/paper/2512.00718/) (2025)
- [Collaborative Low-Rank Adaptation for Pre-Trained Vision Transformers](https://mlsanity.com/paper/2512.24603/) (2025)
- [1%>100%: High-Efficiency Visual Adapter with Complex Linear Projection Optimization](https://mlsanity.com/paper/2602.10513/) (2026)
- [YOLO-PEFT: Parameter-Efficient Fine-Tuning on YOLO Family](https://mlsanity.com/paper/2608.07051/) (2026)

## Abstract (arXiv)

Pre-training & fine-tuning is a prevalent paradigm in computer vision (CV). Recently, parameter-efficient transfer learning (PETL) methods have shown promising performance in adapting to downstream tasks with only a few trainable parameters. Despite their success, the existing PETL methods in CV can be computationally expensive and require large amounts of memory and time cost during training, which limits low-resource users from conducting research and applications on large models. In this work, we propose Parameter, Memory, and Time Efficient Visual Adapter (E3VA) tuning to address this issue. We provide a gradient backpropagation highway for low-rank adapters which eliminates the need for expensive backpropagation through the frozen pre-trained model, resulting in substantial savings of training memory and training time. Furthermore, we optimise the E3VA structure for CV tasks to promote model performance. Extensive experiments on COCO, ADE20K, and Pascal VOC benchmarks show that E3VA can save up to 62.2% training memory and 26.2% training time on average, while achieving comparable performance to full fine-tuning and better performance than most PETL methods. Note that we can even train the Swin-Large-based Cascade Mask RCNN on GTX 1080Ti GPUs with less than 1.5% trainable parameters.