# Revisiting the Parameter Efficiency of Adapters from the Perspective of Precision Redundancy

arXiv: https://arxiv.org/abs/2307.16867 · HTML: https://mlsanity.com/paper/2307.16867/

## Key facts

- Cited by 44 papers in the map (data through 14 Oct 2026).
- Builds on 57 of its 78 references that are in the map.
- First posted on arXiv 31 Jul 2023.

## Builds on (most cited first, 30 of 57)

- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift](https://mlsanity.com/paper/1502.03167/) (2015)
- [Fully Convolutional Networks for Semantic Segmentation](https://mlsanity.com/paper/1411.4038/) (2014)
- [Training data-efficient image transformers & distillation through attention](https://mlsanity.com/paper/2012.12877/) (2020)
- [Parameter-Efficient Transfer Learning for NLP](https://mlsanity.com/paper/1902.00751/) (2019)
- [A ConvNet for the 2020s](https://mlsanity.com/paper/2201.03545/) (2022)
- [Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Coding](https://mlsanity.com/paper/1510.00149/) (2015)
- [Learning to Prompt for Vision-Language Models](https://mlsanity.com/paper/2109.01134/) (2021)
- [Describing Textures in the Wild](https://mlsanity.com/paper/1311.3618/) (2013)
- [Fine-Grained Visual Classification of Aircraft](https://mlsanity.com/paper/1306.5151/) (2013)
- [Visual Prompt Tuning](https://mlsanity.com/paper/2203.12119/) (2022)
- [EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classification](https://mlsanity.com/paper/1709.00029/) (2017)
- [CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoning](https://mlsanity.com/paper/1612.06890/) (2016)
- [Conditional Prompt Learning for Vision-Language Models](https://mlsanity.com/paper/2203.05557/) (2022)
- [Sharpness-Aware Minimization for Efficiently Improving Generalization](https://mlsanity.com/paper/2010.01412/) (2020)
- [On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima](https://mlsanity.com/paper/1609.04836/) (2016)
- [BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-models](https://mlsanity.com/paper/2106.10199/) (2021)
- [Visualizing the Loss Landscape of Neural Nets](https://mlsanity.com/paper/1712.09913/) (2017)
- [Segmenter: Transformer for Semantic Segmentation](https://mlsanity.com/paper/2105.05633/) (2021)
- [XNOR-Net: ImageNet Classification Using Binary Convolutional Neural Networks](https://mlsanity.com/paper/1603.05279/) (2016)
- [A Survey of Quantization Methods for Efficient Neural Network Inference](https://mlsanity.com/paper/2103.13630/) (2021)
- [Remote Sensing Image Scene Classification: Benchmark and State of the Art](https://mlsanity.com/paper/1703.00121/) (2017)
- [AdapterFusion: Non-Destructive Task Composition for Transfer Learning](https://mlsanity.com/paper/2005.00247/) (2020)
- [AdaptFormer: Adapting Vision Transformers for Scalable Visual Recognition](https://mlsanity.com/paper/2205.13535/) (2022)
- [Learned Step Size Quantization](https://mlsanity.com/paper/1902.08153/) (2019)
- [Up or Down? Adaptive Rounding for Post-Training Quantization](https://mlsanity.com/paper/2004.10568/) (2020)
- [Learning multiple visual domains with residual adapters](https://mlsanity.com/paper/1705.08045/) (2017)
- [Compacter: Efficient Low-Rank Hypercomplex Adapter Layers](https://mlsanity.com/paper/2106.04647/) (2021)

## Built on by (most cited first, 30 of 44)

- [Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Survey](https://mlsanity.com/paper/2403.14608/) (2024)
- [Dynamic Adapter Meets Prompt Tuning: Parameter-Efficient Transfer Learning for Point Cloud Analysis](https://mlsanity.com/paper/2403.01439/) (2024)
- [Parameter-Efficient Fine-Tuning in Large Models: A Survey of Methodologies](https://mlsanity.com/paper/2410.19878/) (2024)
- [Parameter-Efficient Fine-Tuning in Spectral Domain for Point Cloud Learning](https://mlsanity.com/paper/2410.08114/) (2024)
- [Visual Fourier Prompt Tuning](https://mlsanity.com/paper/2411.01327/) (2024)
- [Memory-Space Visual Prompting for Efficient Vision-Language Fine-Tuning](https://mlsanity.com/paper/2405.05615/) (2024)
- [Parameter Efficient Fine-tuning of Self-supervised ViTs without Catastrophic Forgetting](https://mlsanity.com/paper/2404.17245/) (2024)
- [M2PT: Multimodal Prompt Tuning for Zero-shot Instruction Learning](https://mlsanity.com/paper/2409.15657/) (2024)
- [Adapter-X: A Novel General Parameter-Efficient Fine-Tuning Framework for Vision](https://mlsanity.com/paper/2406.03051/) (2024)
- [SuperLoRA: Parameter-Efficient Unified Adaptation of Multi-Layer Attention Modules](https://mlsanity.com/paper/2403.11887/) (2024)
- [Low-Rank Interconnected Adaptation across Layers](https://mlsanity.com/paper/2407.09946/) (2024)
- [PMA: Towards Parameter-Efficient Point Cloud Understanding via Point Mamba Adapter](https://mlsanity.com/paper/2505.20941/) (2025)
- [All You Need is One: Capsule Prompt Tuning with a Single Vector](https://mlsanity.com/paper/2510.16670/) (2025)
- [Parameter Efficient Adaptation for Image Restoration with Heterogeneous Mixture-of-Experts](https://mlsanity.com/paper/2312.08881/) (2023)
- [Quantized Prompt for Efficient Generalization of Vision-Language Models](https://mlsanity.com/paper/2407.10704/) (2024)
- [Token Compensator: Altering Inference Cost of Vision Transformer without Re-Tuning](https://mlsanity.com/paper/2408.06798/) (2024)
- [MoST: Efficient Monarch Sparse Tuning for 3D Representation Learning](https://mlsanity.com/paper/2503.18368/) (2025)
- [Mantis: Mamba-native Tuning is Efficient for 3D Point Cloud Foundation Models](https://mlsanity.com/paper/2605.03438/) (2026)
- [Efficient Fine-tuning of Audio Spectrogram Transformers via Soft Mixture of Adapters](https://mlsanity.com/paper/2402.00828/) (2024)
- [MEPT: Mixture of Expert Prompt Tuning as a Manifold Mapper](https://mlsanity.com/paper/2509.00996/) (2025)
- [iVPT: Improving Task-relevant Information Sharing in Visual Prompt Tuning by Cross-layer Dynamic Connection](https://mlsanity.com/paper/2404.05207/) (2024)
- [Parameter-Efficient and Memory-Efficient Tuning for Vision Transformer: A Disentangled Approach](https://mlsanity.com/paper/2407.06964/) (2024)
- [Faster Parameter-Efficient Tuning with Token Redundancy Reduction](https://mlsanity.com/paper/2503.20282/) (2025)
- [LSPT: Long-term Spatial Prompt Tuning for Visual Representation Learning](https://mlsanity.com/paper/2402.17406/) (2024)
- [AdaIR: Exploiting Underlying Similarities of Image Restoration Tasks with Adapters](https://mlsanity.com/paper/2404.11475/) (2024)
- [UniSTD: Towards Unified Spatio-Temporal Learning across Diverse Disciplines](https://mlsanity.com/paper/2503.20748/) (2025)
- [Parameter-Efficient Fine-Tuning for Spiking Point Cloud Models](https://mlsanity.com/paper/2607.29048/) (2026)
- [Rethinking Low-Rank Adaptation in Vision: Exploring Head-Level Responsiveness across Diverse Tasks](https://mlsanity.com/paper/2404.08894/) (2024)
- [Multiple-Exit Tuning: Towards Inference-Efficient Adaptation for Vision Transformer](https://mlsanity.com/paper/2409.13999/) (2024)
- [MNAFT: modality neuron-aware fine-tuning of multimodal large language models for image translation](https://mlsanity.com/paper/2604.16943/) (2026)

## Abstract (arXiv)

Current state-of-the-art results in computer vision depend in part on fine-tuning large pre-trained vision models. However, with the exponential growth of model sizes, the conventional full fine-tuning, which needs to store a individual network copy for each tasks, leads to increasingly huge storage and transmission overhead. Adapter-based Parameter-Efficient Tuning (PET) methods address this challenge by tuning lightweight adapters inserted into the frozen pre-trained models. In this paper, we investigate how to make adapters even more efficient, reaching a new minimum size required to store a task-specific fine-tuned network. Inspired by the observation that the parameters of adapters converge at flat local minima, we find that adapters are resistant to noise in parameter space, which means they are also resistant to low numerical precision. To train low-precision adapters, we propose a computational-efficient quantization method which minimizes the quantization error. Through extensive experiments, we find that low-precision adapters exhibit minimal performance degradation, and even 1-bit precision is sufficient for adapters. The experimental results demonstrate that 1-bit adapters outperform all other PET methods on both the VTAB-1K benchmark and few-shot FGVC tasks, while requiring the smallest storage size. Our findings show, for the first time, the significant potential of quantization techniques in PET, providing a general solution to enhance the parameter efficiency of adapter-based PET methods. Code: https://github.com/JieShibo/PETL-ViT