# Consolidator: Mergeable Adapter with Grouped Connections for Visual Adaptation

arXiv: https://arxiv.org/abs/2305.00603 · HTML: https://mlsanity.com/paper/2305.00603/

## Key facts

- Cited by 18 papers in the map (data through 14 Oct 2026).
- Builds on 45 of its 60 references that are in the map.
- First posted on arXiv 30 Apr 2023.

## Builds on (most cited first, 30 of 45)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Very Deep Convolutional Networks for Large-Scale Image Recognition](https://mlsanity.com/paper/1409.1556/) (2014)
- [LoRA: Low-Rank Adaptation of Large Language Models](https://mlsanity.com/paper/2106.09685/) (2021)
- [Swin Transformer: Hierarchical Vision Transformer using Shifted Windows](https://mlsanity.com/paper/2103.14030/) (2021)
- [Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networks](https://mlsanity.com/paper/1506.01497/) (2015)
- [Momentum Contrast for Unsupervised Visual Representation Learning](https://mlsanity.com/paper/1911.05722/) (2019)
- [Layer Normalization](https://mlsanity.com/paper/1607.06450/) (2016)
- [Feature Pyramid Networks for Object Detection](https://mlsanity.com/paper/1612.03144/) (2016)
- [Training data-efficient image transformers & distillation through attention](https://mlsanity.com/paper/2012.12877/) (2020)
- [Parameter-Efficient Transfer Learning for NLP](https://mlsanity.com/paper/1902.00751/) (2019)
- [Prefix-Tuning: Optimizing Continuous Prompts for Generation](https://mlsanity.com/paper/2101.00190/) (2021)
- [MobileNets: Efficient Convolutional Neural Networks for Mobile Vision Applications](https://mlsanity.com/paper/1704.04861/) (2017)
- [The Power of Scale for Parameter-Efficient Prompt Tuning](https://mlsanity.com/paper/2104.08691/) (2021)
- [A ConvNet for the 2020s](https://mlsanity.com/paper/2201.03545/) (2022)
- [Gaussian Error Linear Units (GELUs)](https://mlsanity.com/paper/1606.08415/) (2016)
- [Describing Textures in the Wild](https://mlsanity.com/paper/1311.3618/) (2013)
- [Visual Prompt Tuning](https://mlsanity.com/paper/2203.12119/) (2022)
- [EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classification](https://mlsanity.com/paper/1709.00029/) (2017)
- [MMDetection: Open MMLab Detection Toolbox and Benchmark](https://mlsanity.com/paper/1906.07155/) (2019)
- [Unified Perceptual Parsing for Scene Understanding](https://mlsanity.com/paper/1807.10221/) (2018)
- [BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-models](https://mlsanity.com/paper/2106.10199/) (2021)
- [ShuffleNet: An Extremely Efficient Convolutional Neural Network for Mobile Devices](https://mlsanity.com/paper/1707.01083/) (2017)
- [ShuffleNet V2: Practical Guidelines for Efficient CNN Architecture Design](https://mlsanity.com/paper/1807.11164/) (2018)
- [Deep Networks with Stochastic Depth](https://mlsanity.com/paper/1603.09382/) (2016)
- [AdapterFusion: Non-Destructive Task Composition for Transfer Learning](https://mlsanity.com/paper/2005.00247/) (2020)
- [AdaptFormer: Adapting Vision Transformers for Scalable Visual Recognition](https://mlsanity.com/paper/2205.13535/) (2022)
- [P-Tuning v2: Prompt Tuning Can Be Comparable to Fine-tuning Universally Across Scales and Tasks](https://mlsanity.com/paper/2110.07602/) (2021)
- [RepVGG: Making VGG-style ConvNets Great Again](https://mlsanity.com/paper/2101.03697/) (2021)

## Built on by (most cited first, 18 of 18)

- [Revisiting the Parameter Efficiency of Adapters from the Perspective of Precision Redundancy](https://mlsanity.com/paper/2307.16867/) (2023)
- [Visual Tuning](https://mlsanity.com/paper/2305.06061/) (2023)
- [APoLLo: Unified Adapter and Prompt Learning for Vision Language Models](https://mlsanity.com/paper/2312.01564/) (2023)
- [PYRA: Parallel Yielding Re-Activation for Training-Inference Efficient Task Adaptation](https://mlsanity.com/paper/2403.09192/) (2024)
- [Adapters Strike Back](https://mlsanity.com/paper/2406.06820/) (2024)
- [Quantum-PEFT: Ultra parameter-efficient fine-tuning](https://mlsanity.com/paper/2503.05431/) (2025)
- [SuperLoRA: Parameter-Efficient Unified Adaptation of Multi-Layer Attention Modules](https://mlsanity.com/paper/2403.11887/) (2024)
- [YOLO-UniOW: Efficient Universal Open-World Object Detection](https://mlsanity.com/paper/2412.20645/) (2024)
- [PRO-VPT: Distribution-Adaptive Visual Prompt Tuning via Prompt Relocation](https://mlsanity.com/paper/2503.06901/) (2025)
- [Advancing Reliable Test-Time Adaptation of Vision-Language Models under Visual Variations](https://mlsanity.com/paper/2507.09500/) (2025)
- [Quantized Prompt for Efficient Generalization of Vision-Language Models](https://mlsanity.com/paper/2407.10704/) (2024)
- [Token Compensator: Altering Inference Cost of Vision Transformer without Re-Tuning](https://mlsanity.com/paper/2408.06798/) (2024)
- [Referring Expression Comprehension for Small Objects](https://mlsanity.com/paper/2510.03701/) (2025)
- [Towards Efficient Vision-Language Tuning: More Information Density, More Generalizability](https://mlsanity.com/paper/2312.10813/) (2023)
- [PETAH: Parameter Efficient Task Adaptation for Hybrid Transformers in a resource-limited Context](https://mlsanity.com/paper/2410.17661/) (2024)
- [APLA: A Simple Adaptation Method for Vision Transformers](https://mlsanity.com/paper/2503.11335/) (2025)
- [Neutralizing Token Aggregation via Information Augmentation for Efficient Test-Time Adaptation](https://mlsanity.com/paper/2508.03388/) (2025)
- [Parameter-Free Fine-tuning via Redundancy Elimination for Vision Foundation Models](https://mlsanity.com/paper/2504.08915/) (2025)

## Abstract (arXiv)

Recently, transformers have shown strong ability as visual feature extractors, surpassing traditional convolution-based models in various scenarios. However, the success of vision transformers largely owes to their capacity to accommodate numerous parameters. As a result, new challenges for adapting large models to downstream tasks arise. On the one hand, classic fine-tuning tunes all parameters in a huge model for every task and thus easily falls into overfitting, leading to inferior performance. On the other hand, on resource-limited devices, fine-tuning stores a full copy of parameters and thus is usually impracticable for the shortage of storage space. However, few works have focused on how to efficiently and effectively transfer knowledge in a vision transformer. Existing methods did not dive into the properties of visual features, leading to inferior performance. Moreover, some of them bring heavy inference cost though benefiting storage. To tackle these problems, we propose consolidator to modify the pre-trained model with the addition of a small set of tunable parameters to temporarily store the task-specific knowledge while freezing the backbone model. Motivated by the success of group-wise convolution, we adopt grouped connections across the features extracted by fully connected layers to construct tunable parts in a consolidator. To further enhance the model's capacity to transfer knowledge under a constrained storage budget and keep inference efficient, we consolidate the parameters in two stages: 1. between adaptation and storage, and 2. between loading and inference. On a series of downstream visual tasks, our consolidator can reach up to 7.56 better accuracy than full fine-tuning with merely 0.35% parameters, and outperform state-of-the-art parameter-efficient tuning methods by a clear margin. Code is available at https://github.com/beyondhtx/Consolidator.