mlsanityData to 14 Oct 2026

Mini but Mighty: Finetuning ViTs with Mini Adapters

arXiv 2311.03873 ↗First posted 7 Nov 2023cs.CV cs.AIcited by 6

Not on the map. Why?

Abstract (arXiv)

Vision Transformers (ViTs) have become one of the dominant architectures in computer vision, and pre-trained ViT models are commonly adapted to new tasks via fine-tuning. Recent works proposed several parameter-efficient transfer learning methods, such as adapters, to avoid the prohibitive training and storage cost of finetuning. In this work, we observe that adapters perform poorly when the dimension of adapters is small, and we propose MiMi, a training framework that addresses this issue. We start with large adapters which can reach high performance, and iteratively reduce their size. To enable automatic estimation of the hidden dimension of every adapter, we also introduce a new scoring function, specifically designed for adapters, that compares the neuron importance across layers. Our method outperforms existing methods in finding the best trade-off between accuracy and trained parameters across the three dataset benchmarks DomainNet, VTAB, and Multi-task, for a total of 29 datasets.

Key facts. Cited by 6 papers in the map (data through 14 Oct 2026). Builds on 39 of its 62 references that are in the map. First posted on arXiv 7 Nov 2023.

Builds on

39 of 62 references.

References · 39

  1. 2017Attention Is All You Needcited by 60,361
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,380
  4. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,139
  5. 2020Training data-efficient image transformers & distillation through attentioncited by 4,586
  6. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,236
  7. 2015Deep Compression: Compressing Deep Neural Networks with Pruning, Trained Quantization and Huffman Codingcited by 2,329
  8. 2018The Lottery Ticket Hypothesis: Finding Sparse, Trainable Neural Networkscited by 2,262
  9. 2015Learning both Weights and Connections for Efficient Neural Networkscited by 2,031
  10. 2017EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classificationcited by 1,496
  11. 2016CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoningcited by 1,457
  12. 2021BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-modelscited by 1,245
27 more
  1. 2021CvT: Introducing Convolutions to Vision Transformerscited by 947
  2. 2021Towards a Unified View of Parameter-Efficient Transfer Learningcited by 872
  3. 2019Are Sixteen Heads Really Better than One?cited by 816
  4. 2020AdapterFusion: Non-Destructive Task Composition for Transfer Learningcited by 788
  5. 2022AdaptFormer: Adapting Vision Transformers for Scalable Visual Recognitioncited by 719
  6. 2017Channel Pruning for Accelerating Very Deep Neural Networkscited by 702
  7. 2017Learning multiple visual domains with residual adapterscited by 557
  8. 2020Picking Winning Tickets Before Training by Preserving Gradient Flowcited by 471
  9. 2021Compacter: Efficient Low-Rank Hypercomplex Adapter Layerscited by 460
  10. 2019A Large-scale Study of Representation Learning with the Visual Task Adaptation Benchmarkcited by 425
  11. 2018Rotation Equivariant CNNs for Digital Pathologycited by 422
  12. 2021CMT: Convolutional Neural Networks Meet Vision Transformerscited by 310
  13. 2021Parameter-efficient Multi-task Fine-tuning for Transformers via Shared Hypernetworkscited by 284
  14. 2020Comparing Rewinding and Fine-tuning in Neural Network Pruningcited by 240
  15. 2020AdapterDrop: On the Efficiency of Adapters in Transformerscited by 224
  16. 2018Efficient parametrization of multi-domain deep neural networkscited by 168
  17. 2022FacT: Factor-Tuning for Lightweight Adaptation on Vision Transformercited by 133
  18. 2021Efficient Training of Visual Transformers with Small Datasetscited by 108
  19. 2022SparseAdapter: An Easy Approach for Improving the Parameter-Efficiency of Adapterscited by 83
  20. 2017Fine-Grained Car Detection for Visual Census Estimationcited by 76
  21. 2021Beyond Fully-Connected Layers with Quaternions: Parameterization of Hypercomplex Multiplications with 1/n Parameterscited by 66
  22. 2023AutoPEFT: Automatic Configuration Search for Parameter-Efficient Fine-Tuningcited by 49
  23. 2019Identity Crisis: Memorization and Generalization under Extreme Overparameterizationcited by 33
  24. 2018Learning Sparse Neural Networks via Sensitivity-Driven Regularizationcited by 26
  25. 2019Budget-Aware Adapters for Multi-Domain Learningcited by 18
  26. 2022Sparse Structure Search for Parameter-Efficient Tuningcited by 17
  27. 2019Multi-Source Domain Adaptation and Semi-Supervised Domain Adaptation with Focus on Visual Domain Adaptation Challenge 2019cited by 2

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 1; 2025: 4; 2026: 1.

6 papers.

  1. 2024Parameter-Efficient Fine-Tuning for Pre-Trained Vision Models: A Survey and Benchmarkcited by 23
  2. 2025Gradient Inversion Attacks on Parameter-Efficient Fine-Tuningcited by 3
  3. 2025End-to-End Analysis of Charge Stability Diagrams with Transformerscited by 3
  4. 2025F-Adapter: Frequency-Adaptive Parameter-Efficient Fine-Tuning in Scientific Machine Learningcited by 3
  5. 2025Efficient Adaptation of Deep Neural Networks for Semantic Segmentation in Space Applicationscited by 1
  6. 2026From Pen to Pixel: Translating Hand-Drawn Plots into Graphical APIs via a Novel Benchmark and Efficient Adaptercited by 0