mlsanityData to 7 Oct 2026

Convolutional Bypasses Are Better Vision Transformer Adapters

arXiv 2207.07039 ↗First posted 14 Jul 2022cs.CVcited by 120

Not on the map. Why?

Abstract (arXiv)

The pretrain-then-finetune paradigm has been widely adopted in computer vision. But as the size of Vision Transformer (ViT) grows exponentially, the full finetuning becomes prohibitive in view of the heavier storage overhead. Motivated by parameter-efficient transfer learning (PETL) on language transformers, recent studies attempt to insert lightweight adaptation modules (e.g., adapter layers or prompt tokens) to pretrained ViT and only finetune these modules while the pretrained weights are frozen. However, these modules were originally proposed to finetune language models and did not take into account the prior knowledge specifically for visual tasks. In this paper, we propose to construct Convolutional Bypasses (Convpass) in ViT as adaptation modules, introducing only a small amount (less than 0.5% of model parameters) of trainable parameters to adapt the large ViT. Different from other PETL methods, Convpass benefits from the hard-coded inductive bias of convolutional layers and thus is more suitable for visual tasks, especially in the low-data regime. Experimental results on VTAB-1K benchmark and few-shot learning datasets show that Convpass outperforms current language-oriented adaptation modules, demonstrating the necessity to tailor vision-oriented adaptation modules for adapting vision models.

Key facts. Cited by 120 papers in the map (data through 7 Oct 2026). Builds on 39 of its 57 references that are in the map. First posted on arXiv 14 Jul 2022.

Builds on

39 of 57 references.

References · 39

  1. 2017Attention Is All You Needcited by 59,932
  2. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  3. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  4. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  5. 2020Language Models are Few-Shot Learnerscited by 32,199
  6. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  7. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  8. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  9. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  10. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  11. 2016Layer Normalizationcited by 5,579
  12. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
27 more
  1. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  2. 2022A ConvNet for the 2020scited by 3,545
  3. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,360
  4. 2021Learning to Prompt for Vision-Language Modelscited by 2,269
  5. 2013Describing Textures in the Wildcited by 2,110
  6. 2020The Many Faces of Robustness: A Critical Analysis of Out-of-Distribution Generalizationcited by 1,842
  7. 2013Fine-Grained Visual Classification of Aircraftcited by 1,701
  8. 2022Visual Prompt Tuningcited by 1,674
  9. 2019Do ImageNet Classifiers Generalize to ImageNet?cited by 1,507
  10. 2017EuroSAT: A Novel Dataset and Deep Learning Benchmark for Land Use and Land Cover Classificationcited by 1,488
  11. 2016CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoningcited by 1,445
  12. 2022Conditional Prompt Learning for Vision-Language Modelscited by 1,427
  13. 2019Natural Adversarial Examplescited by 1,401
  14. 2022CoCa: Contrastive Captioners are Image-Text Foundation Modelscited by 1,265
  15. 2021BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-modelscited by 1,234
  16. 2019Learning Robust Global Representations by Penalizing Local Predictive Powercited by 1,010
  17. 2021Scaling Vision Transformerscited by 993
  18. 2021CvT: Introducing Convolutions to Vision Transformerscited by 946
  19. 2021Towards a Unified View of Parameter-Efficient Transfer Learningcited by 863
  20. 2017Remote Sensing Image Scene Classification: Benchmark and State of the Artcited by 791
  21. 2020AdapterFusion: Non-Destructive Task Composition for Transfer Learningcited by 780
  22. 2019Big Transfer (BiT): General Visual Representation Learningcited by 766
  23. 2022AdaptFormer: Adapting Vision Transformers for Scalable Visual Recognitioncited by 714
  24. 2021P-Tuning v2: Prompt Tuning Can Be Comparable to Fine-tuning Universally Across Scales and Taskscited by 686
  25. 2021VLMo: Unified Vision-Language Pre-Training with Mixture-of-Modality-Expertscited by 424
  26. 2018Rotation Equivariant CNNs for Digital Pathologycited by 420
  27. 2022Neural Prompt Searchcited by 133

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 4; 2023: 38; 2024: 49; 2025: 12; 2026: 17.

120 papers.

Sort:

By area:

  1. 2024Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Surveycited by 641
  2. 2023AIM: Adapting Image Models for Efficient Video Action Recognitioncited by 145
  3. 2023Visual Prompt Multi-Modal Trackingcited by 141
  4. 2022FacT: Factor-Tuning for Lightweight Adaptation on Vision Transformercited by 133
  5. 2023SimDA: Simple Diffusion Adapter for Efficient Video Generationcited by 101
  6. 2023Review of Large Vision Models and Visual Prompt Engineeringcited by 77
  7. 2023Sensitivity-Aware Visual Parameter-Efficient Fine-Tuningcited by 74
  8. 2023Towards Efficient Visual Adaption via Structural Re-parameterizationcited by 73
  9. 2024Convolution Meets LoRA: Parameter Efficient Finetuning for Segment Anything Modelcited by 67
  10. 2024CricaVPR: Cross-image Correlation-aware Representation Learning for Visual Place Recognitioncited by 61
  11. 2022Visual Query Tuning: Towards Effective Usage of Intermediate Representations for Parameter and Memory Efficient Transfer Learningcited by 57
  12. 2024Self-Expansion of Pre-trained Models with Mixture of Adapters for Continual Learningcited by 53
38 more
  1. 2024Towards Seamless Adaptation of Pre-trained Models for Visual Place Recognitioncited by 49
  2. 2023Revisiting the Parameter Efficiency of Adapters from the Perspective of Precision Redundancycited by 44
  3. 2024Parameter-Efficient Fine-Tuning in Large Models: A Survey of Methodologiescited by 44
  4. 2025Parameter-Efficient Fine-Tuning for Foundation Modelscited by 43
  5. 2023Dual-path Adaptation from Image to Video Transformerscited by 40
  6. 2023SAMAug: Point Prompt Augmentation for Segment Anything Modelcited by 38
  7. 2024ALoRA: Allocating Low-Rank Adaptation for Fine-tuning Large Language Modelscited by 36
  8. 2023Visual Tuningcited by 34
  9. 2024Temporal Adaptive RGBT Tracking with Modality Promptcited by 30
  10. 2024AUFormer: Vision Transformers are Parameter-Efficient Facial Action Unit Detectorscited by 29
  11. 2024Few-shot Adaptation of Multi-modal Foundation Models: A Surveycited by 28
  12. 2024Time-, Memory- and Parameter-Efficient Visual Adaptationcited by 28
  13. 2024MoE-FFD: Mixture of Experts for Generalized and Parameter-Efficient Face Forgery Detectioncited by 27
  14. 2023Res-Tuning: A Flexible and Efficient Tuning Paradigm via Unbinding Tuner from Backbonecited by 26
  15. 2024Dynamic Tuning Towards Parameter and Inference Efficiency for ViT Adaptationcited by 25
  16. 2023Parameter-efficient is not sufficient: Exploring Parameter, Memory, and Time Efficient Adapter Tuning for Dense Predictionscited by 24
  17. 2023Parameter-Efficient Transfer Learning for Remote Sensing Image-Text Retrievalcited by 24
  18. 2024A Simple yet Effective Network based on Vision Transformer for Camouflaged Object and Salient Object Detectioncited by 23
  19. 2024Parameter-Efficient Fine-Tuning for Pre-Trained Vision Models: A Survey and Benchmarkcited by 21
  20. 2023SCT: A Simple Baseline for Parameter-Efficient Fine-Tuning via Salient Channelscited by 20
  21. 2023Rethinking Vision Transformer and Masked Autoencoder in Multimodal Face Anti-Spoofingcited by 19
  22. 2023Consolidator: Mergeable Adapter with Grouped Connections for Visual Adaptationcited by 18
  23. 2023VLAB: Enhancing Video Language Pre-training by Feature Adapting and Blendingcited by 17
  24. 2023Modality-missing RGBT Tracking: Invertible Prompt Learning and High-quality Benchmarkscited by 17
  25. 2024PYRA: Parallel Yielding Re-Activation for Training-Inference Efficient Task Adaptationcited by 17
  26. 2024Random Masking Finds Winning Tickets for Parameter Efficient Fine-tuningcited by 16
  27. 2024Towards Data-Centric Face Anti-Spoofing: Improving Cross-domain Generalization via Physics-based Data Synthesiscited by 16
  28. 2024Lessons and Insights from a Unifying Study of Parameter-Efficient Fine-Tuning (PEFT) in Visual Recognitioncited by 16
  29. 2025RingMoE: Mixture-of-Modality-Experts Multi-Modal Foundation Models for Universal Remote Sensing Image Interpretationcited by 16
  30. 2023Audio-Visual Deception Detection: DOLOS Dataset and Parameter-Efficient Crossmodal Learningcited by 14
  31. 2023Overview of the PromptCBLUE Shared Task in CHIP2023cited by 14
  32. 2022Prompt Tuning with Soft Context Sharing for Vision-Language Modelscited by 13
  33. 2024UMG-CLIP: A Unified Multi-Granularity Vision Generalist for Open-World Understandingcited by 13
  34. 2023On the Efficacy of Differentially Private Few-shot Image Classificationcited by 12
  35. 2023Rehearsal-Free Domain Continual Face Anti-Spoofing: Generalize More and Forget Lesscited by 12
  36. 2023ZEETAD: Adapting Pretrained Vision-Language Model for Zero-Shot End-to-End Temporal Action Detectioncited by 12
  37. 2024Sparse-Tuning: Adapting Vision Transformers with Efficient Fine-tuning and Inferencecited by 12
  38. 2022Exploring Parameter-Efficient Fine-Tuning to Enable Foundation Models in Federated Learningcited by 11

50 most cited of 120 All 120 (JSON)