mlsanityData to 7 Oct 2026

AdaptFormer: Adapting Vision Transformers for Scalable Visual Recognition

arXiv 2205.13535 ↗First posted 26 May 2022cs.CVcited by 714

Not on the map. Why?

Cited by 59 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Pretraining Vision Transformers (ViTs) has achieved great success in visual recognition. A following scenario is to adapt a ViT to various image and video recognition tasks. The adaptation is challenging because of heavy computation and memory storage. Each model needs an independent and complete finetuning process to adapt to different tasks, which limits its transferability to different visual domains. To address this challenge, we propose an effective adaptation approach for Transformer, namely AdaptFormer, which can adapt the pre-trained ViTs into many different image and video tasks efficiently. It possesses several benefits more appealing than prior arts. Firstly, AdaptFormer introduces lightweight modules that only add less than 2% extra parameters to a ViT, while it is able to increase the ViT's transferability without updating its original pre-trained parameters, significantly outperforming the existing 100% fully fine-tuned models on action recognition benchmarks. Secondly, it can be plug-and-play in different Transformers and scalable to many visual tasks. Thirdly, extensive experiments on five image and video datasets show that AdaptFormer largely improves ViTs in the target domains. For example, when updating just 1.5% extra parameters, it achieves about 10% and 19% relative improvement compared to the fully fine-tuned models on Something-Something~v2 and HMDB51, respectively. Code is available at https://github.com/ShoufaChen/AdaptFormer.

Key facts. Cited by 714 papers in the map (data through 7 Oct 2026). Builds on 88 of its 103 references that are in the map. First posted on arXiv 26 May 2022.

Builds on

88 of 103 references.

References · 88

  1. 2017Attention Is All You Needcited by 59,932
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  6. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  7. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  8. 2021Swin Transformer: Hierarchical Vision Transformer using Shifted Windowscited by 12,082
  9. 2020End-to-End Object Detection with Transformerscited by 7,384
  10. 2021Masked Autoencoders Are Scalable Vision Learnerscited by 7,339
  11. 2015Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shiftcited by 6,979
  12. 2021Emerging Properties in Self-Supervised Vision Transformerscited by 6,893
38 more
  1. 2016Layer Normalizationcited by 5,579
  2. 2017mixup: Beyond Empirical Risk Minimizationcited by 5,155
  3. 2016SGDR: Stochastic Gradient Descent with Warm Restartscited by 4,993
  4. 2018GLUE: A Multi-Task Benchmark and Analysis Platform for Natural Language Understandingcited by 4,854
  5. 2020Training data-efficient image transformers & distillation through attentioncited by 4,560
  6. 2014Going Deeper with Convolutionscited by 4,231
  7. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  8. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  9. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  10. 2015Delving Deep into Rectifiers: Surpassing Human-Level Performance on ImageNet Classificationcited by 3,590
  11. 2016Gaussian Error Linear Units (GELUs)cited by 3,431
  12. 2020Deformable DETR: Deformable Transformers for End-to-End Object Detectioncited by 3,410
  13. 2017Quo Vadis, Action Recognition? A New Model and the Kinetics Datasetcited by 3,259
  14. 2021SegFormer: Simple and Efficient Design for Semantic Segmentation with Transformerscited by 2,923
  15. 2019CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Featurescited by 2,875
  16. 2019XLNet: Generalized Autoregressive Pretraining for Language Understandingcited by 2,801
  17. 2021BEiT: BERT Pre-Training of Image Transformerscited by 2,360
  18. 2021Pyramid Vision Transformer: A Versatile Backbone for Dense Prediction without Convolutionscited by 1,960
  19. 2017The Kinetics Human Action Video Datasetcited by 1,894
  20. 2021SwinIR: Image Restoration Using Swin Transformercited by 1,869
  21. 2022Visual Prompt Tuningcited by 1,674
  22. 2021ViViT: A Video Vision Transformercited by 1,610
  23. 2021An Empirical Study of Training Self-Supervised Vision Transformerscited by 1,610
  24. 2021Is Space-Time Attention All You Need for Video Understanding?cited by 1,609
  25. 2021Decision Transformer: Reinforcement Learning via Sequence Modelingcited by 1,528
  26. 2017Accurate, Large Minibatch SGD: Training ImageNet in 1 Hourcited by 1,479
  27. 2020Point Transformercited by 1,462
  28. 2021MLP-Mixer: An all-MLP Architecture for Visioncited by 1,441
  29. 2020Rethinking Semantic Segmentation from a Sequence-to-Sequence Perspective with Transformerscited by 1,368
  30. 2022VideoMAE: Masked Autoencoders are Data-Efficient Learners for Self-Supervised Video Pre-Trainingcited by 1,305
  31. 2021BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-modelscited by 1,234
  32. 2021Tokens-to-Token ViT: Training Vision Transformers from Scratch on ImageNetcited by 1,096
  33. 2017The "something something" video database for learning and evaluating visual common sensecited by 1,092
  34. 2021Video Swin Transformercited by 1,090
  35. 2015Unsupervised Visual Representation Learning by Context Predictioncited by 1,062
  36. 2021Scaling Vision Transformerscited by 993
  37. 2017Revisiting Unreasonable Effectiveness of Data in Deep Learning Eracited by 916
  38. 2021Multiscale Vision Transformerscited by 896

50 most cited of 88 All 88 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 20; 2023: 135; 2024: 227; 2025: 201; 2026: 131.

714 papers.

Sort:

By area:

  1. 2024Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Surveycited by 641
  2. 2023Medical SAM Adapter: Adapting Segment Anything Model for Medical Image Segmentationcited by 485
  3. 2022Scaling & Shifting Your Features: A New Baseline for Efficient Model Tuningcited by 292
  4. 2024InfLoRA: Interference-Free Low-Rank Adaptation for Continual Learningcited by 189
  5. 2023DreamVideo: Composing Your Dream Videos with Customized Subject and Motioncited by 186
  6. 2023Revisiting Class-Incremental Learning with Pre-Trained Models: Generalizability and Adaptivity are All You Needcited by 183
  7. 2023Forgery-aware Adaptive Transformer for Generalizable Synthetic Image Detectioncited by 180
  8. 2023RanPAC: Random Projections and Pre-trained Models for Continual Learningcited by 162
  9. 2024Continual Learning with Pre-Trained Models: A Surveycited by 151
  10. 2023AIM: Adapting Image Models for Efficient Video Action Recognitioncited by 145
  11. 2023Visual Prompt Multi-Modal Trackingcited by 141
  12. 2022Neural Prompt Searchcited by 133
38 more
  1. 2022FacT: Factor-Tuning for Lightweight Adaptation on Vision Transformercited by 133
  2. 2023A Unified Continual Learning Framework with General Parameter-Efficient Tuningcited by 124
  3. 2022Convolutional Bypasses Are Better Vision Transformer Adapterscited by 120
  4. 2024Expandable Subspace Ensemble for Pre-Trained Model-Based Class-Incremental Learningcited by 115
  5. 2023Explicit Visual Prompting for Low-Level Structure Segmentationscited by 113
  6. 2024Revisiting Zeroth-Order Optimization for Memory-Efficient LLM Fine-Tuning: A Benchmarkcited by 113
  7. 2023Parameter-Efficient Orthogonal Finetuning via Butterfly Factorizationcited by 112
  8. 2024Low-Rank Few-Shot Adaptation of Vision-Language Modelscited by 103
  9. 2023SimDA: Simple Diffusion Adapter for Efficient Video Generationcited by 101
  10. 2023Cheap and Quick: Efficient Vision-Language Instruction Tuning for Large Language Modelscited by 100
  11. 2024GOAT-Bench: A Benchmark for Multi-Modal Lifelong Navigationcited by 93
  12. 2023One-for-All: Generalized LoRA for Parameter-Efficient Fine-tuningcited by 84
  13. 2023DiffFit: Unlocking Transferability of Large Diffusion Models via Simple Parameter-Efficient Fine-Tuningcited by 82
  14. 2023Stronger, Fewer, & Superior: Harnessing Vision Foundation Models for Domain Generalized Semantic Segmentationcited by 82
  15. 2023LION : Empowering Multimodal Large Language Model with Dual-Level Visual Knowledgecited by 80
  16. 2023Learning without Forgetting for Vision-Language Modelscited by 79
  17. 2023Review of Large Vision Models and Visual Prompt Engineeringcited by 77
  18. 2023Sensitivity-Aware Visual Parameter-Efficient Fine-Tuningcited by 74
  19. 2023Towards Efficient Visual Adaption via Structural Re-parameterizationcited by 73
  20. 2023BlackVIP: Black-Box Visual Prompting for Robust Transfer Learningcited by 70
  21. 2023The effectiveness of MAE pre-pretraining for billion-scale pretrainingcited by 68
  22. 2024OneTracker: Unifying Visual Object Tracking with Foundation Models and Efficient Tuningcited by 64
  23. 2022Conv-Adapter: Exploring Parameter Efficient Transfer Learning for ConvNetscited by 62
  24. 2023VMT-Adapter: Parameter-Efficient Transfer Learning for Multi-Task Dense Scene Understandingcited by 62
  25. 2023Bridging Vision and Language Encoders: Parameter-Efficient Tuning for Referring Image Segmentationcited by 61
  26. 2024CricaVPR: Cross-image Correlation-aware Representation Learning for Visual Place Recognitioncited by 61
  27. 2024Task-Customized Mixture of Adapters for General Image Fusioncited by 59
  28. 2023ViDA: Homeostatic Visual Domain Adapter for Continual Test Time Adaptationcited by 58
  29. 2023Long-Tail Learning with Foundation Model: Heavy Fine-Tuning Hurtscited by 58
  30. 2022Visual Query Tuning: Towards Effective Usage of Intermediate Representations for Parameter and Memory Efficient Transfer Learningcited by 57
  31. 2023Diversity-Aware Meta Visual Promptingcited by 57
  32. 2023Improving Visual Prompt Tuning for Self-supervised Vision Transformerscited by 57
  33. 2023Parameter-Efficient Fine-Tuning for Medical Image Analysis: The Missed Opportunitycited by 54
  34. 2022Pro-tuning: Unified Prompt Tuning for Vision Taskscited by 53
  35. 2023LoRAPrune: Structured Pruning Meets Low-Rank Parameter-Efficient Fine-Tuningcited by 53
  36. 2024Self-Expansion of Pre-trained Models with Mixture of Adapters for Continual Learningcited by 53
  37. 2024Dynamic Adapter Meets Prompt Tuning: Parameter-Efficient Transfer Learning for Point Cloud Analysiscited by 52
  38. 2024Toward Visual Grounding: A Surveycited by 52

50 most cited of 714 All 714 (JSON)