mlsanityData to 7 Oct 2026

VL-Adapter: Parameter-Efficient Transfer Learning for Vision-and-Language Tasks

arXiv 2112.06825 ↗First posted 13 Dec 2021cs.CV cs.AI cs.CLcited by 319

Not on the map. Why?

Cited by 61 in Multimodal LLMs / vision-language models, but placed by its own abstract, which didn’t fit.

Abstract (arXiv)

Recently, fine-tuning language models pre-trained on large text corpora have provided huge improvements on vision-and-language (V&L) tasks as well as on pure language tasks. However, fine-tuning the entire parameter set of pre-trained models becomes impractical since the model size is growing rapidly. Hence, in this paper, we introduce adapter-based parameter-efficient transfer learning techniques to V&L models such as VL-BART and VLT5. We evaluate our methods in a unified multi-task setup on both image-text and video-text benchmarks. For the image-text tasks, we use four diverse V&L datasets: VQAv2, GQA, NLVR2 , and MSCOCO image captioning. For video-text tasks, we use TVQA, How2QA, TVC, and YC2C. With careful training and thorough experiments, we benchmark three popular adapter-based methods (Adapter, Hyperformer, Compacter) against the standard full fine-tuning and the recently proposed prompt-tuning approach. We also enhance the efficiency and performance of adapters by sharing their weights to attain knowledge across tasks. Our results demonstrate that training the adapter with the weight-sharing technique (4.18% of total parameters for image-text tasks and 3.39% for video-text tasks) can match the performance of fine-tuning the entire model. Lastly, we present a comprehensive analysis including the combination of adapter and task-specific prompts and the impact of V&L pre-training on adapters. Our code is available at: https://github.com/ylsung/VL_adapter.

Key facts. Cited by 319 papers in the map (data through 7 Oct 2026). Builds on 49 of its 58 references that are in the map. First posted on arXiv 13 Dec 2021.

Builds on

49 of 58 references.

References · 49

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  6. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  7. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  8. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  9. 2019RoBERTa: A Robustly Optimized BERT Pretraining Approachcited by 13,266
  10. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,617
  11. 2019BART: Denoising Sequence-to-Sequence Pre-training for Natural Language Generation, Translation, and Comprehensioncited by 5,874
  12. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
37 more
  1. 2021Prefix-Tuning: Optimizing Continuous Prompts for Generationcited by 3,878
  2. 2021The Power of Scale for Parameter-Efficient Prompt Tuningcited by 3,769
  3. 2016Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotationscited by 2,646
  4. 2016Making the V in VQA Matter: Elevating the Role of Image Understanding in Visual Question Answeringcited by 2,498
  5. 2018Deep contextualized word representationscited by 2,285
  6. 2021Learning to Prompt for Vision-Language Modelscited by 2,269
  7. 2019ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Taskscited by 2,219
  8. 2019LXMERT: Learning Cross-Modality Encoder Representations from Transformerscited by 1,548
  9. 2015Microsoft COCO Captions: Data Collection and Evaluation Servercited by 1,447
  10. 2017Bottom-Up and Top-Down Attention for Image Captioning and Visual Question Answeringcited by 1,310
  11. 2021BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-modelscited by 1,234
  12. 2014Deep Visual-Semantic Alignments for Generating Image Descriptionscited by 966
  13. 2021Towards a Unified View of Parameter-Efficient Transfer Learningcited by 863
  14. 2021Multimodal Few-Shot Learning with Frozen Language Modelscited by 657
  15. 2017Towards Automatic Learning of Procedures from Web Instructional Videoscited by 616
  16. 2017Learning multiple visual domains with residual adapterscited by 553
  17. 2018TVQA: Localized, Compositional Video Question Answeringcited by 463
  18. 2021Compacter: Efficient Low-Rank Hypercomplex Adapter Layerscited by 458
  19. 2018A Corpus for Reasoning About Natural Language Grounded in Photographscited by 457
  20. 2021Tip-Adapter: Training-free CLIP-Adapter for Better Vision-Language Modelingcited by 429
  21. 2021Unifying Vision-and-Language Tasks via Text Generationcited by 411
  22. 2020HERO: Hierarchical Encoder for Video+Language Omni-representation Pre-trainingcited by 362
  23. 2021High-Performance Large-Scale Image Recognition Without Normalizationcited by 359
  24. 2020Parameter-Efficient Transfer Learning with Diff Pruningcited by 351
  25. 2021How Much Can CLIP Benefit Vision-and-Language Tasks?cited by 321
  26. 2021An Empirical Study of GPT-3 for Few-Shot Knowledge-Based VQAcited by 298
  27. 2020Pixel-BERT: Aligning Image Pixels with Text by Deep Multi-Modal Transformerscited by 288
  28. 2021Parameter-efficient Multi-task Fine-tuning for Transformers via Shared Hypernetworkscited by 278
  29. 2019Visual Entailment: A Novel Task for Fine-Grained Image Understandingcited by 250
  30. 2021PPT: Pre-trained Prompt Tuning for Few-shot Learningcited by 232
  31. 2020TVR: A Large-Scale Dataset for Video-Subtitle Moment Retrievalcited by 215
  32. 2021Training Neural Networks with Fixed Sparse Maskscited by 212
  33. 2018Efficient parametrization of multi-domain deep neural networkscited by 168
  34. 2021UniPELT: A Unified Framework for Parameter-Efficient Language Model Tuningcited by 162
  35. 2021VALUE: A Multi-Task Benchmark for Video-and-Language Understanding Evaluationcited by 95
  36. 2021Beyond Fully-Connected Layers with Quaternions: Parameterization of Hypercomplex Multiplications with 1/n Parameterscited by 66
  37. 2020Conditionally Adaptive Multi-Task Learning: Improving Transfer Learning in NLP Using Fewer Parameters & Less Datacited by 50

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 32; 2023: 104; 2024: 85; 2025: 68; 2026: 30.

319 papers.

Sort:

By area:

  1. 2023Adding Conditional Control to Text-to-Image Diffusion Modelscited by 5,466
  2. 2022Flamingo: a Visual Language Model for Few-Shot Learningcited by 4,337
  3. 2024DoRA: Weight-Decomposed Low-Rank Adaptationcited by 740
  4. 2024Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Surveycited by 641
  5. 2023LLaMA-Adapter V2: Parameter-Efficient Visual Instruction Modelcited by 602
  6. 2022Vision Transformer Adapter for Dense Predictionscited by 564
  7. 2022Multimodal Learning with Transformers: A Surveycited by 399
  8. 2023CodeT5+: Open Code Large Language Models for Code Understanding and Generationcited by 327
  9. 2023Side Adapter Network for Open-Vocabulary Semantic Segmentationcited by 296
  10. 2023Personalize Segment Anything Model with One Shotcited by 250
  11. 2022LST: Ladder Side-Tuning for Parameter and Memory Efficient Transfer Learningcited by 228
  12. 2023Parameter-Efficient Fine-Tuning Methods for Pretrained Language Models: A Critical Review and Assessmentcited by 227
38 more
  1. 2023Self-Chained Image-Language Model for Video Localization and Question Answeringcited by 200
  2. 2024Boosting Continual Learning of Vision-Language Models via Mixture-of-Experts Adapterscited by 199
  3. 2022ST-Adapter: Parameter-Efficient Image-to-Video Transfer Learningcited by 192
  4. 2022Patching open-vocabulary models by interpolating weightscited by 187
  5. 2024Mixture of LoRA Expertscited by 179
  6. 2023RS5M and GeoRSCLIP: A Large Scale Vision-Language Dataset and A Large Vision-Language Model for Remote Sensingcited by 150
  7. 2022ELEVATER: A Benchmark and Toolkit for Evaluating Language-Augmented Visual Modelscited by 141
  8. 2022Linearly Mapping from Image to Text Spacecited by 139
  9. 2023Continual Diffusion: Continual Customization of Text-to-Image Diffusion with C-LoRAcited by 138
  10. 2024Low-Rank Few-Shot Adaptation of Vision-Language Modelscited by 103
  11. 2022Plug-and-Play VQA: Zero-shot VQA by Conjoining Large Pretrained Models with Zero Trainingcited by 88
  12. 2022SmallCap: Lightweight Image Captioning Prompted with Retrieval Augmentationcited by 82
  13. 2023Modular Deep Learningcited by 82
  14. 2022Parameter-efficient Model Adaptation for Vision Transformerscited by 81
  15. 2024Pay Attention to Your Neighbours: Training-Free Open-Vocabulary Semantic Segmentationcited by 76
  16. 2022StoryDALL-E: Adapting Pretrained Text-to-Image Transformers for Story Continuationcited by 74
  17. 2023Towards Efficient Visual Adaption via Structural Re-parameterizationcited by 73
  18. 2022Vision Transformers are Parameter-Efficient Audio-Visual Learnerscited by 72
  19. 2022Teaching Structured Vision&Language Concepts to Vision&Language Modelscited by 71
  20. 2024LoRA-Pro: Are Low-Rank Adapters Properly Optimized?cited by 70
  21. 2023GraphAdapter: Tuning Vision-Language Models With Dual Knowledge Graphcited by 69
  22. 2024Ctrl-Adapter: An Efficient and Versatile Framework for Adapting Diverse Controls to Any Diffusion Modelcited by 64
  23. 2022Video Transformers: A Surveycited by 63
  24. 2023BadCLIP: Trigger-Aware Prompt Learning for Backdoor Attacks on CLIPcited by 63
  25. 2023VMT-Adapter: Parameter-Efficient Transfer Learning for Multi-Task Dense Scene Understandingcited by 62
  26. 2023Bridging Vision and Language Encoders: Parameter-Efficient Tuning for Referring Image Segmentationcited by 61
  27. 2023HOICLIP: Efficient Knowledge Transfer for HOI Detection with Vision-Language Modelscited by 60
  28. 2023A Survey on Open-Vocabulary Detection and Segmentation: Past, Present, and Futurecited by 59
  29. 2023FedDAT: An Approach for Foundation Model Finetuning in Multi-Modal Heterogeneous Federated Learningcited by 59
  30. 2023ViDA: Homeostatic Visual Domain Adapter for Continual Test Time Adaptationcited by 58
  31. 2024Multi-Task Dense Prediction via Mixture of Low-Rank Expertscited by 58
  32. 2023Diversity-Aware Meta Visual Promptingcited by 57
  33. 2023Towards Language Models That Can See: Computer Vision Through the LENS of Natural Languagecited by 53
  34. 2022Polyhistor: Parameter-Efficient Multi-Task Adaptation for Dense Vision Taskscited by 52
  35. 2023An Empirical Study of Multimodal Model Mergingcited by 52
  36. 2023UPop: Unified and Progressive Pruning for Compressing Vision-Language Transformerscited by 51
  37. 2023Unified Coarse-to-Fine Alignment for Video-Text Retrievalcited by 49
  38. 2023Procedure-Aware Pretraining for Instructional Video Understandingcited by 48

50 most cited of 319 All 319 (JSON)