# Cross-Modal Adapter for Vision-Language Retrieval

arXiv: https://arxiv.org/abs/2211.09623 · HTML: https://mlsanity.com/paper/2211.09623/

## Key facts

- Cited by 35 papers in the map (data through 14 Oct 2026).
- Builds on 37 of its 48 references that are in the map.
- First posted on arXiv 17 Nov 2022.

## Builds on (most cited first, 30 of 37)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [Densely Connected Convolutional Networks](https://mlsanity.com/paper/1608.06993/) (2016)
- [BLIP: Bootstrapping Language-Image Pre-training for Unified Vision-Language Understanding and Generation](https://mlsanity.com/paper/2201.12086/) (2022)
- [Parameter-Efficient Transfer Learning for NLP](https://mlsanity.com/paper/1902.00751/) (2019)
- [Prefix-Tuning: Optimizing Continuous Prompts for Generation](https://mlsanity.com/paper/2101.00190/) (2021)
- [The Power of Scale for Parameter-Efficient Prompt Tuning](https://mlsanity.com/paper/2104.08691/) (2021)
- [VQA: Visual Question Answering](https://mlsanity.com/paper/1505.00468/) (2015)
- [Learning to Prompt for Vision-Language Models](https://mlsanity.com/paper/2109.01134/) (2021)
- [Language Models as Knowledge Bases?](https://mlsanity.com/paper/1909.01066/) (2019)
- [Visual Prompt Tuning](https://mlsanity.com/paper/2203.12119/) (2022)
- [Align before Fuse: Vision and Language Representation Learning with Momentum Distillation](https://mlsanity.com/paper/2107.07651/) (2021)
- [CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoning](https://mlsanity.com/paper/1612.06890/) (2016)
- [Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Models](https://mlsanity.com/paper/1505.04870/) (2015)
- [Oscar: Object-Semantics Aligned Pre-training for Vision-Language Tasks](https://mlsanity.com/paper/2004.06165/) (2020)
- [Towards a Unified View of Parameter-Efficient Transfer Learning](https://mlsanity.com/paper/2110.04366/) (2021)
- [Florence: A New Foundation Model for Computer Vision](https://mlsanity.com/paper/2111.11432/) (2021)
- [Dense-Captioning Events in Videos](https://mlsanity.com/paper/1705.00754/) (2017)
- [AdaptFormer: Adapting Vision Transformers for Scalable Visual Recognition](https://mlsanity.com/paper/2205.13535/) (2022)
- [MaPLe: Multi-modal Prompt Learning](https://mlsanity.com/paper/2210.03117/) (2022)
- [Localizing Moments in Video with Natural Language](https://mlsanity.com/paper/1708.01641/) (2017)
- [CLIP4Clip: An Empirical Study of CLIP for End to End Video Clip Retrieval](https://mlsanity.com/paper/2104.08860/) (2021)
- [VideoCLIP: Contrastive Pre-training for Zero-shot Video-Text Understanding](https://mlsanity.com/paper/2109.14084/) (2021)
- [Image as a Foreign Language: BEiT Pretraining for All Vision and Vision-Language Tasks](https://mlsanity.com/paper/2208.10442/) (2022)
- [MAttNet: Modular Attention Network for Referring Expression Comprehension](https://mlsanity.com/paper/1801.08186/) (2018)
- [X-CLIP: End-to-End Multi-grained Contrastive Learning for Video-Text Retrieval](https://mlsanity.com/paper/2207.07285/) (2022)

## Built on by (most cited first, 30 of 35)

- [Unified Coarse-to-Fine Alignment for Video-Text Retrieval](https://mlsanity.com/paper/2309.10091/) (2023)
- [Video-Language Understanding: A Survey from Model Architecture, Model Training, and Data Perspectives](https://mlsanity.com/paper/2406.05615/) (2024)
- [Prompt Switch: Efficient CLIP Adaptation for Text-Video Retrieval](https://mlsanity.com/paper/2308.07648/) (2023)
- [Visual Tuning](https://mlsanity.com/paper/2305.06061/) (2023)
- [Troika: Multi-Path Cross-Modal Traction for Compositional Zero-Shot Learning](https://mlsanity.com/paper/2303.15230/) (2023)
- [DGL: Dynamic Global-Local Prompt Tuning for Text-Video Retrieval](https://mlsanity.com/paper/2401.10588/) (2024)
- [Parameter-Efficient Transfer Learning for Remote Sensing Image-Text Retrieval](https://mlsanity.com/paper/2308.12509/) (2023)
- [Selective Vision-Language Subspace Projection for Few-shot CLIP](https://mlsanity.com/paper/2407.16977/) (2024)
- [RAP: Efficient Text-Video Retrieval with Sparse-and-Correlated Adapter](https://mlsanity.com/paper/2405.19465/) (2024)
- [Cardiac Copilot: Automatic Probe Guidance for Echocardiography with World Model](https://mlsanity.com/paper/2406.13165/) (2024)
- [RGNet: A Unified Clip Retrieval and Grounding Network for Long Videos](https://mlsanity.com/paper/2312.06729/) (2023)
- [Side4Video: Spatial-Temporal Side Network for Memory-Efficient Image-to-Video Transfer Learning](https://mlsanity.com/paper/2311.15769/) (2023)
- [M2IST: Multi-Modal Interactive Side-Tuning for Efficient Referring Expression Comprehension](https://mlsanity.com/paper/2407.01131/) (2024)
- [Structure-aware World Model for Probe Guidance via Large-scale Self-supervised Pre-train](https://mlsanity.com/paper/2406.19756/) (2024)
- [MaPPER: Multimodal Prior-guided Parameter Efficient Tuning for Referring Expression Comprehension](https://mlsanity.com/paper/2409.13609/) (2024)
- [TaCA: Upgrading Your Visual Foundation Model with Task-agnostic Compatible Adapter](https://mlsanity.com/paper/2306.12642/) (2023)
- [DARA: Domain- and Relation-aware Adapters Make Parameter-efficient Tuning for Visual Grounding](https://mlsanity.com/paper/2405.06217/) (2024)
- [RETTA: Retrieval-Enhanced Test-Time Adaptation for Zero-Shot Video Captioning](https://mlsanity.com/paper/2405.07046/) (2024)
- [Deep Transfer Learning: Model Framework and Error Analysis](https://mlsanity.com/paper/2410.09383/) (2024)
- [READ: Recurrent Adapter with Partial Video-Language Alignment for Parameter-Efficient Transfer Learning in Low-Resource Video-Language Modeling](https://mlsanity.com/paper/2312.06950/) (2023)
- [Efficient Remote Sensing with Harmonized Transfer Learning and Modality Alignment](https://mlsanity.com/paper/2404.18253/) (2024)
- [CLIP model is an Efficient Online Lifelong Learner](https://mlsanity.com/paper/2405.15155/) (2024)
- [OneEncoder: A Lightweight Framework for Progressive Alignment of Modalities](https://mlsanity.com/paper/2409.11059/) (2024)
- [Diversity Covariance-Aware Prompt Learning for Vision-Language Models](https://mlsanity.com/paper/2503.01531/) (2025)
- [FORGE: Forensic Reasoning with Grounded Evidence](https://mlsanity.com/paper/2503.15867/) (2025)
- [Beyond Coarse-Grained Matching in Video-Text Retrieval](https://mlsanity.com/paper/2410.12407/) (2024)
- [SPECTRUM: Semantic Processing and Emotion-informed video-Captioning Through Retrieval and Understanding Modalities](https://mlsanity.com/paper/2411.01975/) (2024)
- [Representation Discrepancy Bridging Method for Remote Sensing Image-Text Retrieval](https://mlsanity.com/paper/2505.16756/) (2025)
- [A Resource-Efficient Training Framework for Remote Sensing Text--Image Retrieval](https://mlsanity.com/paper/2501.10638/) (2025)
- [HMID-Net: An Exploration of Masked Image Modeling and Knowledge Distillation in Hyperbolic Space](https://mlsanity.com/paper/2507.09487/) (2025)

## Abstract (arXiv)

Vision-language retrieval is an important multi-modal learning topic, where the goal is to retrieve the most relevant visual candidate for a given text query. Recently, pre-trained models, e.g., CLIP, show great potential on retrieval tasks. However, as pre-trained models are scaling up, fully fine-tuning them on donwstream retrieval datasets has a high risk of overfitting. Moreover, in practice, it would be costly to train and store a large model for each task. To overcome the above issues, we present a novel Cross-Modal Adapter for parameter-efficient transfer learning. Inspired by adapter-based methods, we adjust the pre-trained model with a few parameterization layers. However, there are two notable differences. First, our method is designed for the multi-modal domain. Secondly, it allows encoder-level implicit cross-modal interactions between vision and language encoders. Although surprisingly simple, our approach has three notable benefits: (1) reduces the vast majority of fine-tuned parameters, (2) saves training time, and (3) allows all the pre-trained parameters to be fixed, enabling the pre-trained model to be shared across datasets. Extensive experiments demonstrate that, without bells and whistles, our approach outperforms adapter-based methods on image-text retrieval datasets (MSCOCO, Flickr30K) and video-text retrieval datasets (MSR-VTT, DiDeMo, and ActivityNet).