# Dynamic Adapter with Semantics Disentangling for Cross-lingual Cross-modal Retrieval

arXiv: https://arxiv.org/abs/2412.13510 · HTML: https://mlsanity.com/paper/2412.13510/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 41 references that are in the map.
- First posted on arXiv 18 Dec 2024.

## Builds on (most cited first, 23 of 23)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [Prefix-Tuning: Optimizing Continuous Prompts for Generation](https://mlsanity.com/paper/2101.00190/) (2021)
- [Microsoft COCO Captions: Data Collection and Evaluation Server](https://mlsanity.com/paper/1504.00325/) (2015)
- [MAD-X: An Adapter-Based Framework for Multi-Task Cross-Lingual Transfer](https://mlsanity.com/paper/2005.00052/) (2020)
- [Multi30K: Multilingual English-German Image Descriptions](https://mlsanity.com/paper/1605.00459/) (2016)
- [Causal Intervention for Weakly-Supervised Semantic Segmentation](https://mlsanity.com/paper/2009.12547/) (2020)
- [Cross Modal Retrieval with Querybank Normalisation](https://mlsanity.com/paper/2112.12777/) (2021)
- [UC2: Universal Cross-lingual Cross-modal Vision-and-Language Pre-training](https://mlsanity.com/paper/2104.00332/) (2021)
- [Semantics Disentangling for Text-to-Image Generation](https://mlsanity.com/paper/1904.01480/) (2019)
- [Semantics Disentangling for Generalized Zero-Shot Learning](https://mlsanity.com/paper/2101.07978/) (2021)
- [Learning Disentangled Semantic Representation for Domain Adaptation](https://mlsanity.com/paper/2012.11807/) (2020)
- [MURAL: Multimodal, Multitask Retrieval Across Languages](https://mlsanity.com/paper/2109.05125/) (2021)
- [DoRA: Enhancing Parameter-Efficient Fine-Tuning with Dynamic Rank Distribution](https://mlsanity.com/paper/2405.17357/) (2024)
- [You Can Ground Earlier than See: An Effective and Efficient Pipeline for Temporal Sentence Grounding in Compressed Videos](https://mlsanity.com/paper/2303.07863/) (2023)
- [Cross-View Language Modeling: Towards Unified Cross-Lingual Cross-Modal Pre-training](https://mlsanity.com/paper/2206.00621/) (2022)
- [Cross-Lingual Cross-Modal Retrieval with Noise-Robust Learning](https://mlsanity.com/paper/2208.12526/) (2022)
- [Progressive Localization Networks for Language-based Moment Localization](https://mlsanity.com/paper/2102.01282/) (2021)
- [CL2CM: Improving Cross-Lingual Cross-Modal Retrieval via Cross-Lingual Knowledge Transfer](https://mlsanity.com/paper/2312.08984/) (2023)
- [Image search using multilingual texts: a cross-modal learning approach between image and text](https://mlsanity.com/paper/1903.11299/) (2019)
- [Mitigate Negative Transfer with Similarity Heuristic Lifelong Prompt Tuning](https://mlsanity.com/paper/2406.12251/) (2024)
- [Disentangling Masked Autoencoders for Unsupervised Domain Generalization](https://mlsanity.com/paper/2407.07544/) (2024)

## Built on by (most cited first, 1 of 1)

- [DARA: Degradation-Aware Low-Rank Residual Adaptation with Original-to-Corrupted Distillation for Corruption-Robust Animal Re-Identification](https://mlsanity.com/paper/2607.16644/) (2026)

## Abstract (arXiv)

Existing cross-modal retrieval methods typically rely on large-scale vision-language pair data. This makes it challenging to efficiently develop a cross-modal retrieval model for under-resourced languages of interest. Therefore, Cross-lingual Cross-modal Retrieval (CCR), which aims to align vision and the low-resource language (the target language) without using any human-labeled target-language data, has gained increasing attention. As a general parameter-efficient way, a common solution is to utilize adapter modules to transfer the vision-language alignment ability of Vision-Language Pretraining (VLP) models from a source language to a target language. However, these adapters are usually static once learned, making it difficult to adapt to target-language captions with varied expressions. To alleviate it, we propose Dynamic Adapter with Semantics Disentangling (DASD), whose parameters are dynamically generated conditioned on the characteristics of the input captions. Considering that the semantics and expression styles of the input caption largely influence how to encode it, we propose a semantic disentangling module to extract the semantic-related and semantic-agnostic features from the input, ensuring that generated adapters are well-suited to the characteristics of input caption. Extensive experiments on two image-text datasets and one video-text dataset demonstrate the effectiveness of our model for cross-lingual cross-modal retrieval, as well as its good compatibility with various VLP models.