mlsanityData to 7 Oct 2026

MultiWay-Adapater: Adapting large-scale multi-modal models for scalable image-text retrieval

arXiv 2309.01516 ↗First posted 4 Sep 2023cs.CV cs.AI cs.LGcited by 9

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

As Multimodal Large Language Models (MLLMs) grow in size, adapting them to specialized tasks becomes increasingly challenging due to high computational and memory demands. Indeed, traditional fine-tuning methods are costly, due to the need for extensive, task-specific training. While efficient adaptation methods exist that aim to reduce these costs, in practice they suffer from shallow inter-modal alignment, which severely hurts model effectiveness. To tackle these computational challenges and improve inter-modal alignment, we introduce the MultiWay-Adapter (MWA), a novel framework featuring an 'Alignment Enhancer'. This enhancer deepens inter-modal alignment, enabling high transferability with minimal tuning effort. Our experiments show that unlike prior efficient tuning approaches, MWA maintains model effectiveness, while reducing training time by up-to 57%. MWA is also lightweight, increasing model size by only 2-3% (in terms of parameters) for state-of-the-art foundation models like BEiT-3 Large. These results demonstrate that MWA provides an efficient and effective adaptation method for MLLMs, significantly broadening their applicability.

Key facts. Cited by 9 papers in the map (data through 7 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 23 of its 31 references that are in the map. First posted on arXiv 4 Sep 2023.

Builds on

23 of 31 references.

References · 23

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,498
  3. 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
  4. 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
  5. 2019Parameter-Efficient Transfer Learning for NLPcited by 4,195
  6. 2021Scaling Up Visual and Vision-Language Representation Learning With Noisy Text Supervisioncited by 3,848
  7. 2021Align before Fuse: Vision and Language Representation Learning with Momentum Distillationcited by 1,612
  8. 2015Flickr30k Entities: Collecting Region-to-Phrase Correspondences for Richer Image-to-Sentence Modelscited by 1,352
  9. 2017Multimodal Machine Learning: A Survey and Taxonomycited by 1,004
  10. 2019VL-BERT: Pre-training of Generic Visual-Linguistic Representationscited by 954
  11. 2021Towards a Unified View of Parameter-Efficient Transfer Learningcited by 863
  12. 2022AdaptFormer: Adapting Vision Transformers for Scalable Visual Recognitioncited by 714
11 more
  1. 2021LiT: Zero-Shot Transfer with Locked-image text Tuningcited by 572
  2. 2017Learning multiple visual domains with residual adapterscited by 553
  3. 2022Image as a Foreign Language: BEiT Pretraining for All Vision and Vision-Language Taskscited by 510
  4. 2021VLMo: Unified Vision-Language Pre-Training with Mixture-of-Modality-Expertscited by 424
  5. 2020Parameter-Efficient Transfer Learning with Diff Pruningcited by 351
  6. 2021VL-Adapter: Parameter-Efficient Transfer Learning for Vision-and-Language Taskscited by 319
  7. 2022Efficient Adapter Transfer of Self-Supervised Speech Models for Automatic Speech Recognitioncited by 31
  8. 2021An Adapter Based Pre-Training for Efficient and Scalable Self-Supervised Speech Representation Learningcited by 19
  9. 2022Using Adapters to Overcome Catastrophic Forgetting in End-to-End Automatic Speech Recognitioncited by 15
  10. 2022Parameter-efficient transfer learning of pre-trained Transformer models for speaker verification using adapterscited by 8
  11. 2022Adapted Multimodal BERT with Layer-wise Fusion for Sentiment Analysiscited by 2

Built on by

Cited by year

Citing papers per year: 2024: 4; 2025: 5.

9 papers.

  1. 2024Knowledge Graphs Meet Multi-Modal Learning: A Comprehensive Surveycited by 50
  2. 2024A Comprehensive Survey and Guide to Multimodal Large Language Models in Vision-Language Taskscited by 27
  3. 2025Re-Imagining Multimodal Instruction Tuning: A Representation Viewcited by 9
  4. 2025Mixture-of-Mamba: Enhancing Multi-Modal State-Space Models with Modality-Aware Sparsitycited by 7
  5. 2025CATCH: A Modular Cross-domain Adaptive Template with Hookcited by 6
  6. 2025Diffusion Augmented Retrieval: A Training-Free Approach to Interactive Text-to-Image Retrievalcited by 5
  7. 2024Understanding and Mitigating Human-Labelling Errors in Supervised Contrastive Learningcited by 3
  8. 2024Multi-Point Positional Insertion Tuning for Small Object Detectioncited by 1
  9. 2025RoboEye: Enhancing 2D Robotic Object Identification with Selective 3D Geometric Keypoint Matchingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024AlignGPT: Multi-modal Large Language Models with Adaptive Alignment Capability
  2. 2026Indexing Multimodal Language Models for Large-scale Image Retrieval
  3. 2026Strong Helps Weak: Directional Cross-Modal Alignment Transfer in Multi-modal LLMs
  4. 2024Semantic Alignment for Multimodal Large Language Models
  5. 2025Keeping Yourself is Important in Downstream Tuning Multimodal Large Language Model
  6. 2024Multimodal Needle in a Haystack: Benchmarking Long-Context Capability of Multimodal Large Language Models
  7. 2024IAA: Inner-Adaptor Architecture Empowers Frozen Large Language Model with Multimodal Capabilities
  8. 2025Multimodal Large Language Models for Text-rich Image Understanding: A Comprehensive Review