mlsanityData to 14 Oct 2026

Beyond Classification: Dynamic Adapter Routing for Continual Multimodal Retrieval

arXiv 2605.31229 ↗First posted 29 May 2026cs.CV cs.AIcited by 1

On the map: Multimodal LLMs / vision-language models → Document understanding and multimodal RAG

Abstract (arXiv)

While retrieval is a core function of vision-language models, continually updating these models for retrieval tasks remains critically underexplored. Existing work often approaches continual retrieval through the lens of class-incremental learning (CIL), evaluating both standard CIL methods and retrieval-oriented adaptations in settings that may not fully capture the retrieval-specific dynamics. To address this, we introduce a new, principled evaluation framework for continual multimodal retrieval (CMR) spanning diverse visual domains, and systematically evaluate common approaches within this setting. Our empirical analysis shows that standard CIL methods fail to yield meaningful gains in our more challenging scenario. Therefore, we propose Dynamic Adapter Routing (DAR), a novel approach based on adapters selected through prototype-based routing and combined via model merging.DAR achieves superior performance over the previous baselines and demonstrates strong generalization under out-of-distribution evaluation. Our results highlights the unique challenges of CMR and encourages further research in this direction.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Document understanding and multimodal RAG, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 25 of its 42 references that are in the map. First posted on arXiv 29 May 2026.

Builds on

25 of 42 references.

References · 25

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  3. 2016Overcoming catastrophic forgetting in neural networkscited by 4,787
  4. 2022Editing Models with Task Arithmeticcited by 1,120
  5. 2021Learning to Prompt for Continual Learningcited by 944
  6. 2023TIES-Merging: Resolving Interference When Merging Modelscited by 818
  7. 2018nocaps: novel object captioning at scalecited by 444
  8. 2020AdapterHub: A Framework for Adapting Transformerscited by 444
  9. 2024Boosting Continual Learning of Vision-Language Models via Mixture-of-Experts Adapterscited by 199
  10. 2023Preventing Zero-Shot Transfer Degradation in Continual Learning of Vision-Language Modelscited by 112
  11. 2025No Task Left Behind: Isotropic Model Merging with Common and Task-Specific Subspacescited by 96
  12. 2024MagMax: Leveraging Model Merging for Seamless Continual Learningcited by 67
13 more
  1. 2024ROCOv2: Radiology Objects in COntext Version 2, an Updated Multimodal Image Datasetcited by 54
  2. 2022FS-COCO: Towards Understanding of Freehand Sketches of Common Objects in Contextcited by 49
  3. 2024Class-Incremental Learning with CLIP: Adaptive Representation Adjustment and Parameter Fusioncited by 47
  4. 2023Continual Vision-Language Representation Learning with Off-Diagonal Informationcited by 40
  5. 2023TiC-CLIP: Continual Training of CLIP Modelscited by 38
  6. 2025Accurate and Efficient Low-Rank Model Merging in Core Spacecited by 29
  7. 2024CLAP4CLIP: Continual Learning with Probabilistic Finetuning for Vision-Language Modelscited by 25
  8. 2022ConStruct-VL: Data-Free Continual Structured VL Concepts Learningcited by 20
  9. 2024Semantic Residual Prompts for Continual Learningcited by 19
  10. 2024CoLeCLIP: Open-Domain Continual Learning via Joint Task Prompt and Vocabulary Learningcited by 15
  11. 2020On Class Orderings for Incremental Learningcited by 10
  12. 2021Continual learning in cross-modal retrievalcited by 10
  13. 2025Continual Learning on CLIP via Incremental Prompt Tuning with Intrinsic Textual Anchorscited by 6

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026AlphaWiSE: Adaptive Weight Interpolation for Continual Multimodal Representation Learningcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2026Indexing Multimodal Language Models for Large-scale Image Retrieval
  2. 2025Recurrence Meets Transformers for Universal Multimodal Retrieval
  3. 2025MLLM-CL: Continual Learning for Multimodal Large Language Models
  4. 2024Understanding Retrieval-Augmented Task Adaptation for Vision-Language Models
  5. 2026Beyond Global Similarity: Towards Fine-Grained, Multi-Condition Multimodal Retrieval
  6. 2024Prompt-Aware Adapter: Towards Learning Adaptive Visual Tokens for Multimodal Large Language Models
  7. 2026MHR-Router: Multimodal Heterogeneous Retrieval-Enhanced Adaptive Routing for VLM Inference
  8. 2025mRAG: Elucidating the Design Space of Multi-modal Retrieval-Augmented Generation