mlsanityData to 7 Oct 2026

An Effective Router for Vision-Language Model Selection

arXiv 2606.08970 ↗First posted 8 Jun 2026cs.AIcited by 0

On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning

Abstract (arXiv)

Vision-language models (VLMs) with varying performance and resource requirements are widely deployed, making it difficult for users to select the most appropriate one among numerous VLM candidates. Existing work reveals the performance paradox phenomenon in language models and focuses on routing methods to solve it. However, developing a router for VLM selection is still a critical yet challenging problem, which primarily faces: 1) lack of specialized data, 2) ineffective feature representation, and 3) rigid model space and costly adaptation. In this paper, we construct a multimodal dataset for VLM selection, containing the outputs of seven mainstream VLMs on 32,626 unique image-text queries. We then propose ARMS, a router for VLM selection. ARMS enhances input signals with VLM profiles, employs a simple but effective architecture to improve representations of queries and VLM capabilities. To improve ARMS' adaptation to new VLMs, we propose two extension training strategies: incremental training and independent training. Experimental results on both in-distribution and out-of-distribution test sets demonstrate the effectiveness of ARMS. In particular, using our training strategy, ARMs (only 800M in size) can adapt to a broader VLM space and defeat commercial models like GPT-4o that are hundreds of times larger in scale. Our code, models, and datasets are available in the anonymous repository.

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 13 of its 29 references that are in the map. First posted on arXiv 8 Jun 2026.

Builds on

13 of 29 references.

Closest ancestors · 1

Same direction, newest first

  1. 2024All Languages Matter: Evaluating LMMs on Culturally Diverse 100 Languages25 Nov

Other references · 12

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  2. 2024The Llama 3 Herd of Modelscited by 14,192
  3. 2019DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lightercited by 4,243
  4. 2024Gemini 1.5: Unlocking multimodal understanding across millions of tokens of contextcited by 3,175
  5. 2020DeBERTa: Decoding-enhanced BERT with Disentangled Attentioncited by 2,157
  6. 2024Hybrid LLM: Cost-Efficient and Quality-Aware Query Routingcited by 354
  7. 2024MME-RealWorld: Could Your Multimodal LLM Challenge High-Resolution Real-World Scenarios that are Difficult for Humans?cited by 284
  8. 2024MMT-Bench: A Comprehensive Multimodal Benchmark for Evaluating Large Vision-Language Models Towards Multitask AGIcited by 192
  9. 2024GraphRouter: A Graph-based Router for LLM Selectionscited by 154
  10. 2023Fly-Swat or Cannon? Cost-Effective Language Model Choice via Meta-Modelingcited by 94
  11. 2024TensorOpera Router: A Multi-Model Router for Efficient LLM Inferencecited by 59
  12. 2025Text-Promptable Propagation for Referring Medical Image Sequence Segmentationcited by 4

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2025Mordal: Automated Pretrained Model Selection for Vision Language Models
  2. 2024What matters when building vision-language models?
  3. 2023LOVM: Language-Only Vision Model Selection
  4. 2024Bridge the Modality and Capability Gaps in Vision-Language Model Selection
  5. 202620/20 Vision Language Models: A Prescription for Better VLMs through Data Curation Alone
  6. 2025Vision-Language Model Selection and Reuse for Downstream Adaptation
  7. 2026One Stone, Three Birds: Self-adaptive Optimal Transport for Multi-VLM Selection, Adaptation, and Ensembling
  8. 2026Empirical Recipes for Efficient and Compact Vision-Language Models