MHR-Router: Multimodal Heterogeneous Retrieval-Enhanced Adaptive Routing for VLM Inference
On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning also fits: Document understanding and multimodal RAG
Abstract (arXiv)
Vision-language models (VLMs) exhibit complementary capabilities, yet selecting the most suitable expert for a given image-question query remains challenging due to heterogeneous multimodal inputs and varying expert strengths. Existing routing approaches rely on text-based cues or static model profiles, failing to capture fine-grained visual evidence and expert-specific failure patterns. To address this, we propose MHR-Router, a multimodal heterogeneous retrieval-enhanced adaptive routing framework for VLM inference. It accurately dispatches each multimodal query to the most suitable VLM expert via local retrieval evidence. MHR-Router consists of two stages: (1) Multimodal Routing Representation, which fuses frozen multimodal features into a comprehensive query representation; and (2) Heterogeneous Retrieval-Enhanced Adaptive Routing, which retrieves expert-specific success and failure memories as local evidence and employs a shared scorer to adaptively select the optimal expert. Experiments show that MHR-Router effectively achieves heterogeneous modality routing and consistently improves accuracy over traditional baselines.
Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 22 of its 28 references that are in the map. First posted on arXiv 9 Oct 2026.
Builds on
22 of 28 references.
Closest ancestors · 1
Same direction, newest first
Other references · 21
- 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
- 2025Qwen2.5-VL Technical Reportcited by 5,587
- 2024Qwen2-VL: Enhancing Vision-Language Model's Perception of the World at Any Resolutioncited by 4,234
- 2023Sigmoid Loss for Language Image Pre-Trainingcited by 3,109
- 2024LLaVA-OneVision: Easy Visual Task Transfercited by 2,719
- 2022Learn to Explain: Multimodal Reasoning via Thought Chains for Science Question Answeringcited by 1,919
- 2019Towards VQA Models That Can Readcited by 1,718
- 2024Expanding Performance Boundaries of Open-Source Multimodal Models with Model, Data, and Test-Time Scalingcited by 1,599
- 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoningcited by 1,479
- 2016CLEVR: A Diagnostic Dataset for Compositional Language and Elementary Visual Reasoningcited by 1,457
- 2023FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performancecited by 853
- 2022A-OKVQA: A Benchmark for Visual Question Answering using World Knowledgecited by 710
9 more
- 2024RouteLLM: Learning to Route LLMs with Preference Datacited by 682
- 2025ICL-Router: In-Context Learned Model Representations for LLM Routingcited by 28
- 2026Efficient Inference for Large Vision-Language Models: Bottlenecks, Techniques, and Prospectscited by 9
- 2025VL-RouterBench: A Benchmark for Vision-Language Model Routingcited by 8
- 2025Adaptive Guidance Semantically Enhanced via Multimodal LLM for Edge-Cloud Object Detectioncited by 6
- 2025MoA-Off: Adaptive Heterogeneous Modality-Aware Offloading with Edge-Cloud Collaboration for Efficient Multimodal LLM Inferencecited by 5
- 2025SAEC: Scene-Aware Enhanced Edge-Cloud Collaborative Industrial Vision Inspection with Multimodal LLMcited by 3
- 2026INAR-VL: Input-Aware Routing for Edge-Cloud Vision-Language Inferencecited by 2
- 2026Pro-Router: Token-Aware Progressive Model Routing with Adaptive Edge-Cloud Collaboration for Efficient Multimodal LLM Inferencecited by 1
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2024RoRA-VLM: Robust Retrieval-Augmented Vision Language Models
- 2026MMR-Bench: A Comprehensive Benchmark for Multimodal LLM Routing
- 2025mRAG: Elucidating the Design Space of Multi-modal Retrieval-Augmented Generation
- 2024SURf: Teaching Large Vision-Language Models to Selectively Utilize Retrieved Information
- 2026LatentRouter: Can We Choose the Right Multimodal Model Before Seeing Its Answer?
- 2026Before Thinking, Learn to Decide: Proactive Routing for Efficient Visual Reasoning
- 2024RA-BLIP: Multimodal Adaptive Retrieval-Augmented Bootstrapping Language-Image Pre-training
- 2026Towards Efficient Large Vision-Language Models: A Comprehensive Survey on Inference Strategies