mlsanityData to 7 Oct 2026

MMR-Bench: A Comprehensive Benchmark for Multimodal LLM Routing

arXiv 2601.17814 ↗First posted 25 Jan 2026cs.AIcited by 13

On the map: Multimodal LLMs / vision-language models → MLLM training and visual instruction tuning

Abstract (arXiv)

Multimodal large language models (MLLMs) have advanced rapidly, yet heterogeneity in architecture, alignment strategies, and efficiency means that no single model is uniformly superior across tasks. In practical deployments, workloads span lightweight OCR to complex multimodal reasoning; using one MLLM for all queries either over-provisions compute on easy instances or sacrifices accuracy on hard ones. Query-level model selection (routing) addresses this tension, but extending routing from text-only LLMs to MLLMs is nontrivial due to modality fusion, wide variation in computational cost across models, and the absence of a standardized, budget-aware evaluation. We present MMR-Bench, a unified benchmark that isolates the multimodal routing problem and enables comparison under fixed candidate sets and cost models. MMR-Bench provides (i) a controlled environment with modality-aware inputs and variable compute budgets, (ii) a broad suite of vision-language tasks covering OCR, general VQA, and multimodal math reasoning, and (iii) strong single-model reference, oracle upper bounds, and representative routing policies. Using MMR-Bench, we show that incorporating multimodal signals improves routing quality. Empirically, these cues improve the cost-accuracy frontier and enable the routed system to exceed the strongest single model's accuracy at roughly 33% of its cost. Furthermore, policies trained on a subset of models and tasks generalize zero-shot to new datasets and text-only benchmarks without retuning, establishing MMR-Bench as a foundation for studying adaptive multimodal model selection and efficient MLLM deployment. The code will be available at: https://github.com/Hunter-Wrynn/MMR-Bench.

Key facts. Cited by 13 papers in the map (data through 7 Oct 2026). Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 33 of its 37 references that are in the map. First posted on arXiv 25 Jan 2026.

Builds on

33 of 37 references.

Closest ancestors · 5

Same direction, newest first

  1. 2024LLaVA-NeXT-Interleave: Tackling Multi-image, Video, and 3D in Large Multimodal Models10 Jul
  2. 2024MoE-LLaVA: Mixture of Experts for Large Vision-Language Models29 Jan
  3. 2023Multimodal Large Language Models: A Survey22 Nov
  4. 2023SEED-Bench: Benchmarking Multimodal LLMs with Generative Comprehension30 Jul
  5. 2023A Survey on Multimodal Large Language Models23 Jun

Other references · 28

  1. 2023GPT-4 Technical Reportcited by 18,132
  2. 2023Visual Instruction Tuningcited by 7,792
  3. 2025Qwen2.5-VL Technical Reportcited by 5,418
  4. 2023MMMU: A Massive Multi-discipline Multimodal Understanding and Reasoning Benchmark for Expert AGIcited by 2,044
  5. 2025Gemma 3 Technical Reportcited by 1,686
  6. 2023MathVista: Evaluating Mathematical Reasoning of Foundation Models in Visual Contextscited by 1,582
  7. 2025InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiencycited by 1,419
  8. 2024Are We on the Right Way for Evaluating Large Vision-Language Models?cited by 893
  9. 2023FrugalGPT: How to Use Large Language Models While Reducing Cost and Improving Performancecited by 791
  10. 2024MathVerse: Does Your Multi-modal LLM Truly See the Diagrams in Visual Math Problems?cited by 573
  11. 2022Mind the Gap: Understanding the Modality Gap in Multi-modal Contrastive Representation Learningcited by 553
  12. 2023LLM-Blender: Ensembling Large Language Models with Pairwise Ranking and Generative Fusioncited by 550
16 more
  1. 2024Measuring Multimodal Mathematical Reasoning with MATH-Vision Datasetcited by 516
  2. 2023OCRBench: On the Hidden Mystery of OCR in Large Multimodal Modelscited by 466
  3. 2024RouterBench: A Benchmark for Multi-LLM Routing Systemcited by 222
  4. 2023Large Language Model Routing with Benchmark Datasetscited by 166
  5. 2024GraphRouter: A Graph-based Router for LLM Selectionscited by 154
  6. 2023AutoMix: Automatically Mixing Language Modelscited by 116
  7. 2014A Multi-World Approach to Question Answering about Real-World Scenes based on Uncertain Inputcited by 112
  8. 2025Ovis2.5 Technical Reportcited by 103
  9. 2023Fly-Swat or Cannon? Cost-Effective Language Model Choice via Meta-Modelingcited by 94
  10. 2024RouterDC: Query-Based Router by Dual Contrastive Learning for Assembling Large Language Modelscited by 84
  11. 2025RouterEval: A Comprehensive Benchmark for Routing LLMs to Explore Model-level Scaling Up in LLMscited by 58
  12. 2024EmbedLLM: Learning Compact Representations of Large Language Modelscited by 46
  13. 2025IRT-Router: Effective and Interpretable Multi-LLM Routing via Item Response Theorycited by 40
  14. 2025Capability Instruction Tuning: A New Paradigm for Dynamic LLM Routingcited by 25
  15. 2025RouterArena: An Open Platform for Comprehensive Comparison of LLM Routerscited by 19
  16. 2025Quality-of-Service Aware LLM Routing for Edge Computing with Multiple Expertscited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 13.

13 papers.

Sort:

By area:

  1. 2026Dynamic Model Routing and Cascading for Efficient LLM Inference: A Surveycited by 16
  2. 2026When Routing Collapses: On the Degenerate Convergence of LLM Routerscited by 10
  3. 2026VT-Bench: A Unified Benchmark for Visual-Tabular Multi-Modal Learningcited by 3
  4. 2026From Sampled Outcomes to Capability Distributions: Rethinking Supervision for LLM Routingcited by 3
  5. 2026RouteJudge: An Open Platform for Reproducible and Preference-Aware LLM Routingcited by 1
  6. 2026Dynamo: Dynamic Skill-Tool Evolution for Vision-Language Agentscited by 1
  7. 2026Pre-Inference Routing for Cost-Efficient Document Field Extractioncited by 1
  8. 2026LLMRouter: Unified Infrastructure for Developing, Evaluating, and Deploying LLM Routerscited by 1
  9. 2026ReLope: KL-Regularized LoRA Probes for Multimodal LLM Routingcited by 0
  10. 2026LatentRouter: Can We Choose the Right Multimodal Model Before Seeing Its Answer?cited by 0
  11. 2026DIRECT: When and Where Should You Allocate Test-Time Compute in Embodied Planners?cited by 0
  12. 2026SCOPE-Router: Cost-Aware Open-Set VLM Routing for Execution-Oriented Taskscited by 0
1 more
  1. 2026Routing Should Pay for Itself: Sparse Supervision for Economical LLM Routingcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2024MM-LLMs: Recent Advances in MultiModal Large Language Models
  2. 2026MMR-Life: Piecing Together Real-life Scenes for Multimodal Multi-image Reasoning
  3. 2025Benchmarking Large and Small MLLMs
  4. 2025Evaluating MLLMs with Multimodal Multi-image Reasoning Benchmark
  5. 2024MMT-Bench: A Comprehensive Multimodal Benchmark for Evaluating Large Vision-Language Models Towards Multitask AGI
  6. 2024MM1.5: Methods, Analysis & Insights from Multimodal LLM Fine-tuning
  7. 2024MM-InstructEval: Zero-Shot Evaluation of (Multimodal) Large Language Models on Multimodal Reasoning Tasks
  8. 2024A Survey on Multimodal Benchmarks: In the Era of Large AI Models