OmniMoE-VL: A Sparse Vision-Language Model with Coupled Visual-Depth Routing
On the map: Multimodal LLMs / vision-language models → VLM benchmarks and visual reasoning also fits: Visual token pruning and efficiency
Abstract (arXiv)
Vision-language models (VLMs) increasingly use sparse mixture-of-experts (MoE) to scale language-side computation, yet visual information is typically routed only after passing through a fixed cross-modal interface. This leaves an important decision unresolved: which intermediate visual representations should be exposed to language computation for a given question? We introduce OmniMoE-VL, a sparse VLM with a coupled visual-depth routed projector. For each image-prompt pair, the projector selects a sparse set of intermediate visual depths and reuses the resulting global preference to guide both local patch fusion and dynamic visual injection into the language model. This design enables question-dependent visual access while preserving the native visual-token sequence, and complements token-level expert routing in the vision and language stacks. Across eight image-based benchmarks, OmniMoE-VL achieves an average score of 85.9 with 28B total and 9B activated parameters. Controlled comparisons show that the routed visual interface provides the dominant architectural gain, while matched route and component controls, same-image route analysis, and route interventions further support the value of coupling and question-conditioned visual access.
Key facts. No citing papers in the map yet (data through 7 Oct 2026). Placed in VLM benchmarks and visual reasoning, one of 16 directions in Multimodal LLMs / vision-language models. Builds on 15 of its 22 references that are in the map. First posted on arXiv 26 Sep 2026.
Builds on
15 of 22 references.
References · 15
- 2021LoRA: Low-Rank Adaptation of Large Language Modelscited by 14,084
- 2023BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Modelscited by 6,069
- 2023MMBench: Is Your Multi-modal Model an All-around Player?cited by 2,105
- 2019Towards VQA Models That Can Readcited by 1,696
- 2023Evaluating Object Hallucination in Large Vision-Language Modelscited by 1,696
- 2022ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoningcited by 1,449
- 2020DocVQA: A Dataset for VQA on Document Imagescited by 1,288
- 2023MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilitiescited by 1,221
- 2023OCRBench: On the Hidden Mystery of OCR in Large Multimodal Modelscited by 466
- 2025GLM-4.5V and GLM-4.1V-Thinking: Towards Versatile Multimodal Reasoning with Scalable Reinforcement Learningcited by 439
- 2024MoE-LLaVA: Mixture of Experts for Large Vision-Language Modelscited by 298
- 2024DeepStack: Deeply Stacking Visual Tokens is Surprisingly Simple and Effective for LMMscited by 72
3 more
Built on by
Not cited yet.
Similar, not cited
Close in wording, no citation either way
- 2025A Survey of State of the Art Large Vision Language Models: Alignment, Benchmark, Evaluations and Challenges
- 2024Exploring the Frontier of Vision-Language Models: A Survey of Current Methodologies and Future Directions
- 2024Prismatic VLMs: Investigating the Design Space of Visually-Conditioned Language Models
- 2025AdaptVision: Efficient Vision-Language Models via Adaptive Visual Acquisition
- 2024FiVL: A Framework for Improved Vision-Language Alignment through the Lens of Training, Evaluation and Explainability
- 2026Mixture of Layers: Dynamic Layer Routing for Visual Reasoning
- 2026Searching for Task-Specific Vision Paths: Evolutionary Block Pruning Across Vision-Language Models
- 2023Large Language Models are Visual Reasoning Coordinators