# MACS: Modality-Aware Capacity Scaling for Efficient Multimodal MoE Inference

arXiv: https://arxiv.org/abs/2605.05225 · HTML: https://mlsanity.com/paper/2605.05225/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in MLLM training and visual instruction tuning, one of 16 directions in Multimodal LLMs / vision-language models.
- Builds on 24 of its 35 references that are in the map.
- First posted on arXiv 19 Apr 2026.

## Placed in

- Multimodal LLMs / vision-language models / [MLLM training and visual instruction tuning](https://mlsanity.com/mllm/mllm-training-and-visual-instruction-tuning/)
- Mixture-of-experts / [Multimodal MoE models](https://mlsanity.com/moe/multimodal-moe-models/)

## Builds on (most cited first, 24 of 24)

- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [Improved Baselines with Visual Instruction Tuning](https://mlsanity.com/paper/2310.03744/) (2023)
- [Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity](https://mlsanity.com/paper/2101.03961/) (2021)
- [LLaVA-OneVision: Easy Visual Task Transfer](https://mlsanity.com/paper/2408.03326/) (2024)
- [MMBench: Is Your Multi-modal Model an All-around Player?](https://mlsanity.com/paper/2307.06281/) (2023)
- [Towards VQA Models That Can Read](https://mlsanity.com/paper/1904.08920/) (2019)
- [InternVL3.5: Advancing Open-Source Multimodal Models in Versatility, Reasoning, and Efficiency](https://mlsanity.com/paper/2508.18265/) (2025)
- [ChartQA: A Benchmark for Question Answering about Charts with Visual and Logical Reasoning](https://mlsanity.com/paper/2203.10244/) (2022)
- [Video-MME: The First-Ever Comprehensive Evaluation Benchmark of Multi-modal LLMs in Video Analysis](https://mlsanity.com/paper/2405.21075/) (2024)
- [MM-Vet: Evaluating Large Multimodal Models for Integrated Capabilities](https://mlsanity.com/paper/2308.02490/) (2023)
- [MVBench: A Comprehensive Multi-modal Video Understanding Benchmark](https://mlsanity.com/paper/2311.17005/) (2023)
- [EgoSchema: A Diagnostic Benchmark for Very Long-form Video Language Understanding](https://mlsanity.com/paper/2308.09126/) (2023)
- [LongVideoBench: A Benchmark for Long-context Interleaved Video-Language Understanding](https://mlsanity.com/paper/2407.15754/) (2024)
- [Video-MMMU: Evaluating Knowledge Acquisition from Multi-Discipline Professional Videos](https://mlsanity.com/paper/2501.13826/) (2025)
- [OpenMoE: An Early Effort on Open Mixture-of-Experts Language Models](https://mlsanity.com/paper/2402.01739/) (2024)
- [MoE-Pruner: Pruning Mixture-of-Experts Large Language Model using the Hints from Its Router](https://mlsanity.com/paper/2410.12013/) (2024)
- [Shortcut-connected Expert Parallelism for Accelerating Mixture-of-Experts](https://mlsanity.com/paper/2404.05019/) (2024)
- [Token Activation Map to Visually Explain Multimodal LLMs](https://mlsanity.com/paper/2506.23270/) (2025)
- [STUN: Structured-Then-Unstructured Pruning for Scalable MoE Pruning](https://mlsanity.com/paper/2409.06211/) (2024)
- [LLaMA-MoE v2: Exploring Sparsity of LLaMA from Perspective of Mixture-of-Experts with Post-Training](https://mlsanity.com/paper/2411.15708/) (2024)
- [Capacity-Aware Inference: Mitigating the Straggler Effect in Mixture of Experts](https://mlsanity.com/paper/2503.05066/) (2025)
- [MIT-10M: A Large Scale Parallel Corpus of Multilingual Image Translation](https://mlsanity.com/paper/2412.07147/) (2024)
- [Merge then Realign: Simple and Effective Modality-Incremental Continual Learning for Multimodal LLMs](https://mlsanity.com/paper/2503.07663/) (2025)
- [Evidence Recomposition and Predictive Context Residualization for Visual Attribution in Multimodal Large Language Models](https://mlsanity.com/paper/2509.22415/) (2025)

## Built on by (most cited first, 1 of 1)

- [DecoMoE: Decoupling Visual Propagation and Expert Computation for Efficient Multimodal MoE Inference](https://mlsanity.com/paper/2609.38823/) (2026)

## Abstract (arXiv)

Mixture-of-Experts Multimodal Large Language Models (MoE MLLMs) suffer from a significant efficiency bottleneck during Expert Parallelism (EP) inference due to the straggler effect. This issue is worsened in the multimodal context, as existing token-count-based load balancing methods fail to address two unique challenges: (1) Information Heterogeneity, where numerous redundant visual tokens are treated equally to semantically critical ones, and (2) Modality Dynamics, where varying visual to text ratios across tasks lead to resource misallocation. To address these challenges, we propose MACS (Modality-Aware Capacity Scaling), a training-free inference framework. Specifically, MACS introduces an Entropy-Weighted Load mechanism to quantify the semantic value of visual tokens, addressing information heterogeneity. Additionally, the Dynamic Modality-Adaptive Capacity mechanism allocates expert resources based on the real-time modal composition of the input. Extensive experiments demonstrate that MACS significantly outperforms existing methods on various multimodal benchmarks, providing a novel and robust solution for the efficient deployment of MoE MLLMs in EP inference.