# Dynamic Modality Scheduling for Multimodal Large Models via Confidence, Uncertainty, and Semantic Consistency

arXiv: https://arxiv.org/abs/2506.12724 · HTML: https://mlsanity.com/paper/2506.12724/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 16 of its 18 references that are in the map.
- First posted on arXiv 15 Jun 2025.

## Builds on (most cited first, 16 of 16)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [Visual Instruction Tuning](https://mlsanity.com/paper/2304.08485/) (2023)
- [BLIP-2: Bootstrapping Language-Image Pre-training with Frozen Image Encoders and Large Language Models](https://mlsanity.com/paper/2301.12597/) (2023)
- [Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learning](https://mlsanity.com/paper/1506.02142/) (2015)
- [Flamingo: a Visual Language Model for Few-Shot Learning](https://mlsanity.com/paper/2204.14198/) (2022)
- [ViLBERT: Pretraining Task-Agnostic Visiolinguistic Representations for Vision-and-Language Tasks](https://mlsanity.com/paper/1908.02265/) (2019)
- [GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding](https://mlsanity.com/paper/2006.16668/) (2020)
- [PaLI: A Jointly-Scaled Multilingual Language-Image Model](https://mlsanity.com/paper/2209.06794/) (2022)
- [Person Re-identification Method Based on Color Attack and Joint Defence](https://mlsanity.com/paper/2111.09571/) (2021)
- [Cross-Modality Perturbation Synergy Attack for Person Re-identification](https://mlsanity.com/paper/2401.10090/) (2024)
- [Eliminate Deviation with Deviation for Data Augmentation and a General Multi-modal Data Learning Method](https://mlsanity.com/paper/2101.08533/) (2021)
- [Exploring Color Invariance through Image-Level Ensemble Learning](https://mlsanity.com/paper/2401.10512/) (2024)
- [Beyond Augmentation: Empowering Model Robustness under Extreme Capture Environments](https://mlsanity.com/paper/2407.13640/) (2024)
- [Beyond Dropout: Robust Convolutional Neural Networks Based on Local Feature Masking](https://mlsanity.com/paper/2407.13646/) (2024)
- [Adversarial Learning for Neural PDE Solvers with Sparse Data](https://mlsanity.com/paper/2409.02431/) (2024)
- [Cross-Modality Attack Boosted by Gradient-Evolutionary Multiform Optimization](https://mlsanity.com/paper/2409.17977/) (2024)

## Abstract (arXiv)

Multimodal Large Models (MLLMs) have achieved remarkable progress in vision-language understanding and generation tasks. However, existing MLLMs typically rely on static modality fusion strategies, which treat all modalities equally regardless of their instance-level reliability or semantic contribution. This often leads to suboptimal performance, especially in scenarios with noisy, missing, or misaligned modalities. In this paper, we propose Dynamic Modality Scheduling (DMS), a novel framework that adaptively adjusts the contribution of each modality at a per-sample level. DMS evaluates each modality based on three key factors: (1) textitconfidence, estimated from predictive entropy; (2) textituncertainty, obtained via Monte Carlo dropout; and (3) textitsemantic consistency, computed through inter-modal similarity. These signals are combined through a learnable or rule-based scheduler to generate soft modality weights used in downstream fusion.To ensure stable training, we further introduce a textitModality Weight Consistency Loss, which regularizes the fused representation to stay close to unimodal embeddings proportionally to their assigned weights. Our method is model-agnostic and can be integrated into existing MLLMs such as BLIP-2 and LLaVA. Experimental results on VQA, image-text retrieval, and captioning tasks show that DMS significantly improves both clean and robust performance, especially under modality corruption or dropout conditions. This work provides a general and effective mechanism to enable instance-aware and robustness-enhanced multimodal modeling.