mlsanityData to 14 Oct 2026

BTW: A Non-Parametric Variance Stabilization Framework for Multimodal Model Integration

arXiv 2508.18551 ↗First posted 25 Aug 2025cs.LGcited by 0

On the map: Mixture-of-experts → Multimodal MoE models

Abstract (arXiv)

Mixture-of-Experts (MoE) models have become increasingly powerful in multimodal learning by enabling modular specialization across modalities. However, their effectiveness remains unclear when additional modalities introduce more noise than complementary information. Existing approaches, such as the Partial Information Decomposition, struggle to scale beyond two modalities and lack the resolution needed for instance-level control. We propose Beyond Two-modality Weighting (BTW), a bi-level, non-parametric weighting framework that combines instance-level Kullback-Leibler (KL) divergence and modality-level mutual information (MI) to dynamically adjust modality importance during training. Our method does not require additional parameters and can be applied to an arbitrary number of modalities. Specifically, BTW computes per-example KL weights by measuring the divergence between each unimodal and the current multimodal prediction, and modality-wide MI weights by estimating global alignment between unimodal and multimodal outputs. Extensive experiments on sentiment regression and clinical classification demonstrate that our method significantly improves regression performance and multiclass classification accuracy.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Placed in Multimodal MoE models, one of 12 directions in Mixture-of-experts. Builds on 20 of its 35 references that are in the map. First posted on arXiv 25 Aug 2025.

Builds on

20 of 35 references.

Closest ancestors · 5

Same direction, newest first

  1. 2024Flex-MoE: Modeling Arbitrary Modality Combination via the Flexible Mixture-of-Experts10 Oct
  2. 2024MoMa: Efficient Early-Fusion Pre-training with Mixture of Modality-Aware Experts31 Jul
  3. 2024Uni-MoE: Scaling Unified Multimodal LLMs with Mixture of Experts18 May
  4. 2023Alternating Gradient Descent and Mixture-of-Experts for Integrated Multimodal Perception10 May
  5. 2022Multimodal Contrastive Learning with LIMoE: the Language-Image Mixture of Experts6 Jun

Other references · 15

  1. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,509
  2. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsitycited by 3,073
  3. 2019Multimodal Transformer for Unaligned Multimodal Language Sequencescited by 645
  4. 2016MOSI: Multimodal Corpus of Sentiment Intensity and Subjectivity Analysis in Online Opinion Videoscited by 194
  5. 2021Improving Multimodal Fusion with Hierarchical Mutual Information Maximization for Multimodal Sentiment Analysiscited by 131
  6. 2022ERNIE-ViLG 2.0: Improving Text-to-Image Diffusion Model with Knowledge-Enhanced Mixture-of-Denoising-Expertscited by 112
  7. 2023Quantifying & Modeling Multimodal Interactions: An Information Decomposition Frameworkcited by 83
  8. 2024FuseMoE: Mixture-of-Experts Transformers for Fleximodal Fusioncited by 68
  9. 2023Multi-modal Gated Mixture of Local-to-Global Experts for Dynamic Image Fusioncited by 54
  10. 2021Clinical Outcome Prediction from Admission Notes using Self-Supervised Knowledge Integrationcited by 44
  11. 2022DIME: Fine-grained Interpretations of Multimodal Models via Disentangled Local Explanationscited by 34
  12. 2022One Model, Multiple Modalities: A Sparsely Activated Approach for Text, Sound, Image, Video and Codecited by 26
3 more
  1. 2024Balancing Multimodal Training Through Game-Theoretic Regularizationcited by 15
  2. 2023Dynamic Multimodal Information Bottleneck for Multimodality Classificationcited by 14
  3. 2023Denoising Bottleneck with Mutual Information Maximization for Video Multimodal Fusioncited by 4

Built on by

Not cited yet.

Similar, not cited

Close in wording, no citation either way

  1. 2026C2MOE: Consistency and Complementarity-guided Mixture of Experts for Incomplete Multimodal Emotion Learning
  2. 2026AlignMamba-2: Enhancing Multimodal Fusion and Sentiment Analysis with Modality-Aware Mamba
  3. 2025Multimodal Mixture of Low-Rank Experts for Sentiment Analysis and Emotion Recognition
  4. 2026Tackling Multimodal Learning Challenges with Mixture-of-Expert: A Survey
  5. 2026InstMoE: Adaptive Multimodal Routing with Specialized Experts
  6. 2025Confidence-Aware Self-Distillation for Multimodal Sentiment Analysis with Incomplete Modalities
  7. 2026Explicit Representation Alignment for Multimodal Sentiment Analysis
  8. 2025Multi-Modality Collaborative Learning for Sentiment Analysis