mlsanityData to 7 Oct 2026

FuseMoE: Mixture-of-Experts Transformers for Fleximodal Fusion

arXiv 2402.03226 ↗First posted 5 Feb 2024cs.LGcited by 67

On the map: Mixture-of-experts → Expert routing strategies

Abstract (arXiv)

As machine learning models in critical fields increasingly grapple with multimodal data, they face the dual challenges of handling a wide array of modalities, often incomplete due to missing elements, and the temporal irregularity and sparsity of collected samples. Successfully leveraging this complex data, while overcoming the scarcity of high-quality training samples, is key to improving these models' predictive performance. We introduce ``FuseMoE'', a mixture-of-experts framework incorporated with an innovative gating function. Designed to integrate a diverse number of modalities, FuseMoE is effective in managing scenarios with missing modalities and irregularly sampled data trajectories. Theoretically, our unique gating function contributes to enhanced convergence rates, leading to better performance in multiple downstream tasks. The practical utility of FuseMoE in the real world is validated by a diverse set of challenging prediction tasks.

Key facts. Cited by 67 papers in the map (data through 7 Oct 2026). Placed in Expert routing strategies, one of 12 directions in Mixture-of-experts. Builds on 57 of its 109 references that are in the map. First posted on arXiv 5 Feb 2024.

Builds on

57 of 109 references.

Closest ancestors · 5

Same direction, newest first

  1. 2024CompeteSMoE -- Effective Training of Sparse Mixture of Experts via Competition4 Feb
  2. 2022On the Representation Collapse of Sparse Mixture of Experts20 Apr
  3. 2022Mixture-of-Experts with Expert Choice Routing18 Feb
  4. 2021EvoMoE: An Evolutional Mixture-of-Experts Training Framework via Dense-To-Sparse Gate29 Dec
  5. 2021Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity11 Jan

Other references · 52

  1. 2017Attention Is All You Needcited by 59,932
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  3. 2019Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformercited by 14,491
  4. 2019EfficientNet: Rethinking Model Scaling for Convolutional Neural Networkscited by 6,415
  5. 2016XGBoost: A Scalable Tree Boosting Systemcited by 5,120
  6. 2017Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layercited by 3,413
  7. 2019CheXpert: A Large Chest Radiograph Dataset with Uncertainty Labels and Expert Comparisoncited by 1,410
  8. 2019Publicly Available Clinical BERT Embeddingscited by 995
  9. 2014Deep Visual-Semantic Alignments for Generating Image Descriptionscited by 966
  10. 2021Scaling Vision with Sparse Mixture of Expertscited by 740
  11. 2019Multimodal Transformer for Unaligned Multimodal Language Sequencescited by 637
  12. 2016Recurrent Neural Networks for Multivariate Time Series with Missing Valuescited by 548
38 more
  1. 2017Tensor Fusion Network for Multimodal Sentiment Analysiscited by 429
  2. 2017Multitask learning and benchmarking with clinical time series datacited by 287
  3. 2018Efficient Low-rank Multimodal Fusion with Modality-Specific Factorscited by 268
  4. 2023Multimodal Deep Learningcited by 257
  5. 2023From Sparse to Soft Mixtures of Expertscited by 245
  6. 2021Multi-Time Attention Networks for Irregularly Sampled Time Seriescited by 234
  7. 2019Time2Vec: Learning a Vector Representation of Timecited by 208
  8. 2022Multimodal Contrastive Learning with LIMoE: the Language-Image Mixture of Expertscited by 202
  9. 2015DenseCap: Fully Convolutional Localization Networks for Dense Captioningcited by 176
  10. 2019Set Functions for Time Seriescited by 137
  11. 2021TorchXRayVision: A library of chest X-ray datasets and modelscited by 131
  12. 2021Improving Multimodal Fusion with Hierarchical Mutual Information Maximization for Multimodal Sentiment Analysiscited by 130
  13. 2021Graph-Guided Network for Irregularly Sampled Multivariate Time Seriescited by 110
  14. 2019Hierarchical Transformers for Long Document Classificationcited by 103
  15. 2021Self-Supervised Transformer for Sparse and Irregularly Sampled Multivariate Clinical Time-Seriescited by 96
  16. 2018Multi-attention Recurrent Network for Human Communication Comprehensioncited by 89
  17. 2019Interpolation-Prediction Networks for Irregularly Sampled Time Seriescited by 89
  18. 2023Scaling Vision-Language Models with Sparse Mixture of Expertscited by 88
  19. 2022Sparse Mixture-of-Experts are Domain Generalizable Learnerscited by 85
  20. 2020A review: Deep learning for medical image segmentation using multi-modality fusioncited by 84
  21. 2015Are You Talking to a Machine? Dataset and Methods for Multilingual Image Question Answeringcited by 80
  22. 2022Integrated multimodal artificial intelligence framework for healthcare applicationscited by 69
  23. 2022Clinical-Longformer and Clinical-BigBird: Transformers for long clinical sequencescited by 67
  24. 2023Multi-modal Gated Mixture of Local-to-Global Experts for Dynamic Image Fusioncited by 54
  25. 2021Bi-Bimodal Modality Fusion for Correlation-Controlled Multimodal Sentiment Analysiscited by 45
  26. 2021Product1M: Towards Weakly Supervised Instance-Level Product Retrieval via Cross-modal Pretrainingcited by 42
  27. 2020Generalized Entropy Regularization or: There's Nothing Special about Label Smoothingcited by 40
  28. 2018Multimodal Sentiment Analysis using Hierarchical Fusion with Context Modelingcited by 39
  29. 2019Convergence Rates for Gaussian Mixtures of Expertscited by 39
  30. 2019Using Clinical Notes with Time Series Data for ICU Managementcited by 38
  31. 2022Improving Medical Predictions by Irregular Multimodal Electronic Health Records Modelingcited by 33
  32. 2023Demystifying Softmax Gating Function in Gaussian Mixture of Expertscited by 33
  33. 2023Brainformers: Trading Simplicity for Efficiencycited by 31
  34. 2023Statistical Perspective of Top-K Sparse Softmax Gating Mixture of Expertscited by 28
  35. 2021How to Leverage Multimodal EHR Data for Better Medical Predictions?cited by 26
  36. 2023A Survey of Multimodal Information Fusion for Smart Healthcare: Mapping the Journey from Data to Wisdomcited by 25
  37. 2023A General Theory for Softmax Gating Multinomial Logistic Mixture of Expertscited by 23
  38. 2022Refined Convergence Rates for Maximum Likelihood Estimation under Finite Mixture Modelscited by 22

50 most cited of 52 All 52 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 6; 2025: 25; 2026: 36.

67 papers.

Sort:

By area:

  1. 2024Flex-MoE: Modeling Arbitrary Modality Combination via the Flexible Mixture-of-Expertscited by 69
  2. 2024Sigmoid Gating is More Sample Efficient than Softmax Gating in Mixture of Expertscited by 23
  3. 2025Mixture of Experts (MoE): A Big Data Perspectivecited by 17
  4. 2024Statistical Advantages of Perturbing Cosine Router in Mixture of Expertscited by 16
  5. 2025What You Have is What You Track: Adaptive and Robust Multimodal Trackingcited by 15
  6. 2025MAESTRO : Adaptive Sparse Attention and Robust Learning for Multimodal Dynamic Time Seriescited by 9
  7. 2026Why AI systems don't learn and what to do about it: Lessons on autonomous learning from cognitive sciencecited by 8
  8. 2025MoMA: A Mixture-of-Multimodal-Agents Architecture for Enhancing Clinical Prediction Modellingcited by 7
  9. 2024LIBMoE: A Library for comprehensive benchmarking Mixture of Experts in Large Language Modelscited by 6
  10. 2025RepLoRA: Reparameterizing Low-Rank Adaptation via the Perspective of Mixture of Expertscited by 6
  11. 2025NEXT: Multi-Grained Mixture of Experts via Text-Modulation for Multi-Modal Object Re-Identificationcited by 5
  12. 2025A Knowledge-Driven Diffusion Policy for End-to-End Autonomous Driving Based on Expert Routingcited by 5
38 more
  1. 2025On Zero-Initialized Attention: Optimal Prompt and Gating Factor Estimationcited by 4
  2. 2025MoIIE: Mixture of Intra- and Inter-Modality Experts for Large Vision Language Modelscited by 4
  3. 2025Massively Multimodal Foundation Models: A Framework for Capturing Interactions with Specialized Mixture-of-Expertscited by 4
  4. 2025Convergence Rates for Softmax Gating Mixture of Expertscited by 3
  5. 2025Rethinking Gating Mechanism in Sparse MoE: Handling Arbitrary Modality Inputs with Confidence-Guided Gatecited by 3
  6. 2026Seg-MoE: Multi-Resolution Segment-wise Mixture-of-Experts for Time Series Forecasting Transformerscited by 3
  7. 2024On Parameter Estimation in Deviated Gaussian Mixture of Expertscited by 2
  8. 2024Multimodal Lego: Model Merging and Fine-Tuning Across Topologies and Modalities in Biomedicinecited by 2
  9. 2025MINT: Multimodal Instruction Tuning with Multimodal Interaction Groupingcited by 2
  10. 2025Rethinking Efficient Mixture-of-Experts for Remote Sensing Modality-Missing Classificationcited by 2
  11. 2026MILM: Large Language Models for Multimodal Irregular Time Series with Informative Samplingcited by 2
  12. 2025On Minimax Estimation of Parameters in Softmax-Contaminated Mixture of Expertscited by 1
  13. 2025Hypernetwork-Driven Low-Rank Adaptation Across Attention Headscited by 1
  14. 2025OmniField: Conditioned Neural Fields for Robust Multimodal Spatiotemporal Learningcited by 1
  15. 2026Multi-Modal Time Series Prediction via Mixture of Modulated Expertscited by 1
  16. 2026MissMAC-Bench: Building Solid Benchmark for Missing Modality Issue in Robust Multimodal Affective Computingcited by 1
  17. 2026REMIND: Rethinking Medical High-Modality Learning under Missingness--A Long-Tailed Distribution Perspectivecited by 1
  18. 2026Local Precise Refinement: A Dual-Gated Mixture-of-Experts for Enhancing Foundation Model Generalization against Spectral Shiftscited by 1
  19. 2026TRACE: A Temporal Conditional Estimation for Multimodal Time Series Foundation Modelscited by 1
  20. 2026Multimodal Routing for Interpretable, Robust, and Auditable Clinical Predictioncited by 1
  21. 2026Kepler-Encoder-v0.1: Towards a Multimodal Embedding Model for Robotscited by 1
  22. 2025TAMER: A Test-Time Adaptive MoE-Driven Framework for EHR Representation Learningcited by 0
  23. 2025Foundation-Model-Boosted Multimodal Learning for fMRI-based Neuropathic Pain Drug Response Predictioncited by 0
  24. 2025Layer-Aware Embedding Fusion for LLMs in Text Classificationscited by 0
  25. 2025Two Is Better Than One: Rotations Scale LoRAscited by 0
  26. 2025Neural Inhibition Improves Dynamic Routing and Mixture of Expertscited by 0
  27. 2025Atmos-Bench: 3D Atmospheric Structures for Climate Insightcited by 0
  28. 2025TinyGiantVLM: A Lightweight Vision-Language Architecture for Spatial Reasoning under Resource Constraintscited by 0
  29. 2025BTW: A Non-Parametric Variance Stabilization Framework for Multimodal Model Integrationcited by 0
  30. 2026AdaFuse: Adaptive Multimodal Fusion for Lung Cancer Risk Prediction via Reinforcement Learningcited by 0
  31. 2026Improving Minimax Estimation Rates for Contaminated Mixture of Multinomial Logistic Experts via Expert Heterogeneitycited by 0
  32. 2026Rethinking Multinomial Logistic Mixture of Experts with Sigmoid Gating Functioncited by 0
  33. 2026Align and Adapt: Multimodal Multiview Human Activity Recognition under Arbitrary View Combinationscited by 0
  34. 2026ADaFuSE: Adaptive Diffusion-generated Image and Text Fusion for Interactive Text-to-Image Retrievalcited by 0
  35. 2026Hierarchy-Guided Multimodal Representation Learning for Taxonomic Inferencecited by 0
  36. 2026M-IDoL: Information Decomposition for Modality-Specific and Diverse Representation Learning in Medical Foundation Modelcited by 0
  37. 2026Modality-Aware Zero-Shot Pruning and Sparse Attention for Efficient Multimodal Edge Inferencecited by 0
  38. 2026Toward Structural Multimodal Representations: Specialization, Selection, and Sparsification via Mixture-of-Expertscited by 0

50 most cited of 67 All 67 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2024Dynamic Mixture of Experts: An Auto-Tuning Approach for Efficient Transformer Models
  2. 2025MoTE: Mixture of Ternary Experts for Memory-efficient Large Multimodal Models
  3. 2025CoCoAFusE: Beyond Mixtures of Experts via Model Fusion
  4. 2026Harnessing Domain Specialists in Multimodal Mixture-of-Experts for Efficient Adaptation
  5. 2024Explainable data-driven modeling via mixture of experts: towards effective blending of grey and black-box models
  6. 2025Mixture-of-Transformers Learn Faster: A Theoretical Study on Classification Problems
  7. 2025DriftMoE: A Mixture of Experts Approach to Handle Concept Drifts
  8. 2024On Expert Estimation in Hierarchical Mixture of Experts: Beyond Softmax Gating Functions