# MedMix: Specialization-Consistent Federated Sparse MoEs under Modality Heterogeneity

arXiv: https://arxiv.org/abs/2608.13911 · HTML: https://mlsanity.com/paper/2608.13911/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Medical and biomedical MoE, one of 12 directions in Mixture-of-experts.
- Builds on 21 of its 30 references that are in the map.
- First posted on arXiv 14 Aug 2026.

## Placed in

- Mixture-of-experts / [Medical and biomedical MoE](https://mlsanity.com/moe/medical-and-biomedical-moe/)

## Builds on (most cited first, 21 of 21)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Communication-Efficient Learning of Deep Networks from Decentralized Data](https://mlsanity.com/paper/1602.05629/) (2016)
- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [Outrageously Large Neural Networks: The Sparsely-Gated Mixture-of-Experts Layer](https://mlsanity.com/paper/1701.06538/) (2017)
- [Switch Transformers: Scaling to Trillion Parameter Models with Simple and Efficient Sparsity](https://mlsanity.com/paper/2101.03961/) (2021)
- [Advances and Open Problems in Federated Learning](https://mlsanity.com/paper/1912.04977/) (2019)
- [GShard: Scaling Giant Models with Conditional Computation and Automatic Sharding](https://mlsanity.com/paper/2006.16668/) (2020)
- [Multimodal Machine Learning: A Survey and Taxonomy](https://mlsanity.com/paper/1705.09406/) (2017)
- [SMIL: Multimodal Learning with Severely Missing Modality](https://mlsanity.com/paper/2103.05677/) (2021)
- [HeMIS: Hetero-Modal Image Segmentation](https://mlsanity.com/paper/1607.05194/) (2016)
- [Deep Multimodal Learning with Missing Modality: A Survey](https://mlsanity.com/paper/2409.07825/) (2024)
- [Flex-MoE: Modeling Arbitrary Modality Combination via the Flexible Mixture-of-Experts](https://mlsanity.com/paper/2410.08245/) (2024)
- [FuseMoE: Mixture-of-Experts Transformers for Fleximodal Fusion](https://mlsanity.com/paper/2402.03226/) (2024)
- [FedMultimodal: A Benchmark For Multimodal Federated Learning](https://mlsanity.com/paper/2306.09486/) (2023)
- [Federated Mixture of Experts](https://mlsanity.com/paper/2107.06724/) (2021)
- [FedMoE: Personalized Federated Learning via Heterogeneous Mixture of Experts](https://mlsanity.com/paper/2408.11304/) (2024)
- [FedMAC: Tackling Partial-Modality Missing in Federated Learning with Cross-Modal Aggregation and Contrastive Regularization](https://mlsanity.com/paper/2410.03070/) (2024)
- [Learning Reconfigurable Representations for Multimodal Federated Learning with Missing Data](https://mlsanity.com/paper/2510.22880/) (2025)
- [Aggregation Alignment for Federated Learning with Mixture-of-Experts under Data Heterogeneity](https://mlsanity.com/paper/2603.21276/) (2026)
- [Federated Learning for Time-Series Healthcare Sensing with Incomplete Modalities](https://mlsanity.com/paper/2405.11828/) (2024)

## Abstract (arXiv)

Federated multimodal medical AI faces modality heterogeneity at both the client and sample levels: clients may systematically lack access to specific modality types, while individual records within the same client may contain different partial modality subsets. Sparse Mixture-of-Experts (MoE) architectures are a promising remedy for modality-adaptive computation, but their use in federated learning is fragile under cross-client modality heterogeneity, where locally learned routing policies can diverge across clients and drive experts toward incompatible specializations. Different clients may assign the same observed modality configuration to different experts, or train similarly indexed experts on different missing-modality configurations, causing standard aggregation to misalign or overwrite the expert specialization that sparse MoEs are intended to learn. To address this challenge, we propose MedMix, a semantic-alignment framework for federated multimodal sparse MoEs that coordinates cross-client routing and expert specialization using modality context. At the client side, MedMix uses modality-context-aware routing to guide expert selection using each token's modality identity, position, and incompleteness context. Across clients, it uses consensus-guided routing alignment to construct server-side consensus anchors for shared modality patterns and align local routing distributions across clients. Complementing these routing mechanisms, client-adaptive expert aggregation leverages client-specific modality-pattern prototypes to match and aggregate functionally similar experts across clients. Experiments on real-world multimodal medical datasets show that MedMix achieves the best average F1 across diverse modality heterogeneity and modality incompleteness settings, with especially clear gains under severe heterogeneity.