# Multimodal Federated Learning under Dual-Axis Modality Missingness

arXiv: https://arxiv.org/abs/2608.09240 · HTML: https://mlsanity.com/paper/2608.09240/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 23 of its 36 references that are in the map.
- First posted on arXiv 10 Aug 2026.

## Builds on (most cited first, 23 of 23)

- [Communication-Efficient Learning of Deep Networks from Decentralized Data](https://mlsanity.com/paper/1602.05629/) (2016)
- [Neural Machine Translation by Jointly Learning to Align and Translate](https://mlsanity.com/paper/1409.0473/) (2014)
- [Advances and Open Problems in Federated Learning](https://mlsanity.com/paper/1912.04977/) (2019)
- [Federated Optimization in Heterogeneous Networks](https://mlsanity.com/paper/1812.06127/) (2018)
- [What Uncertainties Do We Need in Bayesian Deep Learning for Computer Vision?](https://mlsanity.com/paper/1703.04977/) (2017)
- [Model-Contrastive Federated Learning](https://mlsanity.com/paper/2103.16257/) (2021)
- [Federated Learning with Personalization Layers](https://mlsanity.com/paper/1912.00818/) (2019)
- [Integrated multimodal artificial intelligence framework for healthcare applications](https://mlsanity.com/paper/2202.12998/) (2022)
- [Flex-MoE: Modeling Arbitrary Modality Combination via the Flexible Mixture-of-Experts](https://mlsanity.com/paper/2410.08245/) (2024)
- [FedMultimodal: A Benchmark For Multimodal Federated Learning](https://mlsanity.com/paper/2306.09486/) (2023)
- [Sample-dependent Adaptive Temperature Scaling for Improved Calibration](https://mlsanity.com/paper/2207.06211/) (2022)
- [Recognizing Detailed Human Context In-the-Wild from Smartphones and Smartwatches](https://mlsanity.com/paper/1609.06354/) (2016)
- [Joint Classification and Prediction CNN Framework for Automatic Sleep Stage Classification](https://mlsanity.com/paper/1805.06546/) (2018)
- [Multimodal Federated Learning on IoT Data](https://mlsanity.com/paper/2109.04833/) (2021)
- [HEALNet: Multimodal Fusion for Heterogeneous Biomedical Data](https://mlsanity.com/paper/2311.09115/) (2023)
- [Automatic Sleep Stage Scoring with Single-Channel EEG Using Convolutional Neural Networks](https://mlsanity.com/paper/1610.01683/) (2016)
- [Cross-Modal Prototype based Multimodal Federated Learning under Severely Missing Modality](https://mlsanity.com/paper/2401.13898/) (2024)
- [Multimodal Federated Learning with Missing Modality via Prototype Mask and Contrast](https://mlsanity.com/paper/2312.13508/) (2023)
- [FedMAC: Tackling Partial-Modality Missing in Federated Learning with Cross-Modal Aggregation and Contrastive Regularization](https://mlsanity.com/paper/2410.03070/) (2024)
- [Federated Modality-specific Encoders and Multimodal Anchors for Personalized Brain Tumor Segmentation](https://mlsanity.com/paper/2403.11803/) (2024)
- [Learning Reconfigurable Representations for Multimodal Federated Learning with Missing Data](https://mlsanity.com/paper/2510.22880/) (2025)
- [Federated Learning for Time-Series Healthcare Sensing with Incomplete Modalities](https://mlsanity.com/paper/2405.11828/) (2024)
- [SelfReplay: Adapting Self-Supervised Sensory Models via Adaptive Meta-Task Replay](https://mlsanity.com/paper/2404.15305/) (2024)

## Abstract (arXiv)

Multimodal federated learning (FL) supports collaborative modeling in privacy-sensitive health-sensing and medical settings, but realistic deployments often exhibit dual-axis modality missingness: clients have different modality sets, and individual samples may contain only subsets of the modalities available locally. Existing methods typically address these two axes separately. We propose Flux, a multimodal federated learning framework built around two complementary components. First, modality-aware confidence tempering learns sample-specific confidence for each modality through mask-aware unimodal supervision and fuses the confidence estimates from observed modalities into a sample-adaptive temperature that adjusts predictive sharpness according to evidence quality and completeness. Second, gradient-decoupled private adaptation applies this temperature only to a client-private prediction pathway, while training the shared federated model with a standard, untempered objective. This enables sample-specific, client-local confidence adaptation without allowing confidence-dependent gradients to perturb shared representation learning. Across four multimodal datasets, Flux achieves the highest average macro-F1 on every dataset, outperforming the strongest dataset-specific baseline by 0.8~2.2 points and by 1.6 points on average. Additional analyses demonstrate favorable calibration, temperature sensitivity to both modality missingness and input corruption, and more stable shared optimization under private-only tempering. Our code is available at https://github.com/AdibaOrz/Flux.