# Boosting Multimodal Federated Learning via Chained Modality Optimization

arXiv: https://arxiv.org/abs/2606.01856 · HTML: https://mlsanity.com/paper/2606.01856/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 32 of its 48 references that are in the map.
- First posted on arXiv 1 Jun 2026.

## Builds on (most cited first, 30 of 32)

- [Communication-Efficient Learning of Deep Networks from Decentralized Data](https://mlsanity.com/paper/1602.05629/) (2016)
- [Federated Optimization in Heterogeneous Networks](https://mlsanity.com/paper/1812.06127/) (2018)
- [Local SGD Converges Fast and Communicates Little](https://mlsanity.com/paper/1805.09767/) (2018)
- [What Makes Training Multi-Modal Classification Networks Hard?](https://mlsanity.com/paper/1905.12681/) (2019)
- [Parallel Restarted SGD with Faster Convergence and Less Communication: Demystifying Why Model Averaging Works for Deep Learning](https://mlsanity.com/paper/1807.06629/) (2018)
- [Balanced Multimodal Learning via On-the-fly Gradient Modulation](https://mlsanity.com/paper/2203.15332/) (2022)
- [Modality Competition: What Makes Joint Training of Multi-modal Network Fail in Deep Learning? (Provably)](https://mlsanity.com/paper/2203.12221/) (2022)
- [PMR: Prototypical Modal Rebalance for Multimodal Learning](https://mlsanity.com/paper/2211.07089/) (2022)
- [Stochastic Controlled Averaging for Federated Learning with Communication Compression](https://mlsanity.com/paper/2308.08165/) (2023)
- [Characterizing and overcoming the greedy nature of learning in multi-modal deep neural networks](https://mlsanity.com/paper/2202.05306/) (2022)
- [Multimodal Federated Learning via Contrastive Representation Ensemble](https://mlsanity.com/paper/2302.08888/) (2023)
- [Heterogeneous Ensemble Knowledge Transfer for Training Large Models in Federated Learning](https://mlsanity.com/paper/2204.12703/) (2022)
- [Multimodal Representation Learning by Alternating Unimodal Adaptation](https://mlsanity.com/paper/2311.10707/) (2023)
- [Boosting Multi-modal Model Performance with Adaptive Gradient Modulation](https://mlsanity.com/paper/2308.07686/) (2023)
- [MMPareto: Boosting Multimodal Learning with Innocent Unimodal Assistance](https://mlsanity.com/paper/2405.17730/) (2024)
- [ReconBoost: Boosting Can Achieve Modality Reconcilement](https://mlsanity.com/paper/2405.09321/) (2024)
- [On Uni-Modal Feature Learning in Supervised Multi-Modal Learning](https://mlsanity.com/paper/2305.01233/) (2023)
- [AutoFed: Heterogeneity-Aware Federated Multimodal Learning for Robust Autonomous Driving](https://mlsanity.com/paper/2302.08646/) (2023)
- [Cross-Modal Prototype based Multimodal Federated Learning under Severely Missing Modality](https://mlsanity.com/paper/2401.13898/) (2024)
- [Multimodal Federated Learning with Missing Modality via Prototype Mask and Contrast](https://mlsanity.com/paper/2312.13508/) (2023)
- [Detached and Interactive Multimodal Learning](https://mlsanity.com/paper/2407.19514/) (2024)
- [Unimodal Training-Multimodal Prediction: Cross-modal Federated Learning with Hierarchical Aggregation](https://mlsanity.com/paper/2303.15486/) (2023)
- [Improving Multimodal Learning Balance and Sufficiency through Data Remixing](https://mlsanity.com/paper/2506.11550/) (2025)
- [Quantum Federated Learning for Multimodal Data: A Modality-Agnostic Approach](https://mlsanity.com/paper/2507.08217/) (2025)
- [Overcome Modal Bias in Multi-modal Federated Learning via Balanced Modality Selection](https://mlsanity.com/paper/2401.00403/) (2023)
- [FedMFS: Federated Multimodal Fusion Learning with Selective Modality Communication](https://mlsanity.com/paper/2310.07048/) (2023)
- [Open-Vocabulary Federated Learning with Multimodal Prototyping](https://mlsanity.com/paper/2404.01232/) (2024)
- [Multimodal Classification via Modal-Aware Interactive Enhancement](https://mlsanity.com/paper/2407.04587/) (2024)
- [Federated Learning for Time-Series Healthcare Sensing with Incomplete Modalities](https://mlsanity.com/paper/2405.11828/) (2024)
- [Contribution Evaluation of Heterogeneous Participants in Federated Learning via Prototypical Representations](https://mlsanity.com/paper/2407.02073/) (2024)

## Abstract (arXiv)

Multimodal Federated Learning (MMFL) enables privacy-preserving collaborative learning across decentralized clients with heterogeneous data and modality availability. However, most existing MMFL methods cast multimodal training as a joint optimization problem, overlooking a key bottleneck: modality competition, where dominant modalities suppress weaker ones and lead to suboptimal global models. To address this, we propose FedMChain, a balanced MMFL framework that structures federated multimodal training as a chain of modality-wise phases. This phase-wise design gives each modality a dedicated local optimization window on multimodal clients to mitigate modality competition, and further promotes cross-modal complementarity via an error-compensated regularizer. On the server side, we employ a sparse sign-guided aggregation strategy that leverages directional sign agreement for robust intra-modality aggregation, avoids destructive averaging, and supports less frequent synchronization to reduce communication overhead. Extensive experiments on multimodal benchmarks demonstrate that FedMChain consistently improves predictive performance while requiring less frequent communication than baselines.