# FedMFS: Federated Multimodal Fusion Learning with Selective Modality Communication

arXiv: https://arxiv.org/abs/2310.07048 · HTML: https://mlsanity.com/paper/2310.07048/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 13 of its 25 references that are in the map.
- First posted on arXiv 10 Oct 2023.

## Builds on (most cited first, 13 of 13)

- [Communication-Efficient Learning of Deep Networks from Decentralized Data](https://mlsanity.com/paper/1602.05629/) (2016)
- [A Unified Approach to Interpreting Model Predictions](https://mlsanity.com/paper/1705.07874/) (2017)
- [The many Shapley values for model explanation](https://mlsanity.com/paper/1908.08474/) (2019)
- [FedKD: Communication Efficient Federated Learning via Knowledge Distillation](https://mlsanity.com/paper/2108.13323/) (2021)
- [Decentralized Federated Learning: A Survey and Perspective](https://mlsanity.com/paper/2306.01603/) (2023)
- [Communication-Efficient Stochastic Zeroth-Order Optimization for Federated Learning](https://mlsanity.com/paper/2201.09531/) (2022)
- [FedMultimodal: A Benchmark For Multimodal Federated Learning](https://mlsanity.com/paper/2306.09486/) (2023)
- [Federated Learning for Connected and Automated Vehicles: A Survey of Existing Approaches and Challenges](https://mlsanity.com/paper/2308.10407/) (2023)
- [Peer-to-Peer Federated Continual Learning for Naturalistic Driving Action Recognition](https://mlsanity.com/paper/2304.07421/) (2023)
- [A Survey of Federated Learning for Connected and Automated Vehicles](https://mlsanity.com/paper/2303.10677/) (2023)
- [On the Convergence of Decentralized Federated Learning Under Imperfect Information Sharing](https://mlsanity.com/paper/2303.10695/) (2023)
- [Federated Transfer-Ordered-Personalized Learning for Driver Monitoring Application](https://mlsanity.com/paper/2301.04829/) (2023)
- [Passive Radio Frequency-based 3D Indoor Positioning System via Ensemble Learning](https://mlsanity.com/paper/2304.06513/) (2023)

## Built on by (most cited first, 4 of 4)

- [Communication-Efficient Multimodal Federated Learning: Joint Modality and Client Selection](https://mlsanity.com/paper/2401.16685/) (2024)
- [FedNMUT -- Federated Noisy Model Update Tracking Convergence Analysis](https://mlsanity.com/paper/2403.13247/) (2024)
- [FedMMKT:Co-Enhancing a Server Text-to-Image Model and Client Task Models in Multi-Modal Federated Learning](https://mlsanity.com/paper/2510.12254/) (2025)
- [Boosting Multimodal Federated Learning via Chained Modality Optimization](https://mlsanity.com/paper/2606.01856/) (2026)

## Abstract (arXiv)

Multimodal federated learning (FL) aims to enrich model training in FL settings where devices are collecting measurements across multiple modalities (e.g., sensors measuring pressure, motion, and other types of data). However, key challenges to multimodal FL remain unaddressed, particularly in heterogeneous network settings: (i) the set of modalities collected by each device will be diverse, and (ii) communication limitations prevent devices from uploading all their locally trained modality models to the server. In this paper, we propose Federated Multimodal Fusion learning with Selective modality communication (FedMFS), a new multimodal fusion FL methodology that can tackle the above mentioned challenges. The key idea is the introduction of a modality selection criterion for each device, which weighs (i) the impact of the modality, gauged by Shapley value analysis, against (ii) the modality model size as a gauge for communication overhead. This enables FedMFS to flexibly balance performance against communication costs, depending on resource constraints and application requirements. Experiments on the real-world ActionSense dataset demonstrate the ability of FedMFS to achieve comparable accuracy to several baselines while reducing the communication overhead by over 4x.