mlsanityData to 7 Oct 2026

Overcome Modal Bias in Multi-modal Federated Learning via Balanced Modality Selection

arXiv 2401.00403 ↗First posted 31 Dec 2023cs.LG cs.CV cs.MMcited by 6

Not on the map. Why?

Abstract (arXiv)

Selecting proper clients to participate in each federated learning (FL) round is critical to effectively harness a broad range of distributed data. Existing client selection methods simply consider the mining of distributed uni-modal data, yet, their effectiveness may diminish in multi-modal FL (MFL) as the modality imbalance problem not only impedes the collaborative local training but also leads to a severe global modality-level bias. We empirically reveal that local training with a certain single modality may contribute more to the global model than training with all local modalities. To effectively exploit the distributed multiple modalities, we propose a novel Balanced Modality Selection framework for MFL (BMSFed) to overcome the modal bias. On the one hand, we introduce a modal enhancement loss during local training to alleviate local imbalance based on the aggregated global prototypes. On the other hand, we propose the modality selection aiming to select subsets of local modalities with great diversity and achieving global modal balance simultaneously. Our extensive experiments on audio-visual, colored-gray, and front-back datasets showcase the superiority of BMSFed over baselines and its effectiveness in multi-modal data exploitation.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Builds on 28 of its 46 references that are in the map. First posted on arXiv 31 Dec 2023.

Builds on

28 of 46 references.

References · 28

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,592
  3. 2018Federated Optimization in Heterogeneous Networkscited by 2,971
  4. 2019Federated Learning: Challenges, Methods, and Future Directionscited by 1,531
  5. 2019On the Convergence of FedAvg on Non-IID Datacited by 1,220
  6. 2019Measuring the Effects of Non-Identical Data Distribution for Federated Visual Classificationcited by 836
  7. 2015Multi-view Convolutional Neural Networks for 3D Shape Recognitioncited by 564
  8. 2021FedProto: Federated Prototype Learning across Heterogeneous Clientscited by 349
  9. 2019What Makes Training Multi-Modal Classification Networks Hard?cited by 340
  10. 2018Audio-Visual Scene Analysis with Self-Supervised Multisensory Featurescited by 335
  11. 2022Balanced Multimodal Learning via On-the-fly Gradient Modulationcited by 259
  12. 2018Learning Not to Learn: Training Deep Neural Networks with Biased Datacited by 255
16 more
  1. 2020Client Selection in Federated Learning: Convergence Analysis and Power-of-Choice Selection Strategiescited by 196
  2. 2020Audiovisual SlowFast Networks for Video Recognitioncited by 144
  3. 2019Federated Learning for Wireless Communications: Motivation, Opportunities and Challengescited by 128
  4. 2022Modality Competition: What Makes Joint Training of Multi-modal Network Fail in Deep Learning? (Provably)cited by 115
  5. 2022PMR: Prototypical Modal Rebalance for Multimodal Learningcited by 114
  6. 2020Client Selection and Bandwidth Allocation in Wireless Federated Learning Networks: A Long-Term Perspectivecited by 102
  7. 2014Lazier Than Lazy Greedycited by 97
  8. 2023Personalized Federated Learning with Feature Alignment and Classifier Collaborationcited by 97
  9. 2022Characterizing and overcoming the greedy nature of learning in multi-modal deep neural networkscited by 92
  10. 2023Multimodal Federated Learning via Contrastive Representation Ensemblecited by 85
  11. 2023Boosting Multi-modal Model Performance with Adaptive Gradient Modulationcited by 71
  12. 2022Tackling Data Heterogeneity in Federated Learning with Class Prototypescited by 66
  13. 2020Straggler-Resilient Federated Learning: Leveraging the Interplay Between Statistical Accuracy and System Heterogeneitycited by 56
  14. 2020Multimodal Categorization of Crisis Events in Social Mediacited by 34
  15. 2021Multimodal Federated Learning on IoT Datacited by 32
  16. 2022Learnable Irrelevant Modality Dropout for Multimodal Action Recognition on Modality-Specific Annotated Videoscited by 14

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 3; 2026: 3.

6 papers.

  1. 2025Multifaceted Evaluation of Audio-Visual Capability for MLLMs: Effectiveness, Efficiency, Generalizability and Robustnesscited by 3
  2. 2025Mitigating Modality Quantity and Quality Imbalance in Multimodal Online Federated Learningcited by 2
  3. 2025Role-Aware Multi-modal federated learning system for detecting phishing webpagescited by 0
  4. 2026PID-Guided Partial Alignment for Multimodal Decentralized Federated Learningcited by 0
  5. 2026Boosting Multimodal Federated Learning via Chained Modality Optimizationcited by 0
  6. 2026FedSocket: Recipient-Executable Knowledge Exchange for Heterogeneous Multimodal Federated Learningcited by 0