# Prototype-guided Bilateral Alignment Multimodal Federated Learning

arXiv: https://arxiv.org/abs/2609.38925 · HTML: https://mlsanity.com/paper/2609.38925/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 22 of its 50 references that are in the map.
- First posted on arXiv 30 Sep 2026.

## Builds on (most cited first, 22 of 22)

- [Sinkhorn Distances: Lightspeed Computation of Optimal Transportation Distances](https://mlsanity.com/paper/1306.0895/) (2013)
- [Exploiting Shared Representations for Personalized Federated Learning](https://mlsanity.com/paper/2102.07078/) (2021)
- [FedProto: Federated Prototype Learning across Heterogeneous Clients](https://mlsanity.com/paper/2105.00243/) (2021)
- [Federated Learning from Pre-Trained Models: A Contrastive Learning Approach](https://mlsanity.com/paper/2209.10083/) (2022)
- [FedDAT: An Approach for Foundation Model Finetuning in Multi-Modal Heterogeneous Federated Learning](https://mlsanity.com/paper/2308.12305/) (2023)
- [The Unbalanced Gromov Wasserstein Distance: Conic Formulation and Relaxation](https://mlsanity.com/paper/2009.04266/) (2020)
- [FedGH: Heterogeneous Federated Learning with Generalized Global Header](https://mlsanity.com/paper/2303.13137/) (2023)
- [FedMultimodal: A Benchmark For Multimodal Federated Learning](https://mlsanity.com/paper/2306.09486/) (2023)
- [FedTGP: Trainable Global Prototypes with Adaptive-Margin-Enhanced Contrastive Learning for Data and Model Heterogeneity in Federated Learning](https://mlsanity.com/paper/2401.03230/) (2024)
- [Cross-Silo Prototypical Calibration for Federated Learning with Non-IID Data](https://mlsanity.com/paper/2308.03457/) (2023)
- [Multimodal Federated Learning on IoT Data](https://mlsanity.com/paper/2109.04833/) (2021)
- [AutoFed: Heterogeneity-Aware Federated Multimodal Learning for Robust Autonomous Driving](https://mlsanity.com/paper/2302.08646/) (2023)
- [FedCross: Towards Accurate Federated Learning via Multi-Model Cross-Aggregation](https://mlsanity.com/paper/2210.08285/) (2022)
- [Leveraging Foundation Models for Multi-modal Federated Learning with Incomplete Modality](https://mlsanity.com/paper/2406.11048/) (2024)
- [Is Aggregation the Only Choice? Federated Learning via Layer-wise Model Recombination](https://mlsanity.com/paper/2305.10730/) (2023)
- [Multimodal Federated Learning in Healthcare: a Review](https://mlsanity.com/paper/2310.09650/) (2023)
- [Advances in Robust Federated Learning: A Survey with Heterogeneity Considerations](https://mlsanity.com/paper/2405.09839/) (2024)
- [Unimodal Training-Multimodal Prediction: Cross-modal Federated Learning with Hierarchical Aggregation](https://mlsanity.com/paper/2303.15486/) (2023)
- [HtFLlib: A Comprehensive Heterogeneous Federated Learning Library and Benchmark](https://mlsanity.com/paper/2506.03954/) (2025)
- [FedPIA -- Permuting and Integrating Adapters leveraging Wasserstein Barycenters for Finetuning Foundation Models in Multi-Modal Federated Learning](https://mlsanity.com/paper/2412.14424/) (2024)
- [FedMobile: Enabling Knowledge Contribution-aware Multi-modal Federated Learning with Incomplete Modalities](https://mlsanity.com/paper/2502.15839/) (2025)
- [FedPall: Prototype-based Adversarial and Collaborative Learning for Federated Learning with Feature Drift](https://mlsanity.com/paper/2507.04781/) (2025)

## Abstract (arXiv)

Multimodal federated learning (MFL) has emerged as a pivotal paradigm for leveraging distributed data to enhance model performance. However, existing methods predominantly rely on idealized assumptions of model homogeneity and balanced modality distributions, rendering them ill-suited for practical scenarios characterized by heterogeneous client architectures and severe modality imbalance. To address these challenges, we propose a Multimodal Federated learning Prototype-guided Bilateral Alignment (MFedPBA) framework. MFedPBA facilitates robust knowledge synergy through a dual alignment mechanism: (i) at the feature level, it aligns heterogeneous feature spaces via a projection encoder optimized by contrastive learning and the Gromov-Wasserstein distance; (ii) at the decision level, it employs an entropy-weighted aggregation of naturally aligned logit prototypes. This novel design achieves robust MFL by jointly tackling heterogeneous feature spaces and collectively aggregating decisions. Extensive experiments demonstrate that our method significantly outperforms state-of-the-art baselines under conditions of model heterogeneity and modality imbalance.