# A Multi-Modal Federated Learning Framework for Remote Sensing Image Classification

arXiv: https://arxiv.org/abs/2503.10262 · HTML: https://mlsanity.com/paper/2503.10262/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 51 references that are in the map.
- First posted on arXiv 13 Mar 2025.

## Builds on (most cited first, 26 of 26)

- [A Simple Framework for Contrastive Learning of Visual Representations](https://mlsanity.com/paper/2002.05709/) (2020)
- [Communication-Efficient Learning of Deep Networks from Decentralized Data](https://mlsanity.com/paper/1602.05629/) (2016)
- [Federated Learning: Challenges, Methods, and Future Directions](https://mlsanity.com/paper/1908.07873/) (2019)
- [Model-Contrastive Federated Learning](https://mlsanity.com/paper/2103.16257/) (2021)
- [FedBN: Federated Learning on Non-IID Features via Local Batch Normalization](https://mlsanity.com/paper/2102.07623/) (2021)
- [FedKD: Communication Efficient Federated Learning via Knowledge Distillation](https://mlsanity.com/paper/2108.13323/) (2021)
- [FedDC: Federated Learning with Non-IID Data via Local Drift Decoupling and Correction](https://mlsanity.com/paper/2203.11751/) (2022)
- [CROMA: Remote Sensing Representations with Contrastive Radar-Optical Masked Autoencoders](https://mlsanity.com/paper/2311.00566/) (2023)
- [Privacy and Robustness in Federated Learning: Attacks and Defenses](https://mlsanity.com/paper/2012.06337/) (2020)
- [Meta-Transformer: A Unified Framework for Multimodal Learning](https://mlsanity.com/paper/2307.10802/) (2023)
- [Personalized Federated Learning with Feature Alignment and Classifier Collaboration](https://mlsanity.com/paper/2306.11867/) (2023)
- [Layer-wised Model Aggregation for Personalized Federated Learning](https://mlsanity.com/paper/2205.03993/) (2022)
- [BigEarthNet-MM: A Large Scale Multi-Modal Multi-Label Benchmark Archive for Remote Sensing Image Classification and Retrieval](https://mlsanity.com/paper/2105.07921/) (2021)
- [Papaya: Practical, Private, and Scalable Federated Learning](https://mlsanity.com/paper/2111.04877/) (2021)
- [Multimodal Federated Learning via Contrastive Representation Ensemble](https://mlsanity.com/paper/2302.08888/) (2023)
- [Unsupervised Domain Adaptation using Feature-Whitening and Consensus Loss](https://mlsanity.com/paper/1903.03215/) (2019)
- [FedDAT: An Approach for Foundation Model Finetuning in Multi-Modal Heterogeneous Federated Learning](https://mlsanity.com/paper/2308.12305/) (2023)
- [Collaborative Federated Learning For Healthcare: Multi-Modal COVID-19 Diagnosis at the Edge](https://mlsanity.com/paper/2101.07511/) (2021)
- [On the Effects of Different Types of Label Noise in Multi-Label Remote Sensing Image Classification](https://mlsanity.com/paper/2207.13975/) (2022)
- [Multi-Label Noise Robust Collaborative Learning for Remote Sensing Image Classification](https://mlsanity.com/paper/2012.10715/) (2020)
- [A Novel Self-Supervised Cross-Modal Image Retrieval Method In Remote Sensing](https://mlsanity.com/paper/2202.11429/) (2022)
- [Federated Learning Across Decentralized and Unshared Archives for Remote Sensing Image Classification](https://mlsanity.com/paper/2311.06141/) (2023)
- [Learning Across Decentralized Multi-Modal Remote Sensing Archives with Federated Learning](https://mlsanity.com/paper/2306.00792/) (2023)
- [Free Lunch for Federated Remote Sensing Target Fine-Grained Classification: A Parameter-Efficient Framework](https://mlsanity.com/paper/2401.01493/) (2024)
- [Transformer-based Federated Learning for Multi-Label Remote Sensing Image Classification](https://mlsanity.com/paper/2405.15405/) (2024)
- [Transformer-based Multi-Modal Learning for Multi Label Remote Sensing Image Classification](https://mlsanity.com/paper/2306.01523/) (2023)

## Built on by (most cited first, 1 of 1)

- [FedRS-Bench: Realistic Federated Learning Datasets and Benchmarks in Remote Sensing](https://mlsanity.com/paper/2505.08325/) (2025)

## Abstract (arXiv)

Federated learning (FL) enables the collaborative training of deep neural networks across decentralized data archives (i.e., clients) without sharing the local data of the clients. Most of the existing FL methods assume that the data distributed across all clients is associated with the same data modality. However, remote sensing (RS) images present in different clients can be associated with diverse data modalities. The joint use of the multi-modal RS data can significantly enhance classification performance. To effectively exploit decentralized and unshared multi-modal RS data, our paper introduces a novel multi-modal FL framework for RS image classification problems. The proposed framework comprises three modules: 1) multi-modal fusion (MF); 2) feature whitening (FW); and 3) mutual information maximization (MIM). The MF module employs iterative model averaging to facilitate learning without accessing multi-modal training data on clients. The FW module aims to address the limitations of training data heterogeneity by aligning data distributions across clients. The MIM module aims to model mutual information by maximizing the similarity between images from different modalities. For the experimental analyses, we focus our attention on multi-label classification and pixel-based classification tasks in RS. The results obtained using two benchmark archives show the effectiveness of the proposed framework when compared to state-of-the-art algorithms in the literature. The code of the proposed framework will be available at https://git.tu-berlin.de/rsim/multi-modal-FL.