# FedFusion: Manifold Driven Federated Learning for Multi-satellite and Multi-modality Fusion

arXiv: https://arxiv.org/abs/2311.09540 · HTML: https://mlsanity.com/paper/2311.09540/

## Key facts

- Cited by 8 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 51 references that are in the map.
- First posted on arXiv 16 Nov 2023.

## Builds on (most cited first, 19 of 19)

- [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020)
- [On the Properties of Neural Machine Translation: Encoder-Decoder Approaches](https://mlsanity.com/paper/1409.1259/) (2014)
- [Depth Map Prediction from a Single Image using a Multi-Scale Deep Network](https://mlsanity.com/paper/1406.2283/) (2014)
- [Model-Contrastive Federated Learning](https://mlsanity.com/paper/2103.16257/) (2021)
- [An Efficient Framework for Clustered Federated Learning](https://mlsanity.com/paper/2006.04088/) (2020)
- [Source Data-absent Unsupervised Domain Adaptation through Hypothesis Transfer and Labeling Transfer](https://mlsanity.com/paper/2012.07297/) (2020)
- [More Diverse Means Better: Multimodal Deep Learning Meets Remote Sensing Imagery Classification](https://mlsanity.com/paper/2008.05457/) (2020)
- [Deep Learning for Hyperspectral Image Classification: An Overview](https://mlsanity.com/paper/1910.12861/) (2019)
- [Coming Down to Earth: Satellite-to-Street View Synthesis for Geo-Localization](https://mlsanity.com/paper/2103.06818/) (2021)
- [Beyond RGB: Very High Resolution Urban Remote Sensing With Multimodal Deep Networks](https://mlsanity.com/paper/1711.08681/) (2017)
- [Multimodal Fusion Transformer for Remote Sensing Image Classification](https://mlsanity.com/paper/2203.16952/) (2022)
- [Scheduling for Cellular Federated Edge Learning with Importance and Channel Awareness](https://mlsanity.com/paper/2004.00490/) (2020)
- [SuperYOLO: Super Resolution Assisted Object Detection in Multimodal Remote Sensing Imagery](https://mlsanity.com/paper/2209.13351/) (2022)
- [Survey: Leakage and Privacy at Inference Time](https://mlsanity.com/paper/2107.01614/) (2021)
- [FedSpace: An Efficient Federated Learning Framework at Satellites and Ground Stations](https://mlsanity.com/paper/2202.01267/) (2022)
- [(Un)informed Consent: Studying GDPR Consent Notices in the Field](https://mlsanity.com/paper/1909.02638/) (2019)
- [High-Performance Distributed ML at Scale through Parameter Server Consistency Models](https://mlsanity.com/paper/1410.8043/) (2014)
- [On the Design of Communication-Efficient Federated Learning for Health Monitoring](https://mlsanity.com/paper/2211.16952/) (2022)
- [PMFL: Partial Meta-Federated Learning for heterogeneous tasks and its applications on real-world medical records](https://mlsanity.com/paper/2112.05321/) (2021)

## Built on by (most cited first, 8 of 8)

- [Diffusion Models Meet Remote Sensing: Principles, Methods, and Perspectives](https://mlsanity.com/paper/2404.08926/) (2024)
- [Multimodal Informative ViT: Information Aggregation and Distribution for Hyperspectral and LiDAR Classification](https://mlsanity.com/paper/2401.03179/) (2024)
- [Exploring Hyperspectral Anomaly Detection with Human Vision: A Small Target Aware Detector](https://mlsanity.com/paper/2401.01093/) (2024)
- [RS-DGC: Exploring Neighborhood Statistics for Dynamic Gradient Compression on Remote Sensing Image Interpretation](https://mlsanity.com/paper/2312.17530/) (2023)
- [Multimodal Federated Learning: A Survey through the Lens of Different FL Paradigms](https://mlsanity.com/paper/2505.21792/) (2025)
- [FedEU: Evidential Uncertainty-Driven Federated Fine-Tuning of Vision Foundation Models for Remote Sensing Image Segmentation](https://mlsanity.com/paper/2603.07468/) (2026)
- [Federated Weather Modeling on Sensor Data](https://mlsanity.com/paper/2605.00322/) (2026)
- [FedRings: A Scalable and Topology-Aware Federated Learning Framework for LEO Satellite Constellations](https://mlsanity.com/paper/2608.03436/) (2026)

## Abstract (arXiv)

Multi-satellite, multi-modality in-orbit fusion is a challenging task as it explores the fusion representation of complex high-dimensional data under limited computational resources. Deep neural networks can reveal the underlying distribution of multi-modal remote sensing data, but the in-orbit fusion of multimodal data is more difficult because of the limitations of different sensor imaging characteristics, especially when the multimodal data follows non-independent identically distribution (Non-IID) distributions. To address this problem while maintaining classification performance, this paper proposes a manifold-driven multi-modality fusion framework, FedFusion, which randomly samples local data on each client to jointly estimate the prominent manifold structure of shallow features of each client and explicitly compresses the feature matrices into a low-rank subspace through cascading and additive approaches, which is used as the feature input of the subsequent classifier. Considering the physical space limitations of the satellite constellation, we developed a multimodal federated learning module designed specifically for manifold data in a deep latent space. This module achieves iterative updating of the sub-network parameters of each client through global weighted averaging, constructing a framework that can represent compact representations of each client. The proposed framework surpasses existing methods in terms of performance on three multimodal datasets, achieving a classification average accuracy of 94.35% while compressing communication costs by a factor of 4. Furthermore, extensive numerical evaluations of real-world satellite images were conducted on the orbiting edge computing architecture based on Jetson TX2 industrial modules, which demonstrated that FedFusion significantly reduced training time by 48.4 minutes (15.18%) while optimizing accuracy.