mlsanityData to 7 Oct 2026

FedDiff: Diffusion Model Driven Federated Learning for Multi-Modal and Multi-Clients

arXiv 2401.02433 ↗First posted 16 Nov 2023cs.CV cs.AI cs.LGcited by 23

On the map: Diffusion models → Inverse problems and image restoration

Abstract (arXiv)

With the rapid development of imaging sensor technology in the field of remote sensing, multi-modal remote sensing data fusion has emerged as a crucial research direction for land cover classification tasks. While diffusion models have made great progress in generative models and image classification tasks, existing models primarily focus on single-modality and single-client control, that is, the diffusion process is driven by a single modal in a single computing node. To facilitate the secure fusion of heterogeneous data from clients, it is necessary to enable distributed multi-modal control, such as merging the hyperspectral data of organization A and the LiDAR data of organization B privately on each base station client. In this study, we propose a multi-modal collaborative diffusion federated learning framework called FedDiff. Our framework establishes a dual-branch diffusion model feature extraction setup, where the two modal data are inputted into separate branches of the encoder. Our key insight is that diffusion models driven by different modalities are inherently complementary in terms of potential denoising steps on which bilateral connections can be built. Considering the challenge of private and efficient communication between multiple clients, we embed the diffusion model into the federated learning communication structure, and introduce a lightweight communication module. Qualitative and quantitative experiments validate the superiority of our framework in terms of image quality and conditional consistency.

Key facts. Cited by 23 papers in the map (data through 7 Oct 2026). Placed in Inverse problems and image restoration, one of 14 directions in Diffusion models. Builds on 33 of its 53 references that are in the map. First posted on arXiv 16 Nov 2023.

Builds on

33 of 53 references.

References · 33

  1. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 29,915
  2. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  3. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,592
  4. 2015Deep Unsupervised Learning using Nonequilibrium Thermodynamicscited by 7,005
  5. 2020Taming Transformers for High-Resolution Image Synthesiscited by 3,229
  6. 2018Federated Optimization in Heterogeneous Networkscited by 2,971
  7. 2019Federated Learning: Challenges, Methods, and Future Directionscited by 1,531
  8. 2021Image Super-Resolution via Iterative Refinementcited by 1,506
  9. 2016Federated Learning: Strategies for Improving Communication Efficiencycited by 1,461
  10. 2014On the Properties of Neural Machine Translation: Encoder-Decoder Approachescited by 1,303
  11. 2020Tackling the Objective Inconsistency Problem in Heterogeneous Federated Optimizationcited by 897
  12. 2021Multiscale Vision Transformerscited by 896
21 more
  1. 2019Towards Federated Learning at Scale: System Designcited by 826
  2. 2017Federated Multi-Task Learningcited by 669
  3. 2021Federated Learning on Non-IID Data Silos: An Experimental Studycited by 566
  4. 2021Label-Efficient Semantic Segmentation with Diffusion Modelscited by 495
  5. 2021FjORD: Fair and Accurate Federated Learning under heterogeneous targets with Ordered Dropoutcited by 192
  6. 2020Source Data-absent Unsupervised Domain Adaptation through Hypothesis Transfer and Labeling Transfercited by 173
  7. 2023DDFM: Denoising Diffusion Model for Multi-Modality Image Fusioncited by 153
  8. 2020More Diverse Means Better: Multimodal Deep Learning Meets Remote Sensing Imagery Classificationcited by 126
  9. 2023Collaborative Diffusion for Multi-Modal Face Generation and Editingcited by 116
  10. 2022CARD: Classification and Regression Diffusion Modelscited by 102
  11. 2017Beyond RGB: Very High Resolution Urban Remote Sensing With Multimodal Deep Networkscited by 64
  12. 2020Scheduling for Cellular Federated Edge Learning with Importance and Channel Awarenesscited by 62
  13. 2022Multimodal Fusion Transformer for Remote Sensing Image Classificationcited by 61
  14. 2023Dif-Fusion: Towards High Color Fidelity in Infrared and Visible Image Fusion with Diffusion Modelscited by 56
  15. 2021Survey: Leakage and Privacy at Inference Timecited by 44
  16. 2020Communication-Efficient Federated Learning with Compensated Overlap-FedAvgcited by 31
  17. 2019(Un)informed Consent: Studying GDPR Consent Notices in the Fieldcited by 23
  18. 2022FedMed-GAN: Federated Domain Translation on Unsupervised Cross-Modality Brain Image Synthesiscited by 21
  19. 2014High-Performance Distributed ML at Scale through Parameter Server Consistency Modelscited by 10
  20. 2023Decomposed Guided Dynamic Filters for Efficient RGB-Guided Depth Completioncited by 10
  21. 2023Transformer-based Multi-Modal Learning for Multi Label Remote Sensing Image Classificationcited by 4

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 4; 2025: 14; 2026: 5.

23 papers.

Sort:

By area:

  1. 2024Diffusion Models Meet Remote Sensing: Principles, Methods, and Perspectivescited by 27
  2. 2025FedRSClip: Federated Learning for Remote Sensing Scene Classification Using Vision-Language Modelscited by 11
  3. 2025Anomaly Detection and Generation with Diffusion Models: A Surveycited by 8
  4. 2025Federated Learning for Diffusion Modelscited by 6
  5. 2024FedDM: Enhancing Communication Efficiency and Handling Data Heterogeneity in Federated Diffusion Modelscited by 5
  6. 2025Diffusion Models for Computational Neuroimaging: A Surveycited by 5
  7. 2025FORLA: Federated Object-centric Representation Learning with Slot Attentioncited by 3
  8. 2025FedDifRC: Unlocking the Potential of Text-to-Image Diffusion Models in Heterogeneous Federated Learningcited by 3
  9. 2025M3amba: CLIP-driven Mamba Model for Multi-modal Remote Sensing Classificationcited by 2
  10. 2025Diffusion Models for Hyperspectral Image Analysis: A Comprehensive Reviewcited by 2
  11. 2024Enhancing Sample Efficiency and Exploration in Reinforcement Learning through the Integration of Diffusion Models and Proximal Policy Optimizationcited by 1
  12. 2024Understanding the Statistical Accuracy-Communication Trade-off in Personalized Federated Learning with Minimax Guaranteescited by 1
11 more
  1. 2025FedRS-Bench: Realistic Federated Learning Datasets and Benchmarks in Remote Sensingcited by 1
  2. 2025FedPhD: Federated Pruning with Hierarchical Learning of Diffusion Modelscited by 1
  3. 2025FedX: Explanation-Guided Pruning for Communication-Efficient Federated Learning in Remote Sensingcited by 1
  4. 2025High-Energy Concentration for Federated Learning in Frequency Domaincited by 1
  5. 2025Mix-modal Federated Learning for MRI Image Segmentationcited by 0
  6. 2025Federated Dialogue-Semantic Diffusion for Emotion Recognition under Incomplete Modalitiescited by 0
  7. 2026GFPL: Generative Federated Prototype Learning for Resource-Constrained and Data-Imbalanced Vision Taskcited by 0
  8. 2026FedEU: Evidential Uncertainty-Driven Federated Fine-Tuning of Vision Foundation Models for Remote Sensing Image Segmentationcited by 0
  9. 2026FedEHR-Gen: Federated Synthetic Time-Series EHR Generation via Latent Space Alignment and Distribution-Aware Aggregationcited by 0
  10. 2026FedOT: Ownership Verification and Leakage Tracing via Watermarks for Federated LDMscited by 0
  11. 2026Toward Interpretable Multimodal Fusion: Heat Conduction Modeling for Hyperspectral and LiDAR Joint Classificationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Balanced Diffusion-Guided Fusion for Multimodal Remote Sensing Classification
  2. 2025UniDiff: Parameter-Efficient Adaptation of Diffusion Models for Land Cover Classification with Multi-Modal Remotely Sensed Imagery and Sparse Annotations
  3. 2023DiffSpectralNet : Unveiling the Potential of Diffusion Models for Hyperspectral Image Classification
  4. 2023DDRF: Denoising Diffusion Model for Remote Sensing Image Fusion
  5. 2024P-MSDiff: Parallel Multi-Scale Diffusion for Remote Sensing Image Segmentation
  6. 2024Remote Diffusion
  7. 2023SpectralDiff: A Generative Framework for Hyperspectral Image Classification with Diffusion Models
  8. 2024SwiMDiff: Scene-wide Matching Contrastive Learning with Diffusion Constraint for Remote Sensing Image