mlsanityData to 7 Oct 2026

CoDream: Exchanging dreams instead of models for federated aggregation with heterogeneous models

arXiv 2402.15968 ↗First posted 25 Feb 2024cs.LG cs.AIcited by 0

Not on the map. Why?

Abstract (arXiv)

Federated Learning (FL) enables collaborative optimization of machine learning models across decentralized data by aggregating model parameters. Our approach extends this concept by aggregating "knowledge" derived from models, instead of model parameters. We present a novel framework called CoDream, where clients collaboratively optimize randomly initialized data using federated optimization in the input data space, similar to how randomly initialized model parameters are optimized in FL. Our key insight is that jointly optimizing this data can effectively capture the properties of the global data distribution. Sharing knowledge in data space offers numerous benefits: (1) model-agnostic collaborative learning, i.e., different clients can have different model architectures; (2) communication that is independent of the model size, eliminating scalability concerns with model parameters; (3) compatibility with secure aggregation, thus preserving the privacy benefits of federated learning; (4) allowing of adaptive optimization of knowledge shared for personalized learning. We empirically validate CoDream on standard FL tasks, demonstrating competitive performance despite not sharing model parameters. Our code: https://mitmedialab.github.io/codream.github.io/

Key facts. No citing papers in the map yet (data through 7 Oct 2026). Builds on 30 of its 44 references that are in the map. First posted on arXiv 25 Feb 2024.

Builds on

30 of 44 references.

References · 30

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  3. 2014Very Deep Convolutional Networks for Large-Scale Image Recognitioncited by 15,267
  4. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  5. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,592
  6. 2016Wide Residual Networkscited by 3,057
  7. 2018Federated Optimization in Heterogeneous Networkscited by 2,971
  8. 2021Model-Contrastive Federated Learningcited by 762
  9. 2020Ensemble Distillation for Robust Model Fusion in Federated Learningcited by 678
  10. 2016Federated Optimization: Distributed Machine Learning for On-Device Intelligencecited by 558
  11. 2019Dreaming to Distill: Data-free Knowledge Transfer via DeepInversioncited by 419
  12. 2020HeteroFL: Computation and Communication Efficient Federated Learning for Heterogeneous Clientscited by 334
18 more
  1. 2017Deep Models Under the GAN: Information Leakage from Collaborative Deep Learningcited by 326
  2. 2018Communication-Efficient On-Device Machine Learning: Federated Distillation and Augmentation under Non-IID Private Datacited by 242
  3. 2021FjORD: Fair and Accurate Federated Learning under heterogeneous targets with Ordered Dropoutcited by 192
  4. 2020FedBE: Making Bayesian Model Ensemble Applicable to Federated Learningcited by 176
  5. 2022Fine-tuning Global Model via Data-Free Knowledge Distillation for Non-IID Federated Learningcited by 172
  6. 2018Expanding the Reach of Federated Learning by Reducing Client Resource Requirementscited by 162
  7. 2022Reconstructing Training Data from Trained Neural Networkscited by 129
  8. 2020Distilled One-Shot Federated Learningcited by 108
  9. 2022FedRolex: Model-Heterogeneous Federated Learning with Rolling Sub-Model Extractioncited by 102
  10. 2023Stochastic Controlled Averaging for Federated Learning with Communication Compressioncited by 100
  11. 2019Cronus: Robust and Heterogeneous Collaborative Learning with Black-Box Knowledge Transfercited by 91
  12. 2021Preservation of the Global Knowledge by Not-True Distillation in Federated Learningcited by 87
  13. 2021FedAUX: Leveraging Unlabeled Auxiliary Data in Federated Learningcited by 67
  14. 2020Federated Learning via Synthetic Datacited by 53
  15. 2022Federated Learning via Decentralized Dataset Distillation in Resource-Constrained Edge Environmentscited by 49
  16. 2021Towards Model Agnostic Federated Learning Using Knowledge Distillationcited by 31
  17. 2022Knowledge Distillation for Federated Learning: a Practical Guidecited by 31
  18. 2022Scalable Collaborative Learning via Representation Sharingcited by 3

Built on by

Not cited yet.