mlsanityData to 7 Oct 2026

FedFed: Feature Distillation against Data Heterogeneity in Federated Learning

arXiv 2310.05077 ↗First posted 8 Oct 2023cs.LGcited by 54

Not on the map. Why?

Abstract (arXiv)

Federated learning (FL) typically faces data heterogeneity, i.e., distribution shifting among clients. Sharing clients' information has shown great potentiality in mitigating data heterogeneity, yet incurs a dilemma in preserving privacy and promoting model performance. To alleviate the dilemma, we raise a fundamental question: textitIs it possible to share partial features in the data to tackle data heterogeneity? In this work, we give an affirmative answer to this question by proposing a novel approach called Federated Feature distillation (FedFed). Specifically, FedFed partitions data into performance-sensitive features (i.e., greatly contributing to model performance) and performance-robust features (i.e., limitedly contributing to model performance). The performance-sensitive features are globally shared to mitigate data heterogeneity, while the performance-robust features are kept locally. FedFed enables clients to train models over local and shared data. Comprehensive experiments demonstrate the efficacy of FedFed in promoting model performance.

Key facts. Cited by 54 papers in the map (data through 7 Oct 2026). Builds on 52 of its 78 references that are in the map. First posted on arXiv 8 Oct 2023.

Builds on

52 of 78 references.

References · 52

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,592
  3. 2017Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithmscited by 4,283
  4. 2019Advances and Open Problems in Federated Learningcited by 3,021
  5. 2018Federated Optimization in Heterogeneous Networkscited by 2,971
  6. 2016Deep Learning with Differential Privacycited by 2,682
  7. 2016Membership Inference Attacks against Machine Learning Modelscited by 2,221
  8. 2018Federated Learning with Non-IID Datacited by 1,099
  9. 2019Deep Leakage from Gradientscited by 1,004
  10. 2020Tackling the Objective Inconsistency Problem in Heterogeneous Federated Optimizationcited by 897
  11. 2018The Secret Sharer: Evaluating and Testing Unintended Memorization in Neural Networkscited by 847
  12. 2019Measuring the Effects of Non-Identical Data Distribution for Federated Visual Classificationcited by 836
38 more
  1. 2015Deep Learning and the Information Bottleneck Principlecited by 793
  2. 2021Model-Contrastive Federated Learningcited by 762
  3. 2020Ensemble Distillation for Robust Model Fusion in Federated Learningcited by 678
  4. 2021Federated Learning on Non-IID Data Silos: An Experimental Studycited by 566
  5. 2021FedBN: Federated Learning on Non-IID Features via Local Batch Normalizationcited by 565
  6. 2020Federated Learning with Matched Averagingcited by 564
  7. 2019Federated Learning with Differential Privacy: Algorithms and Performance Analysiscited by 528
  8. 2017Opening the Black Box of Deep Neural Networks via Informationcited by 494
  9. 2019FedMD: Heterogenous Federated Learning via Model Distillationcited by 454
  10. 2019Bayesian Nonparametric Federated Learning of Neural Networkscited by 385
  11. 2019A Survey on Federated Learning Systems: Vision, Hype and Reality for Data Privacy and Protectioncited by 371
  12. 2021FedProto: Federated Prototype Learning across Heterogeneous Clientscited by 349
  13. 2013The Composition Theorem for Differential Privacycited by 290
  14. 2018Communication-Efficient On-Device Machine Learning: Federated Distillation and Augmentation under Non-IID Private Datacited by 242
  15. 2019The Secret Revealer: Generative Model-Inversion Attacks Against Deep Neural Networkscited by 241
  16. 2021No Fear of Heterogeneity: Classifier Calibration for Federated Learning with Non-IID Datacited by 241
  17. 2020Differentially Private Learning Needs Better Features (or Much More Data)cited by 233
  18. 2019Differentially Private Learning with Adaptive Clippingcited by 218
  19. 2020Client Selection in Federated Learning: Convergence Analysis and Power-of-Choice Selection Strategiescited by 196
  20. 2022Fine-tuning Global Model via Data-Free Knowledge Distillation for Non-IID Federated Learningcited by 172
  21. 2017Generative Adversarial Perturbationscited by 153
  22. 2020Federated Visual Classification with Real-World Data Distributioncited by 126
  23. 2020What makes instance discrimination good for transfer learning?cited by 111
  24. 2022Is Out-of-Distribution Detection Learnable?cited by 109
  25. 2022Federated Learning with Label Distribution Skew via Logits Calibrationcited by 108
  26. 2019Cronus: Robust and Heterogeneous Collaborative Learning with Black-Box Knowledge Transfercited by 91
  27. 2021FedAUX: Leveraging Unlabeled Auxiliary Data in Federated Learningcited by 67
  28. 2020Knowledge-Enriched Distributional Model Inversion Attackscited by 65
  29. 2023DP-Forward: Fine-tuning and Inference on Language Models with Differential Privacy in Forward Passcited by 65
  30. 2021Towards Fair Federated Learning with Zero-Shot Data Augmentationcited by 64
  31. 2020LDP-FL: Practical Private Aggregation in Federated Learning with Local Differential Privacycited by 63
  32. 2020XOR Mixup: Privacy-Preserving Data Augmentation for One-Shot Federated Learningcited by 58
  33. 2021Federated Social Recommendation with Graph Neural Networkcited by 56
  34. 2019G-PATE: Scalable Differentially Private Data Generator via Private Aggregation of Teacher Discriminatorscited by 53
  35. 2020Federated Learning via Synthetic Datacited by 53
  36. 2022Virtual Homogeneity Learning: Defending against Data Heterogeneity in Federated Learningcited by 52
  37. 2020Leakage of Dataset Properties in Multi-Party Machine Learningcited by 46
  38. 2021Differentially Private Federated Knowledge Graphs Embeddingcited by 45

50 most cited of 52 All 52 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2023: 1; 2024: 21; 2025: 20; 2026: 12.

54 papers.

Sort:

By area:

  1. 2024LoRA-FAIR: Federated LoRA Fine-Tuning with Aggregation and Initialization Refinementcited by 37
  2. 2024FedImpro: Measuring and Improving Client Update in Federated Learningcited by 15
  3. 2025FedALT: Federated Fine-Tuning through Adaptive Local Training with Rest-of-World LoRAcited by 13
  4. 2023Federated Learning with Extremely Noisy Clients via Negative Distillationcited by 12
  5. 2024Non-IID data in Federated Learning: A Survey with Taxonomy, Metrics, Methods, Frameworks and Future Directionscited by 12
  6. 2024FuseFL: One-Shot Federated Learning through the Lens of Causality with Progressive Model Fusioncited by 11
  7. 2025FedRSClip: Federated Learning for Remote Sensing Scene Classification Using Vision-Language Modelscited by 11
  8. 2024Federated Graph Condensation with Information Bottleneck Principlescited by 8
  9. 2024Handling Spatial-Temporal Data Heterogeneity for Federated Continual Learning via Tail Anchorcited by 8
  10. 2024PeFAD: A Parameter-Efficient Federated Framework for Time Series Anomaly Detectioncited by 7
  11. 2024Free Lunch for Federated Remote Sensing Target Fine-Grained Classification: A Parameter-Efficient Frameworkcited by 6
  12. 2024Robust Training of Federated Models with Extremely Label Deficiencycited by 6
38 more
  1. 2025Federated Learning for Diffusion Modelscited by 6
  2. 2025Federated Learning for Medical Image Classification: A Comprehensive Benchmarkcited by 6
  3. 2024FedCCL: Federated Dual-Clustered Feature Contrast Under Domain Heterogeneitycited by 5
  4. 2025FedGPS: Statistical Rectification Against Data Heterogeneity in Federated Learningcited by 4
  5. 2024FedHPL: Efficient Heterogeneous Federated Learning with Prompt Tuning and Logit Distillationcited by 3
  6. 2025ConcealGS: Concealing Invisible Copyright Information in 3D Gaussian Splattingcited by 3
  7. 2025AFCL: Analytic Federated Continual Learning for Spatio-Temporal Invariance of Non-IID Datacited by 3
  8. 2024FedDr+: Stabilizing Dot-regression with Global Feature Distillation for Federated Learningcited by 2
  9. 2024On-device Federated Learning in Smartphones for Detecting Depression from Reddit Postscited by 2
  10. 2024FedAli: Personalized Federated Learning Alignment with Prototype Layers for Generalized Mobile Servicescited by 2
  11. 2024Federated Hybrid Training and Self-Adversarial Distillation: Towards Robust Edge Networkscited by 2
  12. 2025Efficient Federated Fine-Tuning of Large Language Models with Layer Dropoutcited by 2
  13. 2024Unsupervised Speaker Diarization in Distributed IoT Networks Using Federated Learningcited by 1
  14. 2024FedCache 2.0: Federated Edge Learning with Knowledge Caching and Dataset Distillationcited by 1
  15. 2024Modality Alignment Meets Federated Broadcastingcited by 1
  16. 2025Privacy-Preserving Dataset Combinationcited by 1
  17. 2025Tackling Data Heterogeneity in Federated Learning through Knowledge Distillation with Inequitable Aggregationcited by 1
  18. 2025On the Fast Adaptation of Delayed Clients in Decentralized Federated Learning: A Centroid-Aligned Distillation Approachcited by 1
  19. 2025Adaptive Federated Distillation for Multi-Domain Non-IID Textual Datacited by 1
  20. 2025Federated CLIP for Resource-Efficient Heterogeneous Medical Image Classificationcited by 1
  21. 2026DeepAFL: Deep Analytic Federated Learningcited by 1
  22. 2026SEMASIA: A Large-Scale Dataset of Semantically Structured Latent Representationscited by 1
  23. 2026FedXDS: Leveraging Model Attribution Methods to counteract Data Heterogeneity in Federated Learningcited by 1
  24. 2024Client2Vec: Improving Federated Learning by Distribution Shifts Aware Client Indexingcited by 0
  25. 2024Learning Unlabeled Clients Divergence for Federated Semi-Supervised Learning via Anchor Model Aggregationcited by 0
  26. 2024UniFed: A Universal Federation of a Mixture of Highly Heterogeneous Medical Image Classification Taskscited by 0
  27. 2025Provably Near-Optimal Federated Ensemble Distillation with Negligible Overheadcited by 0
  28. 2025Guided Model Merging for Hybrid Data Learning: Leveraging Centralized Data to Refine Decentralized Modelscited by 0
  29. 2025Harmonizing Generalization and Personalization in Ring-topology Decentralized Federated Learningcited by 0
  30. 2025PC-MoE: Memory-Efficient and Privacy-Preserving Collaborative Training for Mixture-of-Experts LLMscited by 0
  31. 2025ILoRA: Federated Learning with Low-Rank Adaptation for Heterogeneous Client Aggregationcited by 0
  32. 2025Prediction-space knowledge markets for communication-efficient federated learning on multimedia taskscited by 0
  33. 2025Single-Round Scalable Analytic Federated Learningcited by 0
  34. 2026DeepFusion: Accelerating MoE Training via Federated Knowledge Distillation from Heterogeneous Edge Devicescited by 0
  35. 2026Heterogeneity-Aware Client Selection Methodology For Efficient Federated Learningcited by 0
  36. 2026FedRG: Unleashing the Representation Geometry for Federated Learning with Noisy Clientscited by 0
  37. 2026From Data Heterogeneity to Convergence: A Data-Centric Review of Federated Learningcited by 0
  38. 2026Collaborative Synthetic Data Generation for Knowledge Transfer in Federated Learningcited by 0

50 most cited of 54 All 54 (JSON)