Communication-Efficient On-Device Machine Learning: Federated Distillation and Augmentation under Non-IID Private Data
Not on the map. Why?
Abstract (arXiv)
On-device machine learning (ML) enables the training process to exploit a massive amount of user-generated private data samples. To enjoy this benefit, inter-device communication overhead should be minimized. With this end, we propose federated distillation (FD), a distributed model training algorithm whose communication payload size is much smaller than a benchmark scheme, federated learning (FL), particularly when the model size is large. Moreover, user-generated data samples are likely to become non-IID across devices, which commonly degrades the performance compared to the case with an IID dataset. To cope with this, we propose federated augmentation (FAug), where each device collectively trains a generative model, and thereby augments its local data towards yielding an IID dataset. Empirical studies demonstrate that FD with FAug yields around 26x less communication overhead while achieving 95-98% test accuracy compared to FL.
Key facts. Cited by 242 papers in the map (data through 7 Oct 2026). First posted on arXiv 28 Nov 2018.
Builds on
No reference list yet.
Built on by
Citing papers per year: 2020: 1; 2021: 40; 2022: 37; 2023: 49; 2024: 51; 2025: 44; 2026: 20.
242 papers.
Sort:
By area:
- 2021Towards Personalized Federated Learningcited by 462
- 2021Data-Free Knowledge Distillation for Heterogeneous Federated Learningcited by 380
- 2021FedProto: Federated Prototype Learning across Heterogeneous Clientscited by 349
- 2021No Fear of Heterogeneity: Classifier Calibration for Federated Learning with Non-IID Datacited by 241
- 2021Achieving Linear Speedup with Partial Worker Participation in Non-IID Federated Learningcited by 192
- 2023Heterogeneous Federated Learning: State-of-the-art and Research Challengescited by 160
- 2021Federated Learning for Open Bankingcited by 138
- 2021Distributed Learning in Wireless Networks: Recent Progress and Future Challengescited by 125
- 2021Federated Graph Classification over Non-IID Graphscited by 113
- 2021Multi-Center Federated Learning: Clients Clustering for Better Personalizationcited by 112
- 2021Parameterized Knowledge Transfer for Personalized Federated Learningcited by 108
- 2023Federated Learning for Generalization, Robustness, Fairness: A Survey and Benchmarkcited by 97
38 more
- 2023Deep Model Fusion: A Surveycited by 89
- 2021FedAUX: Leveraging Unlabeled Auxiliary Data in Federated Learningcited by 67
- 2021Towards Fair Federated Learning with Zero-Shot Data Augmentationcited by 64
- 2023FedGH: Heterogeneous Federated Learning with Generalized Global Headercited by 56
- 2023FedFed: Feature Distillation against Data Heterogeneity in Federated Learningcited by 54
- 2022Virtual Homogeneity Learning: Defending against Data Heterogeneity in Federated Learningcited by 52
- 2024FedTGP: Trainable Global Prototypes with Adaptive-Margin-Enhanced Contrastive Learning for Data and Model Heterogeneity in Federated Learningcited by 47
- 2022Preserving Privacy in Federated Learning with Ensemble Cross-Domain Knowledge Distillationcited by 45
- 2022A Survey on Heterogeneous Federated Learningcited by 42
- 2023Federated Learning for Connected and Automated Vehicles: A Survey of Existing Approaches and Challengescited by 39
- 2023pFedLoRA: Model-Heterogeneous Personalized Federated Learning with LoRA Tuningcited by 38
- 2021Emerging Trends in Federated Learning: From Model Fusion to Federated X Learningcited by 35
- 2024Towards Efficient Replay in Federated Incremental Learningcited by 33
- 2021FedZKT: Zero-Shot Knowledge Transfer towards Resource-Constrained Federated Learning with Heterogeneous On-Device Modelscited by 31
- 2022Knowledge Distillation for Federated Learning: a Practical Guidecited by 31
- 2022FedDUAP: Federated Learning with Dynamic Update and Adaptive Pruning Using Shared Data on the Servercited by 29
- 2023Integration of Large Language Models and Federated Learningcited by 28
- 2023Privacy and Fairness in Federated Learning: on the Perspective of Trade-offcited by 26
- 2024FeDeRA:Efficient Fine-tuning of Language Models in Federated Learning Leveraging Weight Decompositioncited by 26
- 2022Data Heterogeneity-Robust Federated Learning via Group Client Selection in Industrial IoTcited by 25
- 2022Federated Learning with GAN-based Data Synthesis for Non-IID Clientscited by 25
- 2023The Best of Both Worlds: Accurate Global and Personalized Models through Federated Learning with Data-Free Hyper-Knowledge Distillationcited by 25
- 2023Federated Domain Generalization: A Surveycited by 25
- 2022FRAug: Tackling Federated Learning with Non-IID Features via Representation Augmentationcited by 24
- 2022Rethinking Data Heterogeneity in Federated Learning: Introducing a New Notion and Standard Benchmarkscited by 24
- 2023A Survey of Trustworthy Federated Learning with Perspectives on Security, Robustness, and Privacycited by 24
- 2020Specialized federated learning using a mixture of expertscited by 23
- 2021Decentralized federated learning of deep neural networks on non-iid datacited by 23
- 2022Exploring the Distributed Knowledge Congruence in Proxy-data-free Federated Distillationcited by 22
- 2023Personalized Decentralized Federated Learning with Knowledge Distillationcited by 21
- 2023A Survey of What to Share in Federated Learning: Perspectives on Model Utility, Privacy Leakage, and Communication Efficiencycited by 21
- 2024pFedMoE: Data-Level Personalization with Mixture of Experts for Model-Heterogeneous Personalized Federated Learningcited by 21
- 2024Overcoming Data and Model Heterogeneities in Decentralized Federated Learning via Synthetic Anchorscited by 21
- 2024Federated Model Heterogeneous Matryoshka Representation Learningcited by 21
- 2024Personalized Federated Domain-Incremental Learning based on Adaptive Knowledge Matchingcited by 21
- 2021Auto-FedAvg: Learnable Federated Averaging for Multi-Institutional Medical Image Segmentationcited by 19
- 2021Architectural Patterns for the Design of Federated Learning Systemscited by 17
- 2021Decentralized and Model-Free Federated Learning: Consensus-Based Distillation in Function Spacecited by 17
50 most cited of 242 All 242 (JSON)
- 2026Upholding Robustness in Federated Learning: Trends, Emerging Strategies, and Research Opportunities23 Sep 2026
- 2026Out-of-Distribution Federated Distillation with Domain-Aware Proxy9 Aug 2026
- 2026EFFEKT: Efficient Federated Knowledge Transfer to Foundation Models8 Aug 2026
- 2026FedDOSE: Federated Learning Framework Decomposing Site Effects for Modeling Brain Dynamic Functional Connectivity7 Aug 2026
- 2026KV Cache Translation across Heterogeneous Large Language Models31 Jul 2026
- 2026FedTopo: Relation-Level Topology Sharing for Model-Heterogeneous Federated Learning29 Jul 2026
- 2026Collate: Collaborative Neural Network Learning for Latency-Critical Edge Systems9 Jul 2026
- 2026TallyTrain: Communication-Efficient Federated Distillation30 Jun 2026
- 2026Optimized Federated Knowledge Distillation with Distributed Neural Architecture Search20 May 2026
- 2026BiFedKD: Bidirectional Federated Knowledge Distillation Framework for Non-IID and Long-Tailed ECG Monitoring14 May 2026
- 2026Function-Space ADMM for Decentralized Federated Learning: A Control Theoretic Perspective10 May 2026
- 2026From Coordinate Matching to Structural Alignment: Rethinking Prototype Alignment in Heterogeneous Federated Learning7 May 2026
38 more
- 2026Internet of Everything in the 6G Era: Paradigms, Enablers, Potentials and Future Directions27 Apr 2026
- 2026On-Orbit Space AI: Federated, Multi-Agent, and Collaborative Algorithms for Satellite Constellations15 Apr 2026
- 2026PubSwap: Public-Data Off-Policy Coordination for Federated RLVR14 Apr 2026
- 2026FedDBP: Enhancing Federated Prototype Learning with Dual-Branch Features and Personalized Global Fusion31 Mar 2026
- 2026Social Hippocampus Memory Learning26 Mar 2026
- 2026SCOPE: Semantic Coreset with Orthogonal Projection Embeddings for Federated learning13 Mar 2026
- 2026FedEMA-Distill: Exponential Moving Average Guided Knowledge Distillation for Robust Federated Learning15 Feb 2026
- 2026Hybrid Federated Learning for Noise-Robust Training8 Jan 2026
- 2025FedDPC : Handling Data Heterogeneity and Partial Client Participation in Federated Learning23 Dec 2025
- 2025Stragglers Can Contribute More: Uncertainty-Aware Distillation for Asynchronous Federated Learning25 Nov 2025
- 2025FedDEAP: Adaptive Dual-Prompt Tuning for Multi-Domain Federated Learning21 Oct 2025
- 2025TAP: Two-Stage Adaptive Personalization of Multi-Task and Multi-Modal Foundation Models in Federated Learning30 Sep 2025
- 2025Adaptive Dual-Mode Distillation with Incentive Schemes for Scalable, Heterogeneous Federated Learning on Non-IID Data26 Sep 2025
- 2025Adaptive Federated Distillation for Multi-Domain Non-IID Textual Data28 Aug 2025
- 2025Enhancing Model Privacy in Federated Learning with Random Masking and Quantization26 Aug 2025
- 2025Federated Distillation on Edge Devices: Efficient Client-Side Filtering for Non-IID Data20 Aug 2025
- 2025FedPromo: Federated Lightweight Proxy Models at the Edge Bring New Domains to Foundation Models5 Aug 2025
- 2025Soft Separation and Distillation: Toward Global Uniformity in Federated Unsupervised Learning2 Aug 2025
- 2025Hypernetworks for Model-Heterogeneous Personalized Federated Learning30 Jul 2025
- 2025Optimal Transceiver Design in Over-the-Air Federated Distillation21 Jul 2025
- 2025FedWCM: Unleashing the Potential of Momentum-based Federated Learning in Long-Tailed Scenarios20 Jul 2025
- 2025Federated Learning for Commercial Image Sources17 Jul 2025
- 2025Feature Distillation is the Better Choice for Model-Heterogeneous Federated Learning14 Jul 2025
- 2025TinyProto: Communication-Efficient Federated Learning with Sparse Prototypes in Resource-Constrained Environments6 Jul 2025
- 2025Heterogeneous Federated Learning with Prototype Alignment and Upscaling6 Jul 2025
- 2025FedCGD: Collective Gradient Divergence Optimized Scheduling for Wireless Federated Learning9 Jun 2025
- 2025HtFLlib: A Comprehensive Heterogeneous Federated Learning Library and Benchmark4 Jun 2025
- 2025Multimodal Federated Learning: A Survey through the Lens of Different FL Paradigms27 May 2025
- 2025Federated Learning-Distillation Alternation for Resource-Constrained IoT26 May 2025
- 2025Federated Learning for Cyber Physical Systems: A Comprehensive Survey8 May 2025
- 2025Soft-Label Caching and Sharpening for Communication-Efficient Federated Distillation28 Apr 2025
- 2025Privacy-Preserving Federated Embedding Learning for Localized Retrieval-Augmented Generation27 Apr 2025
- 2025Learning Critically: Selective Self Distillation in Federated Learning on Non-IID Data20 Apr 2025
- 2025Federated Prototype Graph Learning13 Apr 2025
- 2025Personalizing Federated Learning for Hierarchical Edge Networks with Non-IID Data11 Apr 2025
- 2025Corrected with the Latest Version: Make Robust Asynchronous Federated Learning Possible5 Apr 2025
- 2025Hierarchical Knowledge Structuring for Effective Federated Learning in Heterogeneous Environments4 Apr 2025
- 2025AIGC-assisted Federated Learning for Edge Intelligence: Architecture Design, Research Challenges and Future Directions26 Mar 2025
50 newest of 242 All 242 (JSON)
- 2020Specialized federated learning using a mixture of expertscited by 23
- 2024pFedMoE: Data-Level Personalization with Mixture of Experts for Model-Heterogeneous Personalized Federated Learningcited by 21
- 2024FedMoE-DA: Federated Mixture of Experts via Domain Aware Fine-grained Aggregationcited by 11
- 2022Adaptive Expert Models for Personalization in Federated Learningcited by 4
- 2024FedMoE-DA: Federated Mixture of Experts via Domain Aware Fine-grained Aggregation4 Nov 2024
- 2024pFedMoE: Data-Level Personalization with Mixture of Experts for Model-Heterogeneous Personalized Federated Learning2 Feb 2024
- 2022Adaptive Expert Models for Personalization in Federated Learning15 Jun 2022
- 2020Specialized federated learning using a mixture of experts5 Oct 2020
- 2021Towards Personalized Federated Learningcited by 462
- 2021Data-Free Knowledge Distillation for Heterogeneous Federated Learningcited by 380
- 2021FedProto: Federated Prototype Learning across Heterogeneous Clientscited by 349
- 2021No Fear of Heterogeneity: Classifier Calibration for Federated Learning with Non-IID Datacited by 241
- 2021Achieving Linear Speedup with Partial Worker Participation in Non-IID Federated Learningcited by 192
- 2023Heterogeneous Federated Learning: State-of-the-art and Research Challengescited by 160
- 2021Federated Learning for Open Bankingcited by 138
- 2021Distributed Learning in Wireless Networks: Recent Progress and Future Challengescited by 125
- 2021Federated Graph Classification over Non-IID Graphscited by 113
- 2021Multi-Center Federated Learning: Clients Clustering for Better Personalizationcited by 112
- 2021Parameterized Knowledge Transfer for Personalized Federated Learningcited by 108
- 2023Federated Learning for Generalization, Robustness, Fairness: A Survey and Benchmarkcited by 97
12 most cited of 233 All 233 (JSON)
- 2026Upholding Robustness in Federated Learning: Trends, Emerging Strategies, and Research Opportunities23 Sep 2026
- 2026Out-of-Distribution Federated Distillation with Domain-Aware Proxy9 Aug 2026
- 2026EFFEKT: Efficient Federated Knowledge Transfer to Foundation Models8 Aug 2026
- 2026FedDOSE: Federated Learning Framework Decomposing Site Effects for Modeling Brain Dynamic Functional Connectivity7 Aug 2026
- 2026FedTopo: Relation-Level Topology Sharing for Model-Heterogeneous Federated Learning29 Jul 2026
- 2026Collate: Collaborative Neural Network Learning for Latency-Critical Edge Systems9 Jul 2026
- 2026TallyTrain: Communication-Efficient Federated Distillation30 Jun 2026
- 2026Optimized Federated Knowledge Distillation with Distributed Neural Architecture Search20 May 2026
- 2026BiFedKD: Bidirectional Federated Knowledge Distillation Framework for Non-IID and Long-Tailed ECG Monitoring14 May 2026
- 2026Function-Space ADMM for Decentralized Federated Learning: A Control Theoretic Perspective10 May 2026
- 2026From Coordinate Matching to Structural Alignment: Rethinking Prototype Alignment in Heterogeneous Federated Learning7 May 2026
- 2026Internet of Everything in the 6G Era: Paradigms, Enablers, Potentials and Future Directions27 Apr 2026
12 newest of 233 All 233 (JSON)