mlsanityData to 7 Oct 2026

FedAUX: Leveraging Unlabeled Auxiliary Data in Federated Learning

arXiv 2102.02514 ↗First posted 4 Feb 2021cs.LG cs.DC stat.MLcited by 67

Not on the map. Why?

Abstract (arXiv)

Federated Distillation (FD) is a popular novel algorithmic paradigm for Federated Learning, which achieves training performance competitive to prior parameter averaging based methods, while additionally allowing the clients to train different model architectures, by distilling the client predictions on an unlabeled auxiliary set of data into a student model. In this work we propose FedAUX, an extension to FD, which, under the same set of assumptions, drastically improves performance by deriving maximum utility from the unlabeled auxiliary data. FedAUX modifies the FD training procedure in two ways: First, unsupervised pre-training on the auxiliary data is performed to find a model initialization for the distributed training. Second, (varepsilon, δ)-differentially private certainty scoring is used to weight the ensemble predictions on the auxiliary data according to the certainty of each client model. Experiments on large-scale convolutional neural networks and transformer models demonstrate, that the training performance of FedAUX exceeds SOTA FL baseline methods by a substantial margin in both the iid and non-iid regime, further closing the gap to centralized training performance. Code is available at github.com/fedl-repo/fedaux.

Key facts. Cited by 67 papers in the map (data through 7 Oct 2026). Builds on 52 of its 86 references that are in the map. First posted on arXiv 4 Feb 2021.

Builds on

52 of 86 references.

References · 52

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2018BERT: Pre-training of Deep Bidirectional Transformers for Language Understandingcited by 40,204
  3. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  4. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  5. 2015Distilling the Knowledge in a Neural Networkcited by 10,076
  6. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,592
  7. 2018MobileNetV2: Inverted Residuals and Linear Bottleneckscited by 4,773
  8. 2016Pointer Sentinel Mixture Modelscited by 2,992
  9. 2018Federated Optimization in Heterogeneous Networkscited by 2,971
  10. 2016Deep Learning with Differential Privacycited by 2,682
  11. 2015Character-level Convolutional Networks for Text Classificationcited by 2,293
  12. 2020Understanding Contrastive Representation Learning through Alignment and Uniformity on the Hyperspherecited by 1,393
38 more
  1. 2019On the Convergence of FedAvg on Non-IID Datacited by 1,220
  2. 2017ShuffleNet: An Extremely Efficient Convolutional Neural Network for Mobile Devicescited by 1,194
  3. 2019TinyBERT: Distilling BERT for Natural Language Understandingcited by 1,153
  4. 2018Federated Learning with Non-IID Datacited by 1,099
  5. 2015Aligning Books and Movies: Towards Story-like Visual Explanations by Watching Movies and Reading Bookscited by 1,041
  6. 2020Adaptive Federated Optimizationcited by 1,036
  7. 2019Measuring the Effects of Non-Identical Data Distribution for Federated Visual Classificationcited by 836
  8. 2020Ensemble Distillation for Robust Model Fusion in Federated Learningcited by 678
  9. 2017Federated Multi-Task Learningcited by 669
  10. 2019Clustered Federated Learning: Model-Agnostic Distributed Multi-Task Optimization under Privacy Constraintscited by 499
  11. 2019FedMD: Heterogenous Federated Learning via Model Distillationcited by 454
  12. 2019Agnostic Federated Learningcited by 451
  13. 2020A Unifying Review of Deep and Shallow Anomaly Detectioncited by 423
  14. 2019Robust and Communication-Efficient Federated Learning from Non-IID Datacited by 415
  15. 2019A Survey on Federated Learning Systems: Vision, Hype and Reality for Data Privacy and Protectioncited by 371
  16. 2018Analyzing Federated Learning through an Adversarial Lenscited by 346
  17. 2020Three Approaches for Personalization with Applications to Federated Learningcited by 321
  18. 2019Deep Semi-Supervised Anomaly Detectioncited by 290
  19. 2018Scalable Private Learning with PATEcited by 266
  20. 2018Communication-Efficient On-Device Machine Learning: Federated Distillation and Augmentation under Non-IID Private Datacited by 242
  21. 2020FedBE: Making Bayesian Model Ensemble Applicable to Federated Learningcited by 176
  22. 2019Federated Evaluation of On-device Personalizationcited by 147
  23. 2018Large scale distributed neural network training through online distillationcited by 144
  24. 2020The Multilingual Amazon Reviews Corpuscited by 142
  25. 2020Distillation-Based Semi-Supervised Federated Learning for Communication-Efficient Collaborative Training with Non-IID Private Datacited by 125
  26. 2019One-Shot Federated Learningcited by 124
  27. 2019Zero-Shot Knowledge Distillation in Deep Networkscited by 119
  28. 2020Federated Semi-Supervised Learning with Inter-Client Consistency & Disjoint Learningcited by 112
  29. 2020Distilled One-Shot Federated Learningcited by 108
  30. 2020Federated Unsupervised Representation Learningcited by 92
  31. 2019Cronus: Robust and Heterogeneous Collaborative Learning with Black-Box Knowledge Transfercited by 91
  32. 2019Robust Federated Learning in a Heterogeneous Environmentcited by 88
  33. 2018Algorithms and Theory for Multiple-Source Adaptationcited by 53
  34. 2020Federated Knowledge Distillationcited by 51
  35. 2020Federated Model Distillation with Noise-Free Differential Privacycited by 40
  36. 2019Wireless Federated Distillation for Distributed Edge Learning with Heterogeneous Datacited by 39
  37. 2020Communication-Efficient Federated Distillationcited by 28
  38. 2021FedH2L: Federated Learning with Model and Statistical Heterogeneitycited by 17

50 most cited of 52 All 52 (JSON)

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2021: 4; 2022: 11; 2023: 22; 2024: 18; 2025: 8; 2026: 4.

67 papers.

Sort:

By area:

  1. 2021Data-Free Knowledge Distillation for Heterogeneous Federated Learningcited by 380
  2. 2022A Survey of Machine Unlearningcited by 298
  3. 2022Fine-tuning Global Model via Data-Free Knowledge Distillation for Non-IID Federated Learningcited by 172
  4. 2023Deep Model Fusion: A Surveycited by 89
  5. 2023FedGH: Heterogeneous Federated Learning with Generalized Global Headercited by 56
  6. 2022Personalized Subgraph Federated Learningcited by 54
  7. 2023FedFed: Feature Distillation against Data Heterogeneity in Federated Learningcited by 54
  8. 2023pFedLoRA: Model-Heterogeneous Personalized Federated Learning with LoRA Tuningcited by 38
  9. 2021Local-Global Knowledge Distillation in Heterogeneous Federated Learning with Non-IID Datacited by 31
  10. 2021FedZKT: Zero-Shot Knowledge Transfer towards Resource-Constrained Federated Learning with Heterogeneous On-Device Modelscited by 31
  11. 2022Knowledge Distillation for Federated Learning: a Practical Guidecited by 31
  12. 2023Teacher-Student Architecture for Knowledge Distillation: A Surveycited by 26
38 more
  1. 2022FedCross: Towards Accurate Federated Learning via Multi-Model Cross-Aggregationcited by 21
  2. 2024pFedMoE: Data-Level Personalization with Mixture of Experts for Model-Heterogeneous Personalized Federated Learningcited by 21
  3. 2024Federated Model Heterogeneous Matryoshka Representation Learningcited by 21
  4. 2022One-shot Federated Learning without Server-side Trainingcited by 20
  5. 2024Federated Distillation: A Surveycited by 14
  6. 2022Federated Stain Normalization for Computational Pathologycited by 11
  7. 2023Federated Learning Across Decentralized and Unshared Archives for Remote Sensing Image Classificationcited by 10
  8. 2023FedSSA: Semantic Similarity-based Aggregation for Efficient Model-Heterogeneous Personalized Federated Learningcited by 10
  9. 2024Stable Diffusion-based Data Augmentation for Federated Learning with Non-IID Datacited by 10
  10. 2022Vertical Federated Principal Component Analysis and Its Kernel Extension on Feature-wise Distributed Datacited by 9
  11. 2023pFedES: Model Heterogeneous Personalized Federated Learning with Feature Extractor Sharingcited by 9
  12. 2023TinyHD: Efficient Video Saliency Prediction with Heterogeneous Decoders using Hierarchical Maps Distillationcited by 8
  13. 2024Knowledge Distillation in Federated Learning: a Survey on Long Lasting Challenges and New Solutionscited by 8
  14. 2024Federated Learning with Label-Masking Distillationcited by 7
  15. 2023Continual Adaptation of Vision Transformers for Federated Learningcited by 6
  16. 2023FilFL: Client Filtering for Optimized Client Participation in Federated Learningcited by 5
  17. 2023A Practical Recipe for Federated Learning Under Statistical Heterogeneity Experimental Designcited by 5
  18. 2023Internal Cross-layer Gradients for Extending Homogeneity to Heterogeneity in Federated Learningcited by 5
  19. 2023Jointly Exploring Client Drift and Catastrophic Forgetting in Dynamic Learningcited by 5
  20. 2023Leveraging Foundation Models to Improve Lightweight Clients in Federated Learningcited by 5
  21. 2023Contrastive encoder pre-training-based clustered federated learning for heterogeneous datacited by 5
  22. 2024On the Byzantine-Resilience of Distillation-Based Federated Learningcited by 5
  23. 2024Towards Diverse Device Heterogeneous Federated Learning via Task Arithmetic Knowledge Integrationcited by 5
  24. 2024Decaf: Data Distribution Decompose Attack against Federated Learningcited by 4
  25. 2024Exploring the Practicality of Federated Learning: A Survey Towards the Communication Perspectivecited by 4
  26. 2024Federated Learning Client Pruning for Noisy Labelscited by 4
  27. 2022FedAUXfdp: Differentially Private One-Shot Federated Distillationcited by 3
  28. 2023Confidence-based federated distillation for vision-based lane-centeringcited by 3
  29. 2023Towards Open Federated Learning Platforms: Survey and Vision from Technical and Legal Perspectivescited by 3
  30. 2023SwiftLearn: A Data-Efficient Training Method of Deep Learning Models using Importance Samplingcited by 3
  31. 2024Federated Learning over Connected Modescited by 3
  32. 2024FedDistill: Global Model Distillation for Local Model De-Biasing in Non-IID Federated Learningcited by 3
  33. 2024pFedAFM: Adaptive Feature Mixture for Batch-Level Personalization in Heterogeneous Federated Learningcited by 3
  34. 2024A Unified Solution to Diverse Heterogeneities in One-shot Federated Learningcited by 3
  35. 2021Reward-Based 1-bit Compressed Federated Distillation on Blockchaincited by 2
  36. 2022Exploiting Features and Logits in Heterogeneous Federated Learningcited by 2
  37. 2023Bayesian Personalized Federated Learning with Shared and Personalized Uncertainty Representationscited by 2
  38. 2024Practical Insights into Knowledge Distillation for Pre-Trained Modelscited by 2

50 most cited of 67 All 67 (JSON)