mlsanityData to 7 Oct 2026

You Are Your Own Best Teacher: Achieving Centralized-level Performance in Federated Learning under Heterogeneous and Long-tailed Data

arXiv 2503.06916 ↗First posted 10 Mar 2025cs.LGcited by 5

Not on the map. Why?

Abstract (arXiv)

Data heterogeneity, stemming from local non-IID data and global long-tailed distributions, is a major challenge in federated learning (FL), leading to significant performance gaps compared to centralized learning. Previous research found that poor representations and biased classifiers are the main problems and proposed neural-collapse-inspired synthetic simplex ETF to help representations be closer to neural collapse optima. However, we find that the neural-collapse-inspired methods are not strong enough to reach neural collapse and still have huge gaps to centralized training. In this paper, we rethink this issue from a self-bootstrap perspective and propose FedYoYo (You Are Your Own Best Teacher), introducing Augmented Self-bootstrap Distillation (ASD) to improve representation learning by distilling knowledge between weakly and strongly augmented local samples, without needing extra datasets or models. We further introduce Distribution-aware Logit Adjustment (DLA) to balance the self-bootstrap process and correct biased feature representations. FedYoYo nearly eliminates the performance gap, achieving centralized-level performance even under mixed heterogeneity. It enhances local representation learning, reducing model drift and improving convergence, with feature prototypes closer to neural collapse optimality. Extensive experiments show FedYoYo achieves state-of-the-art results, even surpassing centralized logit adjustment methods by 5.4% under global long-tailed settings.

Key facts. Cited by 5 papers in the map (data through 7 Oct 2026). Builds on 34 of its 46 references that are in the map. First posted on arXiv 10 Mar 2025.

Builds on

34 of 46 references.

References · 34

  1. 2020A Simple Framework for Contrastive Learning of Visual Representationscited by 11,979
  2. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,592
  3. 2020Bootstrap your own latent: A new approach to self-supervised Learningcited by 4,712
  4. 2018Federated Optimization in Heterogeneous Networkscited by 2,971
  5. 2019RandAugment: Practical automated data augmentation with a reduced search spacecited by 2,261
  6. 2019Decoupling Representation and Classifier for Long-Tailed Recognitioncited by 806
  7. 2021Model-Contrastive Federated Learningcited by 762
  8. 2018AutoAugment: Learning Augmentation Policies from Datacited by 728
  9. 2020Prevalence of Neural Collapse during the terminal phase of deep learning trainingcited by 713
  10. 2020Ensemble Distillation for Robust Model Fusion in Federated Learningcited by 678
  11. 2020Long-tail learning via logit adjustmentcited by 605
  12. 2019Be Your Own Teacher: Improve the Performance of Convolutional Neural Networks via Self Distillationcited by 491
22 more
  1. 2021Deep Long-Tailed Learning: A Surveycited by 416
  2. 2021Data-Free Knowledge Distillation for Heterogeneous Federated Learningcited by 380
  3. 2021Understanding Dimensional Collapse in Contrastive Self-supervised Learningcited by 374
  4. 2021No Fear of Heterogeneity: Classifier Calibration for Federated Learning with Non-IID Datacited by 241
  5. 2021TrivialAugment: Tuning-free Yet State-of-the-Art Data Augmentationcited by 225
  6. 2022Fine-tuning Global Model via Data-Free Knowledge Distillation for Non-IID Federated Learningcited by 172
  7. 2022Federated Learning with Label Distribution Skew via Logits Calibrationcited by 108
  8. 2022Divergence-aware Federated Self-Supervised Learningcited by 71
  9. 2022Tackling Data Heterogeneity in Federated Learning with Class Prototypescited by 66
  10. 2022Federated Learning on Heterogeneous and Long-Tailed Data via Classifier Re-Training with Federated Featurescited by 63
  11. 2023Revisiting Weighted Aggregation in Federated Learning with Neural Networkscited by 57
  12. 2023No Fear of Classifier Biases: Neural Collapse Inspired Federated Learning with Synthetic and Fixed Classifiercited by 55
  13. 2021Federated Learning for Internet of Things: Applications, Challenges, and Opportunitiescited by 49
  14. 2022FedFA: Federated Learning with Feature Anchors to Align Features and Classifiers for Heterogeneous Datacited by 49
  15. 2024FedTGP: Trainable Global Prototypes with Adaptive-Margin-Enhanced Contrastive Learning for Data and Model Heterogeneity in Federated Learningcited by 47
  16. 2022Understanding Collapse in Non-Contrastive Siamese Representation Learningcited by 42
  17. 2022FedFM: Anchor-based Feature Matching for Data Heterogeneity in Federated Learningcited by 27
  18. 2022FEDIC: Federated Learning on Non-IID and Long-Tailed Data via Calibrated Distillationcited by 19
  19. 2023Fed-GraB: Federated Long-tailed Learning with Self-Adjusting Gradient Balancercited by 17
  20. 2024Rethinking the Representation in Federated Unsupervised Learning with Non-IID Datacited by 12
  21. 2024FedLoGe: Joint Local and Generic Federated Learning under Long-tailed Datacited by 10
  22. 2024FedUV: Uniformity and Variance for Heterogeneous Federated Learningcited by 8

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2026: 5.

5 papers.

  1. 2026Fine-Tuning Impairs the Balancedness of Foundation Models in Long-tailed Personalized Federated Learningcited by 1
  2. 2026Rethinking Loss Reweighting for Imbalance Learning as an Inverse Problem: A Neural Collapse Point of Viewcited by 0
  3. 2026Towards Federated Long-Tailed Graph Learning: An Energy-Guided Dual Decoupling Approachcited by 0
  4. 2026FedReLa: Imbalanced Federated Learning via Re-Labelingcited by 0
  5. 2026FedLTLib: A Comprehensive Benchmark for Federated Long-Tail Learningcited by 0