mlsanityData to 7 Oct 2026

Watch Your Head: Assembling Projection Heads to Save the Reliability of Federated Models

arXiv 2402.16255 ↗First posted 26 Feb 2024cs.LG cs.AIcited by 3

Not on the map. Why?

Abstract (arXiv)

Federated learning encounters substantial challenges with heterogeneous data, leading to performance degradation and convergence issues. While considerable progress has been achieved in mitigating such an impact, the reliability aspect of federated models has been largely disregarded. In this study, we conduct extensive experiments to investigate the reliability of both generic and personalized federated models. Our exploration uncovers a significant finding: federated models exhibit unreliability when faced with heterogeneous data, demonstrating poor calibration on in-distribution test data and low uncertainty levels on out-of-distribution data. This unreliability is primarily attributed to the presence of biased projection heads, which introduce miscalibration into the federated models. Inspired by this observation, we propose the "Assembled Projection Heads" (APH) method for enhancing the reliability of federated models. By treating the existing projection head parameters as priors, APH randomly samples multiple initialized parameters of projection heads from the prior and further performs targeted fine-tuning on locally available data under varying learning rates. Such a head ensemble introduces parameter diversity into the deterministic model, eliminating the bias and producing reliable predictions via head averaging. We evaluate the effectiveness of the proposed APH method across three prominent federated benchmarks. Experimental results validate the efficacy of APH in model calibration and uncertainty estimation. Notably, APH can be seamlessly integrated into various federated approaches but only requires less than 30% additional computation cost for 100× inferences within large models.

Key facts. Cited by 3 papers in the map (data through 7 Oct 2026). Builds on 34 of its 49 references that are in the map. First posted on arXiv 26 Feb 2024.

Builds on

34 of 49 references.

References · 34

  1. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,592
  2. 2017On Calibration of Modern Neural Networkscited by 4,860
  3. 2015Dropout as a Bayesian Approximation: Representing Model Uncertainty in Deep Learningcited by 4,416
  4. 2016Simple and Scalable Predictive Uncertainty Estimation using Deep Ensemblescited by 3,888
  5. 2019Advances and Open Problems in Federated Learningcited by 3,021
  6. 2019Benchmarking Neural Network Robustness to Common Corruptions and Perturbationscited by 2,857
  7. 2018Neural Tangent Kernel: Convergence and Generalization in Neural Networkscited by 2,597
  8. 2019Natural Adversarial Examplescited by 1,401
  9. 2019Can You Trust Your Model's Uncertainty? Evaluating Predictive Uncertainty Under Dataset Shiftcited by 1,274
  10. 2019Measuring the Effects of Non-Identical Data Distribution for Federated Visual Classificationcited by 836
  11. 2021Model-Contrastive Federated Learningcited by 762
  12. 2018Evidential Deep Learning to Quantify Classification Uncertaintycited by 728
22 more
  1. 2020Ensemble Distillation for Robust Model Fusion in Federated Learningcited by 678
  2. 2017Federated Multi-Task Learningcited by 669
  3. 2020Ditto: Fair and Robust Federated Learning Through Personalizationcited by 665
  4. 2020Personalized Federated Learning with Moreau Envelopescited by 612
  5. 2021Federated Learning on Non-IID Data Silos: An Experimental Studycited by 566
  6. 2019A Simple Baseline for Bayesian Uncertainty in Deep Learningcited by 555
  7. 2021Exploiting Shared Representations for Personalized Federated Learningcited by 537
  8. 2019Federated Learning with Personalization Layerscited by 533
  9. 2019Federated Learning with Differential Privacy: Algorithms and Performance Analysiscited by 528
  10. 2021Federated Learning on Non-IID Data: A Surveycited by 345
  11. 2020Personalized Cross-Silo Federated Learning on Non-IID Datacited by 282
  12. 2021No Fear of Heterogeneity: Classifier Calibration for Federated Learning with Non-IID Datacited by 241
  13. 2020Personalized Federated Learning with First Order Model Optimizationcited by 201
  14. 2022FedDC: Federated Learning with Non-IID Data via Local Drift Decoupling and Correctioncited by 181
  15. 2018Deep Bayesian Bandits Showdown: An Empirical Comparison of Bayesian Deep Networks for Thompson Samplingcited by 166
  16. 2022FedALA: Adaptive Local Aggregation for Personalized Federated Learningcited by 121
  17. 2023Personalized Federated Learning with Feature Alignment and Classifier Collaborationcited by 97
  18. 2021Preservation of the Global Knowledge by Not-True Distillation in Federated Learningcited by 87
  19. 2016Structured and Efficient Variational Deep Learning with Matrix Gaussian Posteriorscited by 82
  20. 2022A Consistent and Differentiable Lp Canonical Calibration Error Estimatorcited by 44
  21. 2018Understanding the Disharmony between Dropout and Batch Normalization by Variance Shiftcited by 36
  22. 2022Single Model Uncertainty Estimation via Stochastic Data Centeringcited by 20

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 1; 2026: 1.

3 papers.

  1. 2024TPFL: A Trustworthy Personalized Federated Learning Framework via Subjective Logiccited by 1
  2. 2025HFedCKD: Toward Robust Heterogeneous Federated Learning via Data-free Knowledge Distillation and Two-way Contrastcited by 0
  3. 2026Temperature Scaling Attack Disrupting Model Confidence in Federated Learningcited by 0