mlsanityData to 14 Oct 2026

A Survey for Federated Learning Evaluations: Goals and Measures

arXiv 2308.11841 ↗First posted 23 Aug 2023cs.LG cs.CR cs.DCcited by 16

Not on the map. Why?

Abstract (arXiv)

Evaluation is a systematic approach to assessing how well a system achieves its intended purpose. Federated learning (FL) is a novel paradigm for privacy-preserving machine learning that allows multiple parties to collaboratively train models without sharing sensitive data. However, evaluating FL is challenging due to its interdisciplinary nature and diverse goals, such as utility, efficiency, and security. In this survey, we first review the major evaluation goals adopted in the existing studies and then explore the evaluation metrics used for each goal. We also introduce FedEval, an open-source platform that provides a standardized and comprehensive evaluation framework for FL algorithms in terms of their utility, efficiency, and security. Finally, we discuss several challenges and future research directions for FL evaluation.

Key facts. Cited by 16 papers in the map (data through 14 Oct 2026). Builds on 69 of its 124 references that are in the map. First posted on arXiv 23 Aug 2023.

Builds on

69 of 124 references.

References · 69

  1. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,669
  2. 2018Federated Optimization in Heterogeneous Networkscited by 2,987
  3. 2016Membership Inference Attacks against Machine Learning Modelscited by 2,281
  4. 2019On the Convergence of FedAvg on Non-IID Datacited by 1,229
  5. 2018Federated Learning with Non-IID Datacited by 1,108
  6. 2020Adaptive Federated Optimizationcited by 1,046
  7. 2019Deep Leakage from Gradientscited by 1,021
  8. 2018How To Backdoor Federated Learningcited by 765
  9. 2020Inverting Gradients -- How easy is it to break privacy in federated learning?cited by 712
  10. 2017Federated Multi-Task Learningcited by 672
  11. 2020Personalized Federated Learning with Moreau Envelopescited by 618
  12. 2021FedBN: Federated Learning on Non-IID Features via Local Batch Normalizationcited by 571
38 more
  1. 2020Federated Learning with Matched Averagingcited by 564
  2. 2019Federated Learning with Differential Privacy: Algorithms and Performance Analysiscited by 536
  3. 2019Local Model Poisoning Attacks to Byzantine-Robust Federated Learningcited by 493
  4. 2020Flower: A Friendly Federated Learning Research Frameworkcited by 455
  5. 2019Robust and Communication-Efficient Federated Learning from Non-IID Datacited by 419
  6. 2019Federated Learning in Mobile Edge Networks: A Comprehensive Surveycited by 408
  7. 2019Fair Resource Allocation in Federated Learningcited by 394
  8. 2019A Survey on Federated Learning Systems: Vision, Hype and Reality for Data Privacy and Protectioncited by 374
  9. 2020FLTrust: Byzantine-robust Federated Learning via Trust Bootstrappingcited by 346
  10. 2021Federated Learning on Non-IID Data: A Surveycited by 346
  11. 2020HeteroFL: Computation and Communication Efficient Federated Learning for Heterogeneous Clientscited by 337
  12. 2020FedML: A Research Library and Benchmark for Federated Machine Learningcited by 300
  13. 2021Federated Learning for Internet of Things: A Comprehensive Surveycited by 252
  14. 2018Communication-Efficient On-Device Machine Learning: Federated Distillation and Augmentation under Non-IID Private Datacited by 243
  15. 2021Achieving Linear Speedup with Partial Worker Participation in Non-IID Federated Learningcited by 193
  16. 2021Federated Learning for Smart Healthcare: A Surveycited by 184
  17. 2020FetchSGD: Communication-Efficient Federated Learning with Sketchingcited by 179
  18. 2017Private federated learning on vertically partitioned data via entity resolution and additively homomorphic encryptioncited by 176
  19. 2017Revisiting Distributed Synchronous SGDcited by 142
  20. 2020Federated Learning for Internet of Things: Recent Advances, Taxonomy, and Open Challengescited by 139
  21. 2021FedScale: Benchmarking Model and System Performance of Federated Learning at Scalecited by 136
  22. 2021Back to the Drawing Board: A Critical Evaluation of Poisoning Attacks on Production Federated Learningcited by 118
  23. 2022Federated Learning with Partial Model Personalizationcited by 114
  24. 2021FedMix: Approximation of Mixup under Mean Augmented Federated Learningcited by 108
  25. 2019Secure Federated Matrix Factorizationcited by 102
  26. 2022FedRolex: Model-Heterogeneous Federated Learning with Rolling Sub-Model Extractioncited by 102
  27. 2020R-GAP: Recursive Gradient Attack on Privacycited by 97
  28. 2021Preservation of the Global Knowledge by Not-True Distillation in Federated Learningcited by 87
  29. 2021Eluding Secure Aggregation in Federated Learning via Model Inconsistencycited by 67
  30. 2022Personalized Subgraph Federated Learningcited by 54
  31. 2022Orchestra: Unsupervised Federated Learning via Globally Consistent Clusteringcited by 45
  32. 2022No Free Lunch Theorem for Security and Utility in Federated Learningcited by 39
  33. 2023Personalized Federated Learning under Mixture of Distributionscited by 37
  34. 2022On Privacy and Personalization in Cross-Silo Federated Learningcited by 36
  35. 2022BlindFL: Vertical Federated Machine Learning without Peeking into Your Datacited by 36
  36. 2022SoteriaFL: A Unified Framework for Private Federated Learning with Communication Compressioncited by 34
  37. 2022Trading Off Privacy, Utility and Efficiency in Federated Learningcited by 33
  38. 2020When Homomorphic Encryption Marries Secret Sharing: Secure Large-Scale Sparse Logistic Regression and Applications in Risk Controlcited by 32

50 most cited of 69 All 69 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 6; 2025: 6; 2026: 4.

16 papers.

Sort:

By area:

  1. 2024Non-IID data in Federated Learning: A Survey with Taxonomy, Metrics, Methods, Frameworks and Future Directionscited by 13
  2. 2024Exploring Vacant Classes in Label-Skewed Federated Learningcited by 10
  3. 2024FedProK: Trustworthy Federated Class-Incremental Learning via Prototypical Feature Knowledge Transfercited by 9
  4. 2024A Survey on Contribution Evaluation in Vertical Federated Learningcited by 5
  5. 2024Towards Active Participant Centric Vertical Federated Learning: Some Representations May Be All You Needcited by 2
  6. 2024Recent Advances in Federated Learning Driven Large Language Models: A Survey on Architecture, Performance, and Securitycited by 1
  7. 2025DROP: Poison Dilution via Knowledge Distillation for Federated Learningcited by 1
  8. 2025Personalized Recommendation Models in Federated Settings: A Surveycited by 1
  9. 2025Securing Genomic Data Against Inference Attacks in Federated Learning Environmentscited by 1
  10. 2026DeepAFL: Deep Analytic Federated Learningcited by 1
  11. 2026Trustworthy Federated Label Distribution Learning under Annotation Quality Disparitycited by 1
  12. 2025FedVSR: Towards Model-Agnostic Federated Learning in Video Super-Resolutioncited by 0
4 more
  1. 2025FDA-Opt: Federated Fine-Tuning via Dynamic Update Schedulescited by 0
  2. 2025Colorectal Cancer Histopathological Grading using Multi-Scale Federated Learningcited by 0
  3. 2026FLAM: Evaluating Model Performance with Aggregatable Measures in Federated Learningcited by 0
  4. 2026A Game-Theoretic Framework for Incentive-Compatible AI training Under Renewable-Energy Constraintscited by 0