mlsanityData to 14 Oct 2026

FLAM: Evaluating Model Performance with Aggregatable Measures in Federated Learning

arXiv 2605.07962 ↗First posted 8 May 2026cs.LG cs.DCcited by 0

Not on the map. Why?

Abstract (arXiv)

Performance evaluation is essential for assessing the quality of machine learning (ML) models and guiding deployment decisions. In federated learning (FL), assessing the performance is challenging because data are distributed across participants. Consequently, the coordinator must rely on locally computed evaluation metrics and aggregate them to assess the global model. A key challenge is that common aggregation strategies, such as weighted averaging based on the local samples per participant, do not always produce the same results as centralized evaluation. Existing definitions of performance evaluation are largely tailored to accuracy and do not generalize to other metrics, leading to inconsistencies between participant-based and centralized evaluation. However, such discrepancies are inconsistent with the FL objective and lead to a wrong calculation of the metric. To address this issue, we examine the underlying reasons for these discrepancies and propose FLAM, a performance evaluation method based on aggregatable measures that yields the same results as centralized evaluation without the need for a global test dataset.

Key facts. No citing papers in the map yet (data through 14 Oct 2026). Builds on 20 of its 37 references that are in the map. First posted on arXiv 8 May 2026.

Builds on

20 of 37 references.

References · 20

  1. 2016Communication-Efficient Learning of Deep Networks from Decentralized Datacited by 7,669
  2. 2020Tackling the Objective Inconsistency Problem in Heterogeneous Federated Optimizationcited by 906
  3. 2018LEAF: A Benchmark for Federated Settingscited by 792
  4. 2020Ensemble Distillation for Robust Model Fusion in Federated Learningcited by 682
  5. 2020Ditto: Fair and Robust Federated Learning Through Personalizationcited by 672
  6. 2018Federated Learning for Mobile Keyboard Predictioncited by 572
  7. 2020Flower: A Friendly Federated Learning Research Frameworkcited by 455
  8. 2020TabTransformer: Tabular Data Modeling Using Contextual Embeddingscited by 409
  9. 2019Fair Resource Allocation in Federated Learningcited by 394
  10. 2022NVIDIA FLARE: Federated Learning from Simulation to Real-Worldcited by 80
  11. 2021LightSecAgg: a Lightweight and Versatile Design for Secure Aggregation in Federated Learningcited by 64
  12. 2020The OARF Benchmark Suite: Characterization and Implications for Federated Learning Systemscited by 31
8 more
  1. 2022TCT: Convexifying Federated Learning using Bootstrapped Neural Tangent Kernelscited by 26
  2. 2023A Survey for Federated Learning Evaluations: Goals and Measurescited by 16
  3. 2023MimiC: Combating Client Dropouts in Federated Learning by Mimicking Central Updatescited by 10
  4. 2020FedEval: A Holistic Evaluation Framework for Federated Learningcited by 9
  5. 2022FLAME: Federated Learning Across Multi-device Environmentscited by 8
  6. 2020Evaluation Framework For Large-scale Federated Learningcited by 4
  7. 2025Research in Collaborative Learning Does Not Serve Cross-Silo Federated Learning in Practicecited by 3
  8. 2025Analyzing the Impact of Participant Failures in Cross-Silo Federated Learningcited by 2

Built on by

Not cited yet.