# FLAM: Evaluating Model Performance with Aggregatable Measures in Federated Learning

arXiv: https://arxiv.org/abs/2605.07962 · HTML: https://mlsanity.com/paper/2605.07962/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 20 of its 37 references that are in the map.
- First posted on arXiv 8 May 2026.

## Builds on (most cited first, 20 of 20)

- [Communication-Efficient Learning of Deep Networks from Decentralized Data](https://mlsanity.com/paper/1602.05629/) (2016)
- [Tackling the Objective Inconsistency Problem in Heterogeneous Federated Optimization](https://mlsanity.com/paper/2007.07481/) (2020)
- [LEAF: A Benchmark for Federated Settings](https://mlsanity.com/paper/1812.01097/) (2018)
- [Ensemble Distillation for Robust Model Fusion in Federated Learning](https://mlsanity.com/paper/2006.07242/) (2020)
- [Ditto: Fair and Robust Federated Learning Through Personalization](https://mlsanity.com/paper/2012.04221/) (2020)
- [Federated Learning for Mobile Keyboard Prediction](https://mlsanity.com/paper/1811.03604/) (2018)
- [Flower: A Friendly Federated Learning Research Framework](https://mlsanity.com/paper/2007.14390/) (2020)
- [TabTransformer: Tabular Data Modeling Using Contextual Embeddings](https://mlsanity.com/paper/2012.06678/) (2020)
- [Fair Resource Allocation in Federated Learning](https://mlsanity.com/paper/1905.10497/) (2019)
- [NVIDIA FLARE: Federated Learning from Simulation to Real-World](https://mlsanity.com/paper/2210.13291/) (2022)
- [LightSecAgg: a Lightweight and Versatile Design for Secure Aggregation in Federated Learning](https://mlsanity.com/paper/2109.14236/) (2021)
- [The OARF Benchmark Suite: Characterization and Implications for Federated Learning Systems](https://mlsanity.com/paper/2006.07856/) (2020)
- [TCT: Convexifying Federated Learning using Bootstrapped Neural Tangent Kernels](https://mlsanity.com/paper/2207.06343/) (2022)
- [A Survey for Federated Learning Evaluations: Goals and Measures](https://mlsanity.com/paper/2308.11841/) (2023)
- [MimiC: Combating Client Dropouts in Federated Learning by Mimicking Central Updates](https://mlsanity.com/paper/2306.12212/) (2023)
- [FedEval: A Holistic Evaluation Framework for Federated Learning](https://mlsanity.com/paper/2011.09655/) (2020)
- [FLAME: Federated Learning Across Multi-device Environments](https://mlsanity.com/paper/2202.08922/) (2022)
- [Evaluation Framework For Large-scale Federated Learning](https://mlsanity.com/paper/2003.01575/) (2020)
- [Research in Collaborative Learning Does Not Serve Cross-Silo Federated Learning in Practice](https://mlsanity.com/paper/2510.12595/) (2025)
- [Analyzing the Impact of Participant Failures in Cross-Silo Federated Learning](https://mlsanity.com/paper/2511.14456/) (2025)

## Abstract (arXiv)

Performance evaluation is essential for assessing the quality of machine learning (ML) models and guiding deployment decisions. In federated learning (FL), assessing the performance is challenging because data are distributed across participants. Consequently, the coordinator must rely on locally computed evaluation metrics and aggregate them to assess the global model. A key challenge is that common aggregation strategies, such as weighted averaging based on the local samples per participant, do not always produce the same results as centralized evaluation. Existing definitions of performance evaluation are largely tailored to accuracy and do not generalize to other metrics, leading to inconsistencies between participant-based and centralized evaluation. However, such discrepancies are inconsistent with the FL objective and lead to a wrong calculation of the metric. To address this issue, we examine the underlying reasons for these discrepancies and propose FLAM, a performance evaluation method based on aggregatable measures that yields the same results as centralized evaluation without the need for a global test dataset.