# Certifiably Robust Model Evaluation in Federated Learning under Meta-Distributional Shifts

arXiv: https://arxiv.org/abs/2410.20250 · HTML: https://mlsanity.com/paper/2410.20250/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 31 of its 52 references that are in the map.
- First posted on arXiv 26 Oct 2024.

## Builds on (most cited first, 30 of 31)

- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Communication-Efficient Learning of Deep Networks from Decentralized Data](https://mlsanity.com/paper/1602.05629/) (2016)
- [Convex optimization](https://mlsanity.com/paper/2106.01946/) (2021)
- [Stochastic First- and Zeroth-order Methods for Nonconvex Stochastic Programming](https://mlsanity.com/paper/1309.5549/) (2013)
- [Federated Learning with Matched Averaging](https://mlsanity.com/paper/2002.06440/) (2020)
- [FedProto: Federated Prototype Learning across Heterogeneous Clients](https://mlsanity.com/paper/2105.00243/) (2021)
- [Certifying Some Distributional Robustness with Principled Adversarial Training](https://mlsanity.com/paper/1710.10571/) (2017)
- [No Fear of Heterogeneity: Classifier Calibration for Federated Learning with Non-IID Data](https://mlsanity.com/paper/2106.05001/) (2021)
- [Heterogeneous Federated Learning: State-of-the-art and Research Challenges](https://mlsanity.com/paper/2307.10616/) (2023)
- [Robust Federated Learning: The Case of Affine Distribution Shifts](https://mlsanity.com/paper/2006.08907/) (2020)
- [What Do We Mean by Generalization in Federated Learning?](https://mlsanity.com/paper/2110.14216/) (2021)
- [Maximum Mean Discrepancy Test is Aware of Adversarial Attacks](https://mlsanity.com/paper/2010.11415/) (2020)
- [Recent Advances on Federated Learning: A Systematic Survey](https://mlsanity.com/paper/2301.01299/) (2023)
- [MetaFed: Federated Learning among Federations with Cyclic Knowledge Distillation for Personalized Healthcare](https://mlsanity.com/paper/2206.08516/) (2022)
- [Device Heterogeneity in Federated Learning: A Superquantile Approach](https://mlsanity.com/paper/2002.11223/) (2020)
- [Curse or Redemption? How Data Heterogeneity Affects the Robustness of Federated Learning](https://mlsanity.com/paper/2102.00655/) (2021)
- [Delving into the Adversarial Robustness of Federated Learning](https://mlsanity.com/paper/2302.09479/) (2023)
- [Adversarial Robustness through Bias Variance Decomposition: A New Perspective for Federated Learning](https://mlsanity.com/paper/2009.09026/) (2020)
- [A Short and General Duality Proof for Wasserstein Distributionally Robust Optimization](https://mlsanity.com/paper/2205.00362/) (2022)
- [An Optimal Transport Approach to Personalized Federated Learning](https://mlsanity.com/paper/2206.02468/) (2022)
- [Distributionally Adaptive Meta Reinforcement Learning](https://mlsanity.com/paper/2210.03104/) (2022)
- [Superquantiles at Work: Machine Learning Applications and Efficient Subgradient Computation](https://mlsanity.com/paper/2201.00508/) (2022)
- [Generalization Bounds with Minimal Dependency on Hypothesis Class via Distributionally Robust Optimization](https://mlsanity.com/paper/2106.11180/) (2021)
- [FedLPS: Heterogeneous Federated Learning for Multiple Tasks with Local Parameter Sharing](https://mlsanity.com/paper/2402.08578/) (2024)
- [A Wasserstein Minimax Framework for Mixed Linear Regression](https://mlsanity.com/paper/2106.07537/) (2021)
- [A Survey of Federated Evaluation in Federated Learning](https://mlsanity.com/paper/2305.08070/) (2023)
- [Beyond the Federation: Topology-aware Federated Learning for Generalization to Unseen Clients](https://mlsanity.com/paper/2407.04949/) (2024)
- [Distributionally Robust Optimization with Bias and Variance Reduction](https://mlsanity.com/paper/2310.13863/) (2023)
- [Federated Learning Aggregation: New Robust Algorithms with Guarantees](https://mlsanity.com/paper/2205.10864/) (2022)
- [Sample Complexity Bounds for Learning High-dimensional Simplices in Noisy Regimes](https://mlsanity.com/paper/2209.05953/) (2022)

## Abstract (arXiv)

We address the challenge of certifying the performance of a federated learning model on an unseen target network using only measurements from the source network that trained the model. Specifically, consider a source network "A" with K clients, each holding private, non-IID datasets drawn from heterogeneous distributions, modeled as samples from a broader meta-distribution μ. Our goal is to provide certified guarantees for the model's performance on a different, unseen network "B", governed by an unknown meta-distribution μ', assuming the deviation between μ and μ' is bounded either in Wasserstein distance or an f-divergence. We derive worst-case uniform guarantees for both the model's average loss and its risk CDF, the latter corresponding to a novel, adversarially robust version of the Dvoretzky-Kiefer-Wolfowitz (DKW) inequality. In addition, we show how the vanilla DKW bound enables principled certification of the model's true performance on unseen clients within the same (source) network. Our bounds are efficiently computable, asymptotically minimax optimal, and preserve clients' privacy. We also establish non-asymptotic generalization bounds that converge to zero as K grows and the minimum per-client sample size exceeds O(log K). Empirical evaluations confirm the practical utility of our bounds across real-world tasks. The project code is available at: github.com/samin-mehdizadeh/Robust-Evaluation-DKW