# Federated Calibration and Evaluation of Binary Classifiers

arXiv: https://arxiv.org/abs/2210.12526 · HTML: https://mlsanity.com/paper/2210.12526/

## Key facts

- Cited by 5 papers in the map (data through 14 Oct 2026).
- Builds on 18 of its 35 references that are in the map.
- First posted on arXiv 22 Oct 2022.

## Builds on (most cited first, 18 of 18)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Communication-Efficient Learning of Deep Networks from Decentralized Data](https://mlsanity.com/paper/1602.05629/) (2016)
- [On Calibration of Modern Neural Networks](https://mlsanity.com/paper/1706.04599/) (2017)
- [Advances and Open Problems in Federated Learning](https://mlsanity.com/paper/1912.04977/) (2019)
- [Federated Learning with Differential Privacy: Algorithms and Performance Analysis](https://mlsanity.com/paper/1911.00222/) (2019)
- [Revisiting the Calibration of Modern Neural Networks](https://mlsanity.com/paper/2106.07998/) (2021)
- [The Skellam Mechanism for Differentially Private Federated Learning](https://mlsanity.com/paper/2110.04995/) (2021)
- [Local Differential Privacy and Its Applications: A Comprehensive Survey](https://mlsanity.com/paper/2008.03686/) (2020)
- [Differential Privacy: An Economic Method for Choosing Epsilon](https://mlsanity.com/paper/1402.3329/) (2014)
- [Private Summation in the Multi-Message Shuffle Model](https://mlsanity.com/paper/2002.00817/) (2020)
- [Towards Formalizing the GDPR's Notion of Singling Out](https://mlsanity.com/paper/1904.06009/) (2019)
- [Locally Private Mean Estimation: Z-test and Tight Confidence Intervals](https://mlsanity.com/paper/1810.08054/) (2018)
- [Answering Range Queries Under Local Differential Privacy](https://mlsanity.com/paper/1812.10942/) (2018)
- [Differentially Private Algorithms for Empirical Machine Learning](https://mlsanity.com/paper/1411.5428/) (2014)
- [Private Protocols for U-Statistics in the Local Model and Beyond](https://mlsanity.com/paper/1910.03861/) (2019)
- [DPAUC: Differentially Private AUC Computation in Federated Learning](https://mlsanity.com/paper/2208.12294/) (2022)
- [Answering Multi-Dimensional Range Queries under Local Differential Privacy](https://mlsanity.com/paper/2009.06538/) (2020)
- [Differentially Private AUC Computation in Vertical Federated Learning](https://mlsanity.com/paper/2205.12412/) (2022)

## Built on by (most cited first, 5 of 5)

- [One-Shot Federated Conformal Prediction](https://mlsanity.com/paper/2302.06322/) (2023)
- [Marginal and training-conditional guarantees in one-shot federated conformal prediction](https://mlsanity.com/paper/2405.12567/) (2024)
- [PAPAYA Federated Analytics Stack: Engineering Privacy, Scalability and Practicality](https://mlsanity.com/paper/2412.02340/) (2024)
- [Private Federated Multiclass Post-hoc Calibration](https://mlsanity.com/paper/2510.01987/) (2025)
- [Federated Computation of ROC and PR Curves](https://mlsanity.com/paper/2510.04979/) (2025)

## Abstract (arXiv)

We address two major obstacles to practical use of supervised classifiers on distributed private data. Whether a classifier was trained by a federation of cooperating clients or trained centrally out of distribution, (1) the output scores must be calibrated, and (2) performance metrics must be evaluated -- all without assembling labels in one place. In particular, we show how to perform calibration and compute precision, recall, accuracy and ROC-AUC in the federated setting under three privacy models (i) secure aggregation, (ii) distributed differential privacy, (iii) local differential privacy. Our theorems and experiments clarify tradeoffs between privacy, accuracy, and data efficiency. They also help decide whether a given application has sufficient data to support federated calibration and evaluation.