# Data-Free Evaluation of User Contributions in Federated Learning

arXiv: https://arxiv.org/abs/2108.10623 · HTML: https://mlsanity.com/paper/2108.10623/

## Key facts

- Cited by 10 papers in the map (data through 14 Oct 2026).
- Builds on 22 of its 37 references that are in the map.
- First posted on arXiv 24 Aug 2021.

## Builds on (most cited first, 22 of 22)

- [Communication-Efficient Learning of Deep Networks from Decentralized Data](https://mlsanity.com/paper/1602.05629/) (2016)
- [Deep Learning with Differential Privacy](https://mlsanity.com/paper/1607.00133/) (2016)
- [Federated Learning: Strategies for Improving Communication Efficiency](https://mlsanity.com/paper/1610.05492/) (2016)
- [Federated Learning with Non-IID Data](https://mlsanity.com/paper/1806.00582/) (2018)
- [Byzantine-Robust Distributed Learning: Towards Optimal Statistical Rates](https://mlsanity.com/paper/1803.01498/) (2018)
- [Federated Learning for Mobile Keyboard Prediction](https://mlsanity.com/paper/1811.03604/) (2018)
- [Deep Interest Network for Click-Through Rate Prediction](https://mlsanity.com/paper/1706.06978/) (2017)
- [Towards Efficient Data Valuation Based on the Shapley Value](https://mlsanity.com/paper/1902.10275/) (2019)
- [Applied Federated Learning: Improving Google Keyboard Query Suggestions](https://mlsanity.com/paper/1812.02903/) (2018)
- [cpSGD: Communication-efficient and differentially-private distributed SGD](https://mlsanity.com/paper/1805.10559/) (2018)
- [Deep Learning with Gaussian Differential Privacy](https://mlsanity.com/paper/1911.11607/) (2019)
- [Measure Contribution of Participants in Federated Learning](https://mlsanity.com/paper/1909.08525/) (2019)
- [Byzantine-Robust Federated Machine Learning through Adaptive Model Averaging](https://mlsanity.com/paper/1909.05125/) (2019)
- [Distributed Statistical Machine Learning in Adversarial Settings: Byzantine Gradient Descent](https://mlsanity.com/paper/1705.05491/) (2017)
- [Free-riders in Federated Learning: Attacks and Defenses](https://mlsanity.com/paper/1911.12560/) (2019)
- [FMore: An Incentive Scheme of Multi-dimensional Auction for Federated Learning in MEC](https://mlsanity.com/paper/2002.09699/) (2020)
- [FedCoin: A Peer-to-Peer Payment System for Federated Learning](https://mlsanity.com/paper/2002.11711/) (2020)
- [Crowdsourced Judgement Elicitation with Endogenous Proficiency](https://mlsanity.com/paper/1303.0799/) (2013)
- [Informed Truthfulness in Multi-Task Peer Prediction](https://mlsanity.com/paper/1603.03151/) (2016)
- [An Exploratory Analysis on Users' Contributions in Federated Learning](https://mlsanity.com/paper/2011.06830/) (2020)
- [Incentives for Federated Learning: a Hypothesis Elicitation Approach](https://mlsanity.com/paper/2007.10596/) (2020)
- [FedServing: A Federated Prediction Serving Framework Based on Incentive Mechanism](https://mlsanity.com/paper/2012.10566/) (2020)

## Built on by (most cited first, 10 of 10)

- [Incentive Mechanisms for Federated Learning: From Economic and Game Theoretic Perspective](https://mlsanity.com/paper/2111.11850/) (2021)
- [Redefining Contributions: Shapley-Driven Federated Learning](https://mlsanity.com/paper/2406.00569/) (2024)
- [Decentral and Incentivized Federated Learning Frameworks: A Systematic Literature Review](https://mlsanity.com/paper/2205.07855/) (2022)
- [Training Machine Learning models at the Edge: A Survey](https://mlsanity.com/paper/2403.02619/) (2024)
- [Incentives in Private Collaborative Machine Learning](https://mlsanity.com/paper/2404.01676/) (2024)
- [Multi-Task Model Personalization for Federated Supervised SVM in Heterogeneous Networks](https://mlsanity.com/paper/2303.10254/) (2023)
- [Adaptive incentive for cross-silo federated learning: A multi-agent reinforcement learning approach](https://mlsanity.com/paper/2302.07493/) (2023)
- [How to Evaluate Participant Contributions in Decentralized Federated Learning](https://mlsanity.com/paper/2505.23246/) (2025)
- [Owen Sampling Accelerates Contribution Estimation in Federated Learning](https://mlsanity.com/paper/2508.21261/) (2025)
- [Knowledge-Free Correlated Agreement for Incentivizing Federated Learning](https://mlsanity.com/paper/2605.04747/) (2026)

## Abstract (arXiv)

Federated learning (FL) trains a machine learning model on mobile devices in a distributed manner using each device's private data and computing resources. A critical issues is to evaluate individual users' contributions so that (1) users' effort in model training can be compensated with proper incentives and (2) malicious and low-quality users can be detected and removed. The state-of-the-art solutions require a representative test dataset for the evaluation purpose, but such a dataset is often unavailable and hard to synthesize. In this paper, we propose a method called Pairwise Correlated Agreement (PCA) based on the idea of peer prediction to evaluate user contribution in FL without a test dataset. PCA achieves this using the statistical correlation of the model parameters uploaded by users. We then apply PCA to designing (1) a new federated learning algorithm called Fed-PCA, and (2) a new incentive mechanism that guarantees truthfulness. We evaluate the performance of PCA and Fed-PCA using the MNIST dataset and a large industrial product recommendation dataset. The results demonstrate that our Fed-PCA outperforms the canonical FedAvg algorithm and other baseline methods in accuracy, and at the same time, PCA effectively incentivizes users to behave truthfully.