# Federated Fairness Analytics: Quantifying Fairness in Federated Learning

arXiv: https://arxiv.org/abs/2408.08214 · HTML: https://mlsanity.com/paper/2408.08214/

## Key facts

- Cited by 4 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 51 references that are in the map.
- First posted on arXiv 15 Aug 2024.

## Builds on (most cited first, 26 of 26)

- [Advances and Open Problems in Federated Learning](https://mlsanity.com/paper/1912.04977/) (2019)
- [Equality of Opportunity in Supervised Learning](https://mlsanity.com/paper/1610.02413/) (2016)
- [A Survey on Bias and Fairness in Machine Learning](https://mlsanity.com/paper/1908.09635/) (2019)
- [Flower: A Friendly Federated Learning Research Framework](https://mlsanity.com/paper/2007.14390/) (2020)
- [Fair Resource Allocation in Federated Learning](https://mlsanity.com/paper/1905.10497/) (2019)
- [Communication-Efficient On-Device Machine Learning: Federated Distillation and Augmentation under Non-IID Private Data](https://mlsanity.com/paper/1811.11479/) (2018)
- [Heterogeneous Federated Learning: State-of-the-art and Research Challenges](https://mlsanity.com/paper/2307.10616/) (2023)
- [The Frontiers of Fairness in Machine Learning](https://mlsanity.com/paper/1810.08810/) (2018)
- [A Principled Approach to Data Valuation for Federated Learning](https://mlsanity.com/paper/2009.06192/) (2020)
- [FairFed: Enabling Group Fairness in Federated Learning](https://mlsanity.com/paper/2110.00857/) (2021)
- [Collaborative Fairness in Federated Learning](https://mlsanity.com/paper/2008.12161/) (2020)
- [Fairness-aware Agnostic Federated Learning](https://mlsanity.com/paper/2010.05057/) (2020)
- [Federated Learning with Fair Averaging](https://mlsanity.com/paper/2104.14937/) (2021)
- [Towards Fairness-Aware Federated Learning](https://mlsanity.com/paper/2111.01872/) (2021)
- [Improving Fairness via Federated Learning](https://mlsanity.com/paper/2110.15545/) (2021)
- [Enforcing fairness in private federated learning via the modified method of differential multipliers](https://mlsanity.com/paper/2109.08604/) (2021)
- [Bandit-based Communication-Efficient Client Selection Strategies for Federated Learning](https://mlsanity.com/paper/2012.08009/) (2020)
- [Minimax Demographic Group Fairness in Federated Learning](https://mlsanity.com/paper/2201.08304/) (2022)
- [FLAIR: Federated Learning Annotated Image Repository](https://mlsanity.com/paper/2207.08869/) (2022)
- [User Fairness in Recommender Systems](https://mlsanity.com/paper/1807.06349/) (2018)
- [Unified Group Fairness on Federated Learning](https://mlsanity.com/paper/2111.04986/) (2021)
- [PrivFairFL: Privacy-Preserving Group Fairness in Federated Learning](https://mlsanity.com/paper/2205.11584/) (2022)
- [Secure Shapley Value for Cross-Silo Federated Learning (Technical Report)](https://mlsanity.com/paper/2209.04856/) (2022)
- [Fairness and Privacy-Preserving in Federated Learning: A Survey](https://mlsanity.com/paper/2306.08402/) (2023)
- [New Metrics to Evaluate the Performance and Fairness of Personalized Federated Learning](https://mlsanity.com/paper/2107.13173/) (2021)
- [FOCUS: Fairness via Agent-Awareness for Federated Learning on Heterogeneous Data](https://mlsanity.com/paper/2207.10265/) (2022)

## Built on by (most cited first, 4 of 4)

- [Fairness in Federated Learning: Trends, Challenges, and Opportunities](https://mlsanity.com/paper/2509.00799/) (2025)
- [Benchmarking Mutual Information-based Loss Functions in Federated Learning](https://mlsanity.com/paper/2504.11877/) (2025)
- [Edge AI in Highly Volatile Environments: Is Fairness Worth the Accuracy Trade-off?](https://mlsanity.com/paper/2511.01737/) (2025)
- [Cumulative Utility Parity for Fair Federated Learning under Intermittent Client Participation](https://mlsanity.com/paper/2602.13651/) (2026)

## Abstract (arXiv)

Federated Learning (FL) is a privacy-enhancing technology for distributed ML. By training models locally and aggregating updates - a federation learns together, while bypassing centralised data collection. FL is increasingly popular in healthcare, finance and personal computing. However, it inherits fairness challenges from classical ML and introduces new ones, resulting from differences in data quality, client participation, communication constraints, aggregation methods and underlying hardware. Fairness remains an unresolved issue in FL and the community has identified an absence of succinct definitions and metrics to quantify fairness; to address this, we propose Federated Fairness Analytics - a methodology for measuring fairness. Our definition of fairness comprises four notions with novel, corresponding metrics. They are symptomatically defined and leverage techniques originating from XAI, cooperative game-theory and networking engineering. We tested a range of experimental settings, varying the FL approach, ML task and data settings. The results show that statistical heterogeneity and client participation affect fairness and fairness conscious approaches such as Ditto and q-FedAvg marginally improve fairness-performance trade-offs. Using our techniques, FL practitioners can uncover previously unobtainable insights into their system's fairness, at differing levels of granularity in order to address fairness challenges in FL. We have open-sourced our work at: https://github.com/oscardilley/federated-fairness.