# Fairness in the Eyes of the Data: Certifying Machine-Learning Models

arXiv: https://arxiv.org/abs/2009.01534 · HTML: https://mlsanity.com/paper/2009.01534/

## Key facts

- Cited by 16 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 3 Sep 2020.

## Built on by (most cited first, 16 of 16)

- [Fair Normalizing Flows](https://mlsanity.com/paper/2106.05937/) (2021)
- [Latent Space Smoothing for Individually Fair Representations](https://mlsanity.com/paper/2111.13650/) (2021)
- [Fairness Score and Process Standardization: Framework for Fairness Certification in Artificial Intelligence Systems](https://mlsanity.com/paper/2201.06952/) (2022)
- [FairProof : Confidential and Certifiable Fairness for Neural Networks](https://mlsanity.com/paper/2402.12572/) (2024)
- [Certifying Some Distributional Fairness with Subpopulation Decomposition](https://mlsanity.com/paper/2205.15494/) (2022)
- [Input-agnostic Certified Group Fairness via Gaussian Parameter Smoothing](https://mlsanity.com/paper/2206.11423/) (2022)
- [An Auditing Test To Detect Behavioral Shift in Language Models](https://mlsanity.com/paper/2410.19406/) (2024)
- [Verifiable Fairness: Privacy-preserving Computation of Fairness for Machine Learning Systems](https://mlsanity.com/paper/2309.06061/) (2023)
- [Secure and Confidential Certificates of Online Fairness](https://mlsanity.com/paper/2410.02777/) (2024)
- [Attesting Distributional Properties of Training Data for Machine Learning](https://mlsanity.com/paper/2308.09552/) (2023)
- [PrivFair: a Library for Privacy-Preserving Fairness Auditing](https://mlsanity.com/paper/2202.04058/) (2022)
- [Equal Confusion Fairness: Measuring Group-Based Disparities in Automated Decision Systems](https://mlsanity.com/paper/2307.00472/) (2023)
- [MPC-Friendly Commitments for Publicly Verifiable Covert Security](https://mlsanity.com/paper/2109.07461/) (2021)
- [DREAM: Debugging and Repairing AutoML Pipelines](https://mlsanity.com/paper/2401.00379/) (2023)
- [Fairness Certification for Natural Language Processing and Large Language Models](https://mlsanity.com/paper/2401.01262/) (2024)
- [Certified but Private: Scalable Zero-Knowledge Proofs for Neural Network Guarantees](https://mlsanity.com/paper/2608.17070/) (2026)

## Abstract (arXiv)

We present a framework that allows to certify the fairness degree of a model based on an interactive and privacy-preserving test. The framework verifies any trained model, regardless of its training process and architecture. Thus, it allows us to evaluate any deep learning model on multiple fairness definitions empirically. We tackle two scenarios, where either the test data is privately available only to the tester or is publicly known in advance, even to the model creator. We investigate the soundness of the proposed approach using theoretical analysis and present statistical guarantees for the interactive test. Finally, we provide a cryptographic technique to automate fairness testing and certified inference with only black-box access to the model at hand while hiding the participants' sensitive data.