# Monitoring Algorithmic Fairness under Partial Observations

arXiv: https://arxiv.org/abs/2308.00341 · HTML: https://mlsanity.com/paper/2308.00341/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Builds on 29 of its 63 references that are in the map.
- First posted on arXiv 1 Aug 2023.

## Builds on (most cited first, 29 of 29)

- [Equality of Opportunity in Supervised Learning](https://mlsanity.com/paper/1610.02413/) (2016)
- [A Survey on Bias and Fairness in Machine Learning](https://mlsanity.com/paper/1908.09635/) (2019)
- [Counterfactual Fairness](https://mlsanity.com/paper/1703.06856/) (2017)
- [Certifying and removing disparate impact](https://mlsanity.com/paper/1412.3756/) (2014)
- [Fair prediction with disparate impact: A study of bias in recidivism prediction instruments](https://mlsanity.com/paper/1610.07524/) (2016)
- [Preventing Fairness Gerrymandering: Auditing and Learning for Subgroup Fairness](https://mlsanity.com/paper/1711.05144/) (2017)
- [The What-If Tool: Interactive Probing of Machine Learning Models](https://mlsanity.com/paper/1907.04135/) (2019)
- [Delayed Impact of Fair Machine Learning](https://mlsanity.com/paper/1803.04383/) (2018)
- [CheXclusion: Fairness gaps in deep chest X-ray classifiers](https://mlsanity.com/paper/2003.00827/) (2020)
- [Runaway Feedback Loops in Predictive Policing](https://mlsanity.com/paper/1706.09847/) (2017)
- [Differentially Private Fair Learning](https://mlsanity.com/paper/1812.02696/) (2018)
- [iFair: Learning Individually Fair Data Representations for Algorithmic Decision Making](https://mlsanity.com/paper/1806.01059/) (2018)
- [Verifying Individual Fairness in Machine Learning Models](https://mlsanity.com/paper/2006.11737/) (2020)
- [Average Individual Fairness: Algorithms, Generalization and Experiments](https://mlsanity.com/paper/1905.10607/) (2019)
- [Probabilistic Verification of Fairness Properties via Concentration](https://mlsanity.com/paper/1812.02573/) (2018)
- [Fair Normalizing Flows](https://mlsanity.com/paper/2106.05937/) (2021)
- [PAC Statistical Model Checking for Markov Decision Processes and Stochastic Games](https://mlsanity.com/paper/1905.04403/) (2019)
- [Justicia: A Stochastic SAT Approach to Formally Verify Fairness](https://mlsanity.com/paper/2009.06516/) (2020)
- [Certifying Robustness to Programmable Data Bias in Decision Trees](https://mlsanity.com/paper/2110.04363/) (2021)
- [Fairness-Aware PAC Learning from Corrupted Data](https://mlsanity.com/paper/2102.06004/) (2021)
- [Probabilistic Verification of Neural Networks Against Group Fairness](https://mlsanity.com/paper/2107.08362/) (2021)
- [Runtime Monitoring of Dynamic Fairness Properties](https://mlsanity.com/paper/2305.04699/) (2023)
- [Algorithmic Fairness Verification with Graphical Models](https://mlsanity.com/paper/2109.09447/) (2021)
- [Real-time Stream-based Monitoring](https://mlsanity.com/paper/1711.03829/) (2017)
- [Monitoring Algorithmic Fairness](https://mlsanity.com/paper/2305.15979/) (2023)
- [Monitoring Event Frequencies](https://mlsanity.com/paper/1910.06097/) (2019)
- [Quantitative and Approximate Monitoring](https://mlsanity.com/paper/2105.08353/) (2021)
- [Runtime Monitoring for Markov Decision Processes](https://mlsanity.com/paper/2105.12322/) (2021)
- [Towards Probabilistic Session-Type Monitoring](https://mlsanity.com/paper/2107.08729/) (2021)

## Built on by (most cited first, 7 of 7)

- [Stream-Based Monitoring of Algorithmic Fairness](https://mlsanity.com/paper/2501.18331/) (2025)
- [Is it Still Fair? A Comparative Evaluation of Fairness Algorithms through the Lens of Covariate Drift](https://mlsanity.com/paper/2409.12428/) (2024)
- [Monitoring Robustness and Individual Fairness](https://mlsanity.com/paper/2506.00496/) (2025)
- [Using AI Alignment Theory to understand the potential pitfalls of regulatory frameworks](https://mlsanity.com/paper/2410.19749/) (2024)
- [Monitoring of Static Fairness](https://mlsanity.com/paper/2507.03048/) (2025)
- [Algorithmic Fairness: A Runtime Perspective](https://mlsanity.com/paper/2507.20711/) (2025)
- [FairMon: A Tool for Monitoring and Visualizing Algorithmic Fairness](https://mlsanity.com/paper/2609.26123/) (2026)

## Abstract (arXiv)

As AI and machine-learned software are used increasingly for making decisions that affect humans, it is imperative that they remain fair and unbiased in their decisions. To complement design-time bias mitigation measures, runtime verification techniques have been introduced recently to monitor the algorithmic fairness of deployed systems. Previous monitoring techniques assume full observability of the states of the (unknown) monitored system. Moreover, they can monitor only fairness properties that are specified as arithmetic expressions over the probabilities of different events. In this work, we extend fairness monitoring to systems modeled as partially observed Markov chains (POMC), and to specifications containing arithmetic expressions over the expected values of numerical functions on event sequences. The only assumptions we make are that the underlying POMC is aperiodic and starts in the stationary distribution, with a bound on its mixing time being known. These assumptions enable us to estimate a given property for the entire distribution of possible executions of the monitored POMC, by observing only a single execution. Our monitors observe a long run of the system and, after each new observation, output updated PAC-estimates of how fair or biased the system is. The monitors are computationally lightweight and, using a prototype implementation, we demonstrate their effectiveness on several real-world examples.