# Measuring Model Biases in the Absence of Ground Truth

arXiv: https://arxiv.org/abs/2103.03417 · HTML: https://mlsanity.com/paper/2103.03417/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- Builds on 7 of its 29 references that are in the map.
- First posted on arXiv 5 Mar 2021.

## Builds on (most cited first, 7 of 7)

- [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014)
- [ImageNet Large Scale Visual Recognition Challenge](https://mlsanity.com/paper/1409.0575/) (2014)
- [Equality of Opportunity in Supervised Learning](https://mlsanity.com/paper/1610.02413/) (2016)
- [Semantics derived automatically from language corpora contain human-like biases](https://mlsanity.com/paper/1608.07187/) (2016)
- [Fair prediction with disparate impact: A study of bias in recidivism prediction instruments](https://mlsanity.com/paper/1610.07524/) (2016)
- [Women also Snowboard: Overcoming Bias in Captioning Models](https://mlsanity.com/paper/1803.09797/) (2018)
- [Predictive Inequity in Object Detection](https://mlsanity.com/paper/1902.11097/) (2019)

## Built on by (most cited first, 24 of 24)

- [Photorealistic Text-to-Image Diffusion Models with Deep Language Understanding](https://mlsanity.com/paper/2205.11487/) (2022)
- [Scaling Vision Transformers](https://mlsanity.com/paper/2106.04560/) (2021)
- [Scaling Vision Transformers to 22 Billion Parameters](https://mlsanity.com/paper/2302.05442/) (2023)
- [Fake it till you make it: Learning transferable representations from synthetic ImageNet clones](https://mlsanity.com/paper/2212.08420/) (2022)
- [Handling Bias in Toxic Speech Detection: A Survey](https://mlsanity.com/paper/2202.00126/) (2022)
- [Re-contextualizing Fairness in NLP: The Case of India](https://mlsanity.com/paper/2209.12226/) (2022)
- [BERTIN: Efficient Pre-Training of a Spanish Language Model using Perplexity Sampling](https://mlsanity.com/paper/2207.06814/) (2022)
- [Measuring Data](https://mlsanity.com/paper/2212.05129/) (2022)
- [Safety and Fairness for Content Moderation in Generative Models](https://mlsanity.com/paper/2306.06135/) (2023)
- [Seeing without Looking: Analysis Pipeline for Child Sexual Abuse Datasets](https://mlsanity.com/paper/2204.14110/) (2022)
- [OASIS Uncovers: High-Quality T2I Models, Same Old Stereotypes](https://mlsanity.com/paper/2501.00962/) (2025)
- [Social Norm Bias: Residual Harms of Fairness-Aware Algorithms](https://mlsanity.com/paper/2108.11056/) (2021)
- [A Systematic Literature Review of Human-Centered, Ethical, and Responsible AI](https://mlsanity.com/paper/2302.05284/) (2023)
- [Algorithmic Censoring in Dynamic Learning Systems](https://mlsanity.com/paper/2305.09035/) (2023)
- [On the Interpretability and Significance of Bias Metrics in Texts: a PMI-based Approach](https://mlsanity.com/paper/2104.06474/) (2021)
- [Data AUDIT: Identifying Attribute Utility- and Detectability-Induced Bias in Task Models](https://mlsanity.com/paper/2304.03218/) (2023)
- [Beyond Demographic Parity: Redefining Equal Treatment](https://mlsanity.com/paper/2303.08040/) (2023)
- [The Equity Framework: Fairness Beyond Equalized Predictive Outcomes](https://mlsanity.com/paper/2205.01072/) (2022)
- [The Undesirable Dependence on Frequency of Gender Bias Metrics Based on Word Embeddings](https://mlsanity.com/paper/2301.00792/) (2023)
- [The Homogenization Problem in LLMs: Towards Meaningful Diversity in AI Safety](https://mlsanity.com/paper/2601.06116/) (2026)
- [Benchmarking bias: Expanding clinical AI model card to incorporate bias reporting of social and non-social factors](https://mlsanity.com/paper/2311.12560/) (2023)
- [Visual Identification of Problematic Bias in Large Label Spaces](https://mlsanity.com/paper/2201.06386/) (2022)
- [Detecting Dataset Bias in Medical AI: A Generalized and Modality-Agnostic Auditing Framework](https://mlsanity.com/paper/2503.09969/) (2025)
- [Contrastive Analysis of Linguistic Representations in Large Language Model Outputs through Structured Synthetic Data Generation and Abstracted N-gram Associations](https://mlsanity.com/paper/2604.17398/) (2026)

## Abstract (arXiv)

The measurement of bias in machine learning often focuses on model performance across identity subgroups (such as man and woman) with respect to groundtruth labels. However, these methods do not directly measure the associations that a model may have learned, for example between labels and identity subgroups. Further, measuring a model's bias requires a fully annotated evaluation dataset which may not be easily available in practice. We present an elegant mathematical solution that tackles both issues simultaneously, using image classification as a working example. By treating a classification model's predictions for a given image as a set of labels analogous to a bag of words, we rank the biases that a model has learned with respect to different identity labels. We use (man, woman) as a concrete example of an identity label set (although this set need not be binary), and present rankings for the labels that are most biased towards one identity or the other. We demonstrate how the statistical properties of different association metrics can lead to different rankings of the most "gender biased" labels, and conclude that normalized pointwise mutual information (nPMI) is most useful in practice. Finally, we announce an open-sourced nPMI visualization tool using TensorBoard.