# Taxonomizing and Measuring Representational Harms: A Look at Image Tagging

arXiv: https://arxiv.org/abs/2305.01776 · HTML: https://mlsanity.com/paper/2305.01776/

## Key facts

- Cited by 25 papers in the map (data through 14 Oct 2026).
- First posted on arXiv 2 May 2023.

## Built on by (most cited first, 25 of 25)

- [Evaluating the Social Impact of Generative AI Systems in Systems and Society](https://mlsanity.com/paper/2306.05949/) (2023)
- [Position is Power: System Prompts as a Mechanism of Bias in Large Language Models (LLMs)](https://mlsanity.com/paper/2505.21091/) (2025)
- [Beyond Behaviorist Representational Harms: A Plan for Measurement and Mitigation](https://mlsanity.com/paper/2402.01705/) (2024)
- [The Case for "Thick Evaluations" of Cultural Representation in AI](https://mlsanity.com/paper/2503.19075/) (2025)
- [Taxonomizing Representational Harms using Speech Act Theory](https://mlsanity.com/paper/2504.00928/) (2025)
- [More of the Same: Persistent Representational Harms Under Increased Representation](https://mlsanity.com/paper/2503.00333/) (2025)
- [Measuring Machine Learning Harms from Stereotypes Requires Understanding Who Is Harmed by Which Errors in What Ways](https://mlsanity.com/paper/2402.04420/) (2024)
- [Who Evaluates AI's Social Impacts? Mapping Coverage and Gaps in First and Third Party Evaluations](https://mlsanity.com/paper/2511.05613/) (2025)
- [Understanding and Meeting Practitioner Needs When Measuring Representational Harms Caused by LLM-Based Systems](https://mlsanity.com/paper/2506.04482/) (2025)
- [Investigating Intersectional Bias in Large Language Models using Confidence Disparities in Coreference Resolution](https://mlsanity.com/paper/2508.07111/) (2025)
- [A Taxonomy of Challenges to Curating Fair Datasets](https://mlsanity.com/paper/2406.06407/) (2024)
- [Multi-Group Proportional Representation in Retrieval](https://mlsanity.com/paper/2407.08571/) (2024)
- [When Stereotypes GTG: The Impact of Predictive Text Suggestions on Gender Bias in Human-AI Co-Writing](https://mlsanity.com/paper/2409.20390/) (2024)
- [The Homogenization Problem in LLMs: Towards Meaningful Diversity in AI Safety](https://mlsanity.com/paper/2601.06116/) (2026)
- [Representational Harms in LLM-Generated Narratives Against Global Majority Nationalities](https://mlsanity.com/paper/2604.22749/) (2026)
- [What Is the Point of Equality in Machine Learning Fairness? Beyond Equality of Opportunity](https://mlsanity.com/paper/2506.16782/) (2025)
- [AI-Assisted Systematization for Evaluating GenAI Systems](https://mlsanity.com/paper/2605.26001/) (2026)
- ["Over-the-Hood" AI Inclusivity Bugs and How 3 AI Product Teams Found and Fixed Them](https://mlsanity.com/paper/2510.19033/) (2025)
- [Representation Fidelity:Auditing Algorithmic Decisions About Humans Using Self-Descriptions](https://mlsanity.com/paper/2603.05136/) (2026)
- [Investigating Counterfactual Unfairness in LLMs towards Identities through Humor](https://mlsanity.com/paper/2604.18729/) (2026)
- [Not What, But How: A Framework for Auditing LLM Responses across Positioning, Generalization, Anthropomorphism, and Maxims](https://mlsanity.com/paper/2606.02493/) (2026)
- [Epistemic Injustice in Language Models: An Audit of Pretraining Filters and Guardrails](https://mlsanity.com/paper/2606.05936/) (2026)
- [On Defining Erasure Harms for NLP](https://mlsanity.com/paper/2606.15815/) (2026)
- [Beyond wheelchairs and blindfolds: Investigating disability stereotypes in T2I models with INCLUDE-BENCH](https://mlsanity.com/paper/2607.08515/) (2026)
- [Beyond "I Can't Help With That": How Child Safety Experts Evaluate AI Chatbot Safety](https://mlsanity.com/paper/2608.07902/) (2026)

## Abstract (arXiv)

In this paper, we examine computational approaches for measuring the "fairness" of image tagging systems, finding that they cluster into five distinct categories, each with its own analytic foundation. We also identify a range of normative concerns that are often collapsed under the terms "unfairness," "bias," or even "discrimination" when discussing problematic cases of image tagging. Specifically, we identify four types of representational harms that can be caused by image tagging systems, providing concrete examples of each. We then consider how different computational measurement approaches map to each of these types, demonstrating that there is not a one-to-one mapping. Our findings emphasize that no single measurement approach will be definitive and that it is not possible to infer from the use of a particular measurement approach which type of harm was intended to be measured. Lastly, equipped with this more granular understanding of the types of representational harms that can be caused by image tagging systems, we show that attempts to mitigate some of these types of harms may be in tension with one another.