# Towards Reliable Assessments of Demographic Disparities in Multi-Label Image Classifiers arXiv: https://arxiv.org/abs/2302.08572 ยท HTML: https://mlsanity.com/paper/2302.08572/ ## Key facts - Cited by 5 papers in the map (data through 14 Oct 2026). - Builds on 32 of its 66 references that are in the map. - First posted on arXiv 16 Feb 2023. ## Builds on (most cited first, 30 of 32) - [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021) - [An Image is Worth 16x16 Words: Transformers for Image Recognition at Scale](https://mlsanity.com/paper/2010.11929/) (2020) - [Microsoft COCO: Common Objects in Context](https://mlsanity.com/paper/1405.0312/) (2014) - [Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations](https://mlsanity.com/paper/1602.07332/) (2016) - [Fair prediction with disparate impact: A study of bias in recidivism prediction instruments](https://mlsanity.com/paper/1610.07524/) (2016) - [Deep Visual Domain Adaptation: A Survey](https://mlsanity.com/paper/1802.03601/) (2018) - [ImageNet-21K Pretraining for the Masses](https://mlsanity.com/paper/2104.10972/) (2021) - [Men Also Like Shopping: Reducing Gender Bias Amplification using Corpus-level Constraints](https://mlsanity.com/paper/1707.09457/) (2017) - [The Values Encoded in Machine Learning Research](https://mlsanity.com/paper/2106.15590/) (2021) - [No Classification without Representation: Assessing Geodiversity Issues in Open Data Sets for the Developing World](https://mlsanity.com/paper/1711.08536/) (2017) - [Does Object Recognition Work for Everyone?](https://mlsanity.com/paper/1906.02659/) (2019) - [REVISE: A Tool for Measuring and Mitigating Bias in Visual Datasets](https://mlsanity.com/paper/2004.07999/) (2020) - [Evaluating CLIP: Towards Characterization of Broader Capabilities and Downstream Implications](https://mlsanity.com/paper/2108.02818/) (2021) - [Towards a Critical Race Methodology in Algorithmic Fairness](https://mlsanity.com/paper/1912.03593/) (2019) - [TResNet: High Performance GPU-Dedicated Architecture](https://mlsanity.com/paper/2003.13630/) (2020) - [Re-labeling ImageNet: from Single to Multi-Labels, from Global to Localized Labels](https://mlsanity.com/paper/2101.05022/) (2021) - [Revisiting Weakly Supervised Pre-Training of Visual Perception Models](https://mlsanity.com/paper/2201.08371/) (2022) - [Decolonial AI: Decolonial Theory as Sociotechnical Foresight in Artificial Intelligence](https://mlsanity.com/paper/2007.04068/) (2020) - [Towards Measuring Fairness in AI: the Casual Conversations Dataset](https://mlsanity.com/paper/2104.02821/) (2021) - [Vision Models Are More Robust And Fair When Pretrained On Uncurated Images Without Supervision](https://mlsanity.com/paper/2202.08360/) (2022) - [A Systematic Study of Bias Amplification](https://mlsanity.com/paper/2201.11706/) (2022) - [Theories of "Gender" in NLP Bias Research](https://mlsanity.com/paper/2205.02526/) (2022) - [Directional Bias Amplification](https://mlsanity.com/paper/2102.12594/) (2021) - [Racial categories in machine learning](https://mlsanity.com/paper/1811.11668/) (2018) - [Designing Disaggregated Evaluations of AI Systems: Choices, Considerations, and Tradeoffs](https://mlsanity.com/paper/2103.06076/) (2021) - [A Step Toward More Inclusive People Annotations for Fairness](https://mlsanity.com/paper/2105.02317/) (2021) - [ImageNet-X: Understanding Model Mistakes with Factor of Variation Annotations](https://mlsanity.com/paper/2211.01866/) (2022) - [Fairness for Unobserved Characteristics: Insights from Technological Impacts on Queer Communities](https://mlsanity.com/paper/2102.04257/) (2021) - [Vision-Language Models Performing Zero-Shot Tasks Exhibit Gender-based Disparities](https://mlsanity.com/paper/2301.11100/) (2023) - [Evidence for Hypodescent in Visual Semantic AI](https://mlsanity.com/paper/2205.10764/) (2022) ## Built on by (most cited first, 5 of 5) - [FACET: Fairness in Computer Vision Evaluation Benchmark](https://mlsanity.com/paper/2309.00035/) (2023) - [A Closer Look at AUROC and AUPRC under Class Imbalance](https://mlsanity.com/paper/2401.06091/) (2024) - [ONEBench to Test Them All: Sample-Level Benchmarking Over Open-Ended Capabilities](https://mlsanity.com/paper/2412.06745/) (2024) - [DIG In: Evaluating Disparities in Image Generations with Indicators for Geographic Diversity](https://mlsanity.com/paper/2308.06198/) (2023) - [Annotations on a Budget: Leveraging Geo-Data Similarity to Balance Model Performance and Annotation Cost](https://mlsanity.com/paper/2403.07687/) (2024) ## Abstract (arXiv) Disaggregated performance metrics across demographic groups are a hallmark of fairness assessments in computer vision. These metrics successfully incentivized performance improvements on person-centric tasks such as face analysis and are used to understand risks of modern models. However, there is a lack of discussion on the vulnerabilities of these measurements for more complex computer vision tasks. In this paper, we consider multi-label image classification and, specifically, object categorization tasks. First, we highlight design choices and trade-offs for measurement that involve more nuance than discussed in prior computer vision literature. These challenges are related to the necessary scale of data, definition of groups for images, choice of metric, and dataset imbalances. Next, through two case studies using modern vision models, we demonstrate that naive implementations of these assessments are brittle. We identify several design choices that look merely like implementation details but significantly impact the conclusions of assessments, both in terms of magnitude and direction (on which group the classifiers work best) of disparities. Based on ablation studies, we propose some recommendations to increase the reliability of these assessments. Finally, through a qualitative analysis we find that concepts with large disparities tend to have varying definitions and representations between groups, with inconsistencies across datasets and annotators. While this result suggests avenues for mitigation through more consistent data collection, it also highlights that ambiguous label definitions remain a challenge when performing model assessments. Vision models are expanding and becoming more ubiquitous; it is even more important that our disparity assessments accurately reflect the true performance of models.