# Pinpointing Why Object Recognition Performance Degrades Across Income Levels and Geographies

arXiv: https://arxiv.org/abs/2304.05391 · HTML: https://mlsanity.com/paper/2304.05391/

## Key facts

- Cited by 6 papers in the map (data through 14 Oct 2026).
- Builds on 17 of its 32 references that are in the map.
- First posted on arXiv 11 Apr 2023.

## Builds on (most cited first, 17 of 17)

- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [BEiT: BERT Pre-Training of Image Transformers](https://mlsanity.com/paper/2106.08254/) (2021)
- [MLP-Mixer: An all-MLP Architecture for Vision](https://mlsanity.com/paper/2105.01601/) (2021)
- [ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustness](https://mlsanity.com/paper/1811.12231/) (2018)
- [ImageNet-21K Pretraining for the Masses](https://mlsanity.com/paper/2104.10972/) (2021)
- [Self-supervised Pretraining of Visual Features in the Wild](https://mlsanity.com/paper/2103.01988/) (2021)
- [Common pitfalls and recommendations for using machine learning to detect and prognosticate for COVID-19 using chest radiographs and CT scans](https://mlsanity.com/paper/2008.06388/) (2020)
- [No Classification without Representation: Assessing Geodiversity Issues in Open Data Sets for the Developing World](https://mlsanity.com/paper/1711.08536/) (2017)
- [Does Object Recognition Work for Everyone?](https://mlsanity.com/paper/1906.02659/) (2019)
- [REVISE: A Tool for Measuring and Mitigating Bias in Visual Datasets](https://mlsanity.com/paper/2004.07999/) (2020)
- [Re-labeling ImageNet: from Single to Multi-Labels, from Global to Localized Labels](https://mlsanity.com/paper/2101.05022/) (2021)
- [Revisiting Weakly Supervised Pre-Training of Visual Perception Models](https://mlsanity.com/paper/2201.08371/) (2022)
- [Vision Models Are More Robust And Fair When Pretrained On Uncurated Images Without Supervision](https://mlsanity.com/paper/2202.08360/) (2022)
- [Adaptive Methods for Real-World Domain Generalization](https://mlsanity.com/paper/2103.15796/) (2021)
- [ImageNet-X: Understanding Model Mistakes with Factor of Variation Annotations](https://mlsanity.com/paper/2211.01866/) (2022)
- [Fairness Indicators for Systematic Assessments of Visual Feature Extractors](https://mlsanity.com/paper/2202.07603/) (2022)
- [Towards A Fairer Landmark Recognition Dataset](https://mlsanity.com/paper/2108.08874/) (2021)

## Built on by (most cited first, 6 of 6)

- [FACET: Fairness in Computer Vision Evaluation Benchmark](https://mlsanity.com/paper/2309.00035/) (2023)
- [Towards Geographic Inclusion in the Evaluation of Text-to-Image Models](https://mlsanity.com/paper/2405.04457/) (2024)
- [Does Progress On Object Recognition Benchmarks Improve Real-World Generalization?](https://mlsanity.com/paper/2307.13136/) (2023)
- [DIG In: Evaluating Disparities in Image Generations with Indicators for Geographic Diversity](https://mlsanity.com/paper/2308.06198/) (2023)
- [Annotations on a Budget: Leveraging Geo-Data Similarity to Balance Model Performance and Annotation Cost](https://mlsanity.com/paper/2403.07687/) (2024)
- [CRAFT: Cultural Russian-Oriented Dataset Adaptation for Focused Text-to-Image Generation](https://mlsanity.com/paper/2505.04851/) (2025)

## Abstract (arXiv)

Despite impressive advances in object-recognition, deep learning systems' performance degrades significantly across geographies and lower income levels raising pressing concerns of inequity. Addressing such performance gaps remains a challenge, as little is understood about why performance degrades across incomes or geographies. We take a step in this direction by annotating images from Dollar Street, a popular benchmark of geographically and economically diverse images, labeling each image with factors such as color, shape, and background. These annotations unlock a new granular view into how objects differ across incomes and regions. We then use these object differences to pinpoint model vulnerabilities across incomes and regions. We study a range of modern vision models, finding that performance disparities are most associated with differences in texture, occlusion, and images with darker lighting. We illustrate how insights from our factor labels can surface mitigations to improve models' performance disparities. As an example, we show that mitigating a model's vulnerability to texture can improve performance on the lower income level. We release all the factor annotations along with an interactive dashboard to facilitate research into more equitable vision systems.