mlsanityData to 14 Oct 2026

FACET: Fairness in Computer Vision Evaluation Benchmark

arXiv 2309.00035 ↗First posted 31 Aug 2023cs.CV cs.AIcited by 58

Not on the map. Why?

Abstract (arXiv)

Computer vision models have known performance disparities across attributes such as gender and skin tone. This means during tasks such as classification and detection, model performance differs for certain classes based on the demographics of the people in the image. These disparities have been shown to exist, but until now there has not been a unified approach to measure these differences for common use-cases of computer vision models. We present a new benchmark named FACET (FAirness in Computer Vision EvaluaTion), a large, publicly available evaluation set of 32k images for some of the most common vision tasks - image classification, object detection and segmentation. For every image in FACET, we hired expert reviewers to manually annotate person-related attributes such as perceived skin tone and hair type, manually draw bounding boxes and label fine-grained person-related classes such as disk jockey or guitarist. In addition, we use FACET to benchmark state-of-the-art vision models and present a deeper understanding of potential performance disparities and challenges across sensitive demographic attributes. With the exhaustive annotations collected, we probe models using single demographics attributes as well as multiple attributes using an intersectional approach (e.g. hair color and perceived skin tone). Our results show that classification, detection, segmentation, and visual grounding models exhibit performance disparities across demographic attributes and intersections of attributes. These harms suggest that not all people represented in datasets receive fair and equitable treatment in these vision tasks. We hope current and future results using our benchmark will contribute to fairer, more robust vision models. FACET is available publicly at https://facet.metademolab.com/

Key facts. Cited by 58 papers in the map (data through 14 Oct 2026). Builds on 59 of its 103 references that are in the map. First posted on arXiv 31 Aug 2023.

Builds on

59 of 103 references.

References · 59

  1. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,744
  2. 2020An Image is Worth 16x16 Words: Transformers for Image Recognition at Scalecited by 30,186
  3. 2014Microsoft COCO: Common Objects in Contextcited by 17,608
  4. 2015Faster R-CNN: Towards Real-Time Object Detection with Region Proposal Networkscited by 9,639
  5. 2023Segment Anythingcited by 8,234
  6. 2020End-to-End Object Detection with Transformerscited by 7,414
  7. 2017Mask R-CNNcited by 7,277
  8. 2014Deep Learning Face Attributes in the Wildcited by 3,693
  9. 2016Equality of Opportunity in Supervised Learningcited by 2,007
  10. 2018ImageNet-trained CNNs are biased towards texture; increasing shape bias improves accuracy and robustnesscited by 1,432
  11. 2018Datasheets for Datasetscited by 1,386
  12. 2018BDD100K: A Diverse Driving Dataset for Heterogeneous Multitask Learningcited by 1,288
38 more
  1. 2018Model Cards for Model Reportingcited by 1,080
  2. 2022Detecting Twenty-thousand Classes using Image-level Supervisioncited by 614
  3. 2021ImageNet-21K Pretraining for the Massescited by 613
  4. 2018The Open Images Dataset V4: Unified image classification, object detection, and visual relationship detection at scalecited by 544
  5. 2017Age Progression/Regression by Conditional Adversarial Autoencodercited by 429
  6. 2017Men Also Like Shopping: Reducing Gender Bias Amplification using Corpus-level Constraintscited by 420
  7. 2021Multimodal datasets: misogyny, pornography, and malignant stereotypescited by 327
  8. 2019Approximating CNNs with Bag-of-local-Features models works surprisingly well on ImageNetcited by 260
  9. 2020Data and its (dis)contents: A survey of dataset development and use in machine learning researchcited by 260
  10. 2022Data Cards: Purposeful and Transparent Dataset Documentation for Responsible AIcited by 171
  11. 2017No Classification without Representation: Assessing Geodiversity Issues in Open Data Sets for the Developing Worldcited by 153
  12. 2019Does Object Recognition Work for Everyone?cited by 151
  13. 2020Towards Accountability for Machine Learning Datasets: Practices from Software Engineering and Infrastructurecited by 138
  14. 2020REVISE: A Tool for Measuring and Mitigating Bias in Visual Datasetscited by 136
  15. 2020The DeepFake Detection Challenge (DFDC) Datasetcited by 132
  16. 2019Lessons from Archives: Strategies for Collecting Sociocultural Data in Machine Learningcited by 116
  17. 2022Revisiting Weakly Supervised Pre-Training of Visual Perception Modelscited by 116
  18. 2021Understanding and Evaluating Racial Biases in Image Captioningcited by 108
  19. 2020Image Representations Learned With Unsupervised Pre-Training Contain Human-like Biasescited by 107
  20. 2021Reduced, Reused and Recycled: The Life of a Dataset in Machine Learning Researchcited by 102
  21. 2018Benchmarking Neural Network Robustness to Common Corruptions and Surface Variationscited by 99
  22. 2018The Dataset Nutrition Label: A Framework To Drive Higher Data Quality Standardscited by 97
  23. 2021Do Datasets Have Politics? Disciplinary Values in Computer Vision Dataset Developmentcited by 85
  24. 2021Towards Measuring Fairness in AI: the Casual Conversations Datasetcited by 80
  25. 2019Predictive Inequity in Object Detectioncited by 77
  26. 2022CrowdWorkSheets: Accounting for Individual and Collective Identities Underlying Crowdsourced Dataset Annotationcited by 59
  27. 2021Directional Bias Amplificationcited by 54
  28. 2021Whose Ground Truth? Accounting for Individual and Collective Identities Underlying Dataset Annotationcited by 52
  29. 2021A Step Toward More Inclusive People Annotations for Fairnesscited by 46
  30. 2022ImageNet-X: Understanding Model Mistakes with Factor of Variation Annotationscited by 44
  31. 2020Bringing the People Back In: Contesting Benchmark Machine Learning Datasetscited by 43
  32. 2015What makes for effective detection proposals?cited by 38
  33. 2022Gender Artifacts in Visual Datasetscited by 38
  34. 2022Towards Transparency in Dermatology Image Datasets with Skin Tone Annotations by Experts, Crowds, and an Algorithmcited by 35
  35. 2021One Label, One Billion Faces: Usage and Consistency of Racial Categories in Computer Visioncited by 33
  36. 2023Vision-Language Models Performing Zero-Shot Tasks Exhibit Gender-based Disparitiescited by 31
  37. 2022Fairness Indicators for Systematic Assessments of Visual Feature Extractorscited by 24
  38. 2022Men Also Do Laundry: Multi-Attribute Bias Amplificationcited by 21

50 most cited of 59 All 59 (JSON)

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2023: 2; 2024: 19; 2025: 22; 2026: 15.

58 papers.

Sort:

By area:

  1. 2024Scissorhands: Scrub Data Influence via Connection Sensitivity in Networkscited by 52
  2. 2025T2ISafety: Benchmark for Assessing Fairness, Toxicity, and Privacy in Image Generationcited by 48
  3. 2024ViSAGe: A Global-Scale Analysis of Visual Stereotypes in Text-to-Image Generationcited by 27
  4. 2023Image Clustering Conditioned on Text Criteriacited by 26
  5. 2024SANER: Annotation-free Societal Attribute Neutralizer for Debiasing CLIPcited by 19
  6. 2024A Unified Debiasing Approach for Vision-Language Models across Modalities and Taskscited by 18
  7. 2024Joint Vision-Language Social Bias Removal for CLIPcited by 18
  8. 2023Quantifying Bias in Text-to-Image Generative Modelscited by 16
  9. 2024Evaluating Fairness in Large Vision-Language Models Across Diverse Demographic Attributes and Promptscited by 15
  10. 2024Conformal Prediction Sets Can Cause Disparate Impactcited by 14
  11. 2024Fairness and Bias Mitigation in Computer Vision: A Surveycited by 13
  12. 2024FairDeDup: Detecting and Mitigating Vision-Language Fairness Disparities in Semantic Dataset Deduplicationcited by 11
38 more
  1. 2025BBQ-V: Benchmarking Visual Stereotype Bias in Large Multimodal Modelscited by 11
  2. 2025CAD-VAE: Leveraging Correlation-Aware Latents for Comprehensive Fair Disentanglementcited by 11
  3. 2024Distributionally Generative Augmentation for Fair Facial Attribute Classificationcited by 8
  4. 2024Organizing Unstructured Image Collections using Natural Languagecited by 7
  5. 2025Bias Is a Subspace, Not a Coordinate: A Geometric Rethinking of Post-hoc Debiasing in Vision-Language Modelscited by 7
  6. 2024FairJob: A Real-World Dataset for Fairness in Online Systemscited by 6
  7. 2024Visual Data Diagnosis and Debiasing with Concept Graphscited by 6
  8. 2025A Survey on 3D Reconstruction Techniques in Plant Phenotyping: From Classical Methods to Neural Radiance Fields (NeRF), 3D Gaussian Splatting (3DGS), and Beyondcited by 6
  9. 2024Hairmony: Fairness-aware hairstyle classificationcited by 5
  10. 2025ALBAR: Adversarial Learning approach to mitigate Biases in Action Recognitioncited by 5
  11. 2025Bias in Gender Bias Benchmarks: How Spurious Features Distort Evaluationcited by 5
  12. 2025Fairness-Aware Grouping for Continuous Sensitive Variables: Application for Debiasing Face Analysis with respect to Skin Tonecited by 4
  13. 2025Person-Centric Annotations of LAION-400M: Auditing Bias and Its Transfer to Modelscited by 4
  14. 2024A Taxonomy of Challenges to Curating Fair Datasetscited by 2
  15. 2024You Never Know: Quantization Induces Inconsistent Biases in Vision-Language Foundation Modelscited by 2
  16. 2025MoENAS: Mixture-of-Expert based Neural Architecture Search for jointly Accurate, Fair, and Robust Edge Deep Neural Networkscited by 2
  17. 2025T2IBias: Uncovering Societal Bias Encoded in the Latent Space of Text-to-Image Generative Modelscited by 2
  18. 2026Beyond Procedure: Substantive Fairness in Conformal Predictioncited by 2
  19. 2026Locating Demographic Bias at the Attention-Head Level in CLIP's Vision Encodercited by 2
  20. 2024Constructing Fair Latent Space for Intersection of Fairness and Explainabilitycited by 1
  21. 2025A Multi-Objective Evaluation Framework for Analyzing Utility-Fairness Trade-Offs in Machine Learning Systemscited by 1
  22. 2025RESFL: An Uncertainty-Aware Framework for Responsible Federated Learning by Balancing Privacy, Fairness and Utilitycited by 1
  23. 2025Interpreting Social Bias in LVLMs via Information Flow Analysis and Multi-Round Dialogue Evaluationcited by 1
  24. 2025Evaluating Fairness and Mitigating Bias in Machine Learning: A Novel Technique using Tensor Data and Bayesian Regressioncited by 1
  25. 2025An Evaluation of Hybrid Annotation Workflows on High-Ambiguity Spatiotemporal Video Footagecited by 1
  26. 2025Privacy Beyond Pixels: Latent Anonymization for Privacy-Preserving Video Understandingcited by 1
  27. 2026Benchmarking Bias Mitigation Toward Fairness Without Harm from Vision to LVLMscited by 1
  28. 2026On the Burden of Achieving Fairness in Conformal Predictioncited by 1
  29. 2024DispaRisk: Auditing Fairness Through Usable Informationcited by 0
  30. 2024Debias your Large Multi-Modal Model at Test-Time via Non-Contrastive Visual Attribute Steeringcited by 0
  31. 2025TrustSkin: A Fairness Pipeline for Trustworthy Facial Affect Analysis Across Skin Tonecited by 0
  32. 2025I Stolenly Swear That I Am Up to (No) Good: Design and Evaluation of Model Stealing Attackscited by 0
  33. 2025On the Reproducibility of "FairCLIP: Harnessing Fairness in Vision-Language Learning''cited by 0
  34. 2025Multi-Modal Sentiment Analysis with Dynamic Attention Fusioncited by 0
  35. 2025Emergent AI Surveillance: Overlearned Person Re-Identification and Its Mitigation in Law Enforcement Contextcited by 0
  36. 2026PrivacyBench: Privacy Isn't Free in Hybrid Privacy-Preserving Vision Systemscited by 0
  37. 2026Skin Tone in the Wild: A Dataset and Benchmark for 10-tone MST Classificationcited by 0
  38. 2026Bias Redistribution in Visual Machine Unlearning: Does Forgetting One Group Harm Another?cited by 0

50 most cited of 58 All 58 (JSON)