mlsanityData to 14 Oct 2026

Designing Disaggregated Evaluations of AI Systems: Choices, Considerations, and Tradeoffs

arXiv 2103.06076 ↗First posted 10 Mar 2021cs.CY cs.AI cs.LGcited by 48

Not on the map. Why?

Abstract (arXiv)

Disaggregated evaluations of AI systems, in which system performance is assessed and reported separately for different groups of people, are conceptually simple. However, their design involves a variety of choices. Some of these choices influence the results that will be obtained, and thus the conclusions that can be drawn; others influence the impacts -- both beneficial and harmful -- that a disaggregated evaluation will have on people, including the people whose data is used to conduct the evaluation. We argue that a deeper understanding of these choices will enable researchers and practitioners to design careful and conclusive disaggregated evaluations. We also argue that better documentation of these choices, along with the underlying considerations and tradeoffs that have been made, will help others when interpreting an evaluation's results and conclusions.

Key facts. Cited by 48 papers in the map (data through 14 Oct 2026). Builds on 17 of its 75 references that are in the map. First posted on arXiv 10 Mar 2021.

Builds on

17 of 75 references.

References · 17

  1. 2018Datasheets for Datasetscited by 1,386
  2. 2016Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddingscited by 1,312
  3. 2018Model Cards for Model Reportingcited by 1,080
  4. 2016Fair prediction with disparate impact: A study of bias in recidivism prediction instrumentscited by 672
  5. 2016Inherent Trade-Offs in the Fair Determination of Risk Scorescited by 586
  6. 2020Closing the AI Accountability Gap: Defining an End-to-End Framework for Internal Algorithmic Auditingcited by 384
  7. 2019Measurement and Fairnesscited by 308
  8. 2019Does Object Recognition Work for Everyone?cited by 151
  9. 2019Towards a Critical Race Methodology in Algorithmic Fairnesscited by 125
  10. 2020Saving Face: Investigating the Ethical Concerns of Facial Recognition Auditingcited by 89
  11. 2020"What We Can't Measure, We Can't Understand": Challenges to Demographic Data Procurement in the Pursuit of Fairnesscited by 56
  12. 2018Towards Accountable AI: Hybrid Human-Machine Analyses for Characterizing System Failurecited by 49
5 more
  1. 2018Racial categories in machine learningcited by 48
  2. 2019Characterizing the Variability in Face Recognition Accuracy Relative to Racecited by 40
  3. 2016Identifying Unknown Unknowns in the Open World: Representations and Policies for Guided Explorationcited by 26
  4. 2018Understanding Unequal Gender Classification Accuracy from Face Imagescited by 25
  5. 2019Awareness in Practice: Tensions in Access to Sensitive Attribute Data for Antidiscriminationcited by 23

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2021: 3; 2022: 9; 2023: 16; 2024: 8; 2025: 5; 2026: 7.

48 papers.

Sort:

By area:

  1. 2022PaLM: Scaling Language Modeling with Pathwayscited by 5,326
  2. 2023PaLM 2 Technical Reportcited by 1,162
  3. 2025Phi-4-reasoning Technical Reportcited by 151
  4. 2023Cultural Bias and Cultural Alignment of Large Language Modelscited by 149
  5. 2023AI Transparency in the Age of LLMs: A Human-Centered Research Roadmapcited by 123
  6. 2023Fairlearn: Assessing and Improving Fairness of AI Systemscited by 74
  7. 2021Assessing the Fairness of AI Systems: AI Practitioners' Processes, Challenges, and Needs for Supportcited by 63
  8. 2022Towards Intersectionality in Machine Learning: Including More Identities, Handling Underrepresentation, and Performing Evaluationcited by 59
  9. 2023AART: AI-Assisted Red-Teaming with Diverse Data Generation for New LLM-powered Applicationscited by 54
  10. 2022Evaluation Gaps in Machine Learning Practicecited by 49
  11. 2022Gender Artifacts in Visual Datasetscited by 38
  12. 2023Vision-Language Models Performing Zero-Shot Tasks Exhibit Gender-based Disparitiescited by 31
36 more
  1. 2022Understanding Machine Learning Practitioners' Data Documentation Perceptions, Needs, Challenges, and Desideratacited by 28
  2. 2024Farsight: Fostering Responsible AI Awareness During AI Application Prototypingcited by 28
  3. 2022From plane crashes to algorithmic harm: applicability of safety engineering frameworks for responsible MLcited by 27
  4. 2023Consensus and Subjectivity of Skin Tone Annotation for ML Fairnesscited by 22
  5. 2023Designerly Understanding: Information Needs for Model Transparency to Support Design Ideation for AI-Powered User Experiencecited by 18
  6. 2022Weak Proxies are Sufficient and Preferable for Fairness with Missing Sensitive Attributescited by 17
  7. 2021Measuring Hidden Bias within Face Recognition via Racial Phenotypescited by 16
  8. 2023Can Fairness be Automated? Guidelines and Opportunities for Fairness-aware AutoMLcited by 16
  9. 2022Demographic-Reliant Algorithmic Fairness: Characterizing the Risks of Demographic Data Collection in the Pursuit of Fairnesscited by 13
  10. 2023Scaling Laws Do Not Scalecited by 13
  11. 2021FairCanary: Rapid Continuous Explainable Fairnesscited by 12
  12. 2023Angler: Helping Machine Translation Practitioners Prioritize Model Improvementscited by 12
  13. 2023fAIlureNotes: Supporting Designers in Understanding the Limits of AI Models for Computer Vision Taskscited by 10
  14. 2023Racial Bias within Face Recognition: A Surveycited by 9
  15. 2024A structured regression approach for evaluating model performance across intersectional subgroupscited by 9
  16. 2023Explaining CLIP's performance disparities on data from blind/low vision userscited by 7
  17. 2025Rigor in AI: Doing Rigorous AI Work Requires a Broader, Responsible AI-Informed Conception of Rigorcited by 7
  18. 2022When Personalization Harms: Reconsidering the Use of Group Attributes in Predictioncited by 6
  19. 2023Towards Reliable Assessments of Demographic Disparities in Multi-Label Image Classifierscited by 5
  20. 2024Policy Maps: Tools for Guiding the Unbounded Space of LLM Behaviorscited by 5
  21. 2026Long-Tail Knowledge in Large Language Models: Taxonomy, Mechanisms, Interventions and Implicationscited by 5
  22. 2024Reconsidering Sentence-Level Sign Language Translationcited by 4
  23. 2024(Beyond) Reasonable Doubt: Challenges that Public Defenders Face in Scrutinizing AI in Courtcited by 3
  24. 2025Addressing Pitfalls in Auditing Practices of Automatic Speech Recognition Technologies: A Case Study of People with Aphasiacited by 3
  25. 2024Analysing and Organising Human Communications for AI Fairness-Related Decisions: Use Cases from the Public Sectorcited by 2
  26. 2026Who Defines "Best"? Towards Interactive, User-Defined Evaluation of LLM Leaderboardscited by 2
  27. 2023Centering the Margins: Outlier-Based Identification of Harmed Populations in Toxicity Detectioncited by 1
  28. 2024SureMap: Simultaneous Mean Estimation for Single-Task and Multi-Task Disaggregated Evaluationcited by 1
  29. 2025We're Still Doing It (All) Wrong: Recommender Systems, Fifteen Years Latercited by 1
  30. 2024Privacy-Preserving Race/Ethnicity Estimation for Algorithmic Bias Measurement in the U.Scited by 0
  31. 2025What Makes Code Generation Ethically Sourced?cited by 0
  32. 2026Autoscoring Anticlimax: A Meta-analytic Understanding of AI's Short-answer Shortcomings and Wording Weaknessescited by 0
  33. 2026Teachers' Perceived Benefits and Risks of AI Across Fifty-Five Countries: An Audit of LLM Alignment and Steerabilitycited by 0
  34. 2026Mechanisms of Misgeneralization in Physical Sequence Modelingcited by 0
  35. 2026Productionized Fairness Measurement Under Privacy Constraintscited by 0
  36. 2026Prediction-Powered Smoothing and Validation for Disaggregated AI Evaluationcited by 0