mlsanityData to 7 Oct 2026

AI Mismatches: Identifying Potential Algorithmic Harms Before AI Development

arXiv 2502.18682 ↗First posted 25 Feb 2025cs.HC cs.AIcited by 10

Not on the map. Why?

Abstract (arXiv)

AI systems are often introduced with high expectations, yet many fail to deliver, resulting in unintended harm and missed opportunities for benefit. We frequently observe significant "AI Mismatches", where the system's actual performance falls short of what is needed to ensure safety and co-create value. These mismatches are particularly difficult to address once development is underway, highlighting the need for early-stage intervention. Navigating complex, multi-dimensional risk factors that contribute to AI Mismatches is a persistent challenge. To address it, we propose an AI Mismatch approach to anticipate and mitigate risks early on, focusing on the gap between realistic model performance and required task performance. Through an analysis of 774 AI cases, we extracted a set of critical factors, which informed the development of seven matrices that map the relationships between these factors and highlight high-risk areas. Through case studies, we demonstrate how our approach can help reduce risks in AI development.

Key facts. Cited by 10 papers in the map (data through 7 Oct 2026). Builds on 44 of its 128 references that are in the map. First posted on arXiv 25 Feb 2025.

Builds on

44 of 128 references.

References · 44

  1. 2018Datasheets for Datasetscited by 1,371
  2. 2018Model Cards for Model Reportingcited by 1,068
  3. 2020Language (Technology) is Power: A Critical Survey of "Bias" in NLPcited by 951
  4. 2016Fair prediction with disparate impact: A study of bias in recidivism prediction instrumentscited by 671
  5. 2016Inherent Trade-Offs in the Fair Determination of Risk Scorescited by 582
  6. 2019Measurement and Fairnesscited by 299
  7. 2024Knowledge Conflicts for LLMs: A Surveycited by 274
  8. 2018Improving fairness in machine learning systems: What do industry practitioners need?cited by 262
  9. 2022Sociotechnical Harms of Algorithmic Systems: Scoping a Taxonomy for Harm Reductioncited by 137
  10. 2022The Fallacy of AI Functionalitycited by 88
  11. 2023The Participatory Turn in AI Design: Theoretical Foundations and the Current State of Practicecited by 86
  12. 2019Problem Formulation and Fairnesscited by 85
32 more
  1. 2021Towards a Science of Human-AI Decision Making: A Survey of Empirical Studiescited by 85
  2. 2020A Case for Humans-in-the-Loop: Decisions in the Presence of Erroneous Algorithmic Scorescited by 74
  3. 2018The age of secrecy and unfairness in recidivism predictioncited by 53
  4. 2019Unremarkable AI: Fitting Intelligent Decision Support into Critical, Clinical Decision-Making Processescited by 53
  5. 2018Residual Unfairness in Fair Machine Learning from Prejudiced Datacited by 49
  6. 2022Evaluation Gaps in Machine Learning Practicecited by 48
  7. 2022Improving Human-AI Partnerships in Child Welfare: Understanding Worker Practices, Challenges, and Desires for Algorithmic Decision Supportcited by 37
  8. 2022Measuring Representational Harms in Image Captioningcited by 36
  9. 2021Emergent Unfairness in Algorithmic Fairness-Accuracy Trade-Off Researchcited by 34
  10. 2017Accelerating Innovation Through Analogy Miningcited by 32
  11. 2022A Validity Perspective on Evaluating the Justified Use of Data-driven Decision-making Algorithmscited by 30
  12. 2022Understanding Practices, Challenges, and Opportunities for User-Engaged Algorithm Auditing in Industry Practicecited by 30
  13. 2023Understanding Frontline Workers' and Unhoused Individuals' Perspectives on AI Used in Homeless Servicescited by 29
  14. 2022Seeing Like a Toolkit: How Toolkits Envision the Work of AI Ethicscited by 28
  15. 2023Ground(less) Truth: A Causal Framework for Proxy Labels in Human-Algorithm Decision-Makingcited by 25
  16. 2020A Human-Centered Review of the Algorithms used within the U.S. Child Welfare Systemcited by 23
  17. 2021A Framework of High-Stakes Algorithmic Decision-Making for the Public Sector Developed through a Case Study of Child-Welfarecited by 23
  18. 2020Overcoming Failures of Imagination in AI Infused System Development and Deploymentcited by 22
  19. 2022Augmenting Scientific Creativity with an Analogical Search Enginecited by 22
  20. 2021Towards A Process Model for Co-Creating AI Experiencescited by 18
  21. 2022Imagining new futures beyond predictive systems in child welfare: A qualitative study with impacted stakeholderscited by 15
  22. 2023Investigating Practices and Opportunities for Cross-functional Collaboration around AI Fairness in Industry Practicecited by 15
  23. 2024The Situate AI Guidebook: Co-Designing a Toolkit to Support Multi-Stakeholder Early-stage Deliberations Around Public Sector AI Proposalscited by 15
  24. 2021Machine Learning Uncertainty as a Design Material: A Post-Phenomenological Inquirycited by 14
  25. 2023Algorithmic Harms in Child Welfare: Uncertainties in Practice, Organization, and Street-level Decision-Makingcited by 12
  26. 2021Orienting, Framing, Bridging, Magic, and Counseling: How Data Scientists Navigate the Outer Loop of Client Collaborations in Industry and Academiacited by 10
  27. 2023Rethinking "Risk" in Algorithmic Systems Through A Computational Narrative Analysis of Casenotes in Child-Welfarecited by 10
  28. 2024Are We Asking the Right Questions?: Designing for Community Stakeholders' Interactions with AI in Policingcited by 10
  29. 2022Unpacking Invisible Work Practices, Constraints, and Latent Power Relationships in Child Welfare through Casenote Analysiscited by 8
  30. 2024Temporal Blind Spots in Large Language Modelscited by 7
  31. 2024A Human-Centered Review of Algorithms in Homelessness Researchcited by 7
  32. 2024(Un)making AI Magic: a Design Taxonomycited by 5

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2025: 5; 2026: 5.

10 papers.

  1. 2025Rigor in AI: Doing Rigorous AI Work Requires a Broader, Responsible AI-Informed Conception of Rigorcited by 7
  2. 2025The Hardness of Achieving Impact in AI for Social Impact Research: A Ground-Level View of Challenges & Opportunitiescited by 1
  3. 2025Measurement as Bricolage: Examining How Data Scientists Construct Target Variables for Predictive Modeling Taskscited by 1
  4. 2025ECHO: A Participatory Framework for Bias-Anchored AI Harm Anticipationcited by 1
  5. 2026Designing Multi-Robot Ground Video Sensemaking with Public Safety Professionalscited by 1
  6. 2025Telling Speculative Stories to Help Humans Imagine the Harms of Healthcare AIcited by 0
  7. 2026Intelligent Reasoning Cues: A Framework and Case Study of the Roles of AI Information in Complex Decisionscited by 0
  8. 2026Red Teaming LLMs as Socio-Technical Practice: From Exploration and Data Creation to Evaluationcited by 0
  9. 2026AI Meets Mathematics Education: A Case Study on Supporting an Instructor in a Large Mathematics Class with Context-Aware AIcited by 0
  10. 2026Human-AI Coordination Zones: A Framework for Designing Human-in-the-Loop Experiences with Agentic AIcited by 0