# "Over-the-Hood" AI Inclusivity Bugs and How 3 AI Product Teams Found and Fixed Them

arXiv: https://arxiv.org/abs/2510.19033 · HTML: https://mlsanity.com/paper/2510.19033/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 12 of its 81 references that are in the map.
- First posted on arXiv 21 Oct 2025.

## Builds on (most cited first, 12 of 12)

- [Using an LLM to Help With Code Understanding](https://mlsanity.com/paper/2307.08177/) (2023)
- [Human-Centered Artificial Intelligence: Reliable, Safe & Trustworthy](https://mlsanity.com/paper/2002.04087/) (2020)
- [Towards Fairer Datasets: Filtering and Balancing the Distribution of the People Subtree in the ImageNet Hierarchy](https://mlsanity.com/paper/1912.07726/) (2019)
- [Designing Creative AI Partners with COFI: A Framework for Modeling Interaction in Human-AI Co-Creative Systems](https://mlsanity.com/paper/2204.07666/) (2022)
- [Taxonomizing and Measuring Representational Harms: A Look at Image Tagging](https://mlsanity.com/paper/2305.01776/) (2023)
- [Reconfiguring Diversity and Inclusion for AI Ethics](https://mlsanity.com/paper/2105.02407/) (2021)
- [Grape Cold Hardiness Prediction via Multi-Task Learning](https://mlsanity.com/paper/2209.10585/) (2022)
- [An HCAI Methodological Framework (HCAI-MF): Putting It Into Action to Enable Human-Centered AI](https://mlsanity.com/paper/2311.16027/) (2023)
- [Measuring User Experience Inclusivity in Human-AI Interaction via Five User Problem-Solving Styles](https://mlsanity.com/paper/2108.00588/) (2021)
- [How to Debug Inclusivity Bugs? A Debugging Process with Information Architecture](https://mlsanity.com/paper/2202.13303/) (2022)
- [Retrospective End-User Walkthrough: A Method for Assessing How People Combine Multiple AI Models in Decision-Making Systems](https://mlsanity.com/paper/2305.07530/) (2023)
- [Inclusive Design of AI's Explanations: Just for Those Previously Left Out, or for Everyone?](https://mlsanity.com/paper/2404.13217/) (2024)

## Abstract (arXiv)

While much research has shown the presence of AI's "under-the-hood" biases (e.g., algorithmic, training data, etc.), what about "over-the-hood" inclusivity biases: barriers in user-facing AI products that disproportionately exclude users with certain problem-solving approaches? Recent research has begun to report the existence of such biases -- but what do they look like, how prevalent are they, and how can developers find and fix them? To find out, we conducted a field study with 3 AI product teams, to investigate what kinds of AI inclusivity bugs exist uniquely in user-facing AI products, and whether/how AI product teams might harness an existing (non-AI-oriented) inclusive design method to find and fix them. The teams' work resulted in identifying 6 types of AI inclusivity bugs arising 83 times, fixes covering 47 of these bug instances, and a new variation of the GenderMag inclusive design method, GenderMag-for-AI, that is especially effective at detecting certain kinds of AI inclusivity bugs.