# Investigating the Capabilities and Limitations of Machine Learning for Identifying Bias in English Language Data with Information and Heritage Professionals

arXiv: https://arxiv.org/abs/2504.00860 · HTML: https://mlsanity.com/paper/2504.00860/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 35 of its 150 references that are in the map.
- First posted on arXiv 1 Apr 2025.

## Builds on (most cited first, 30 of 35)

- [A Survey on Bias and Fairness in Machine Learning](https://mlsanity.com/paper/1908.09635/) (2019)
- [Enriching Word Vectors with Subword Information](https://mlsanity.com/paper/1607.04606/) (2016)
- [Language (Technology) is Power: A Critical Survey of "Bias" in NLP](https://mlsanity.com/paper/2005.14050/) (2020)
- [AI and the Everything in the Whole Wide World Benchmark](https://mlsanity.com/paper/2111.15366/) (2021)
- [Mitigating Gender Bias in Natural Language Processing: Literature Review](https://mlsanity.com/paper/1906.08976/) (2019)
- [Improving fairness in machine learning systems: What do industry practitioners need?](https://mlsanity.com/paper/1812.05239/) (2018)
- [Data and its (dis)contents: A survey of dataset development and use in machine learning research](https://mlsanity.com/paper/2012.05345/) (2020)
- [Questioning the AI: Informing Design Practices for Explainable AI User Experiences](https://mlsanity.com/paper/2001.02478/) (2020)
- [Large image datasets: A pyrrhic win for computer vision?](https://mlsanity.com/paper/2006.16923/) (2020)
- [The Values Encoded in Machine Learning Research](https://mlsanity.com/paper/2106.15590/) (2021)
- [Participatory Research for Low-resourced Machine Translation: A Case Study in African Languages](https://mlsanity.com/paper/2010.02353/) (2020)
- [Intrinsic Bias Metrics Do Not Correlate with Application Bias](https://mlsanity.com/paper/2012.15859/) (2020)
- [Toward a Perspectivist Turn in Ground Truthing for Predictive Computing](https://mlsanity.com/paper/2109.04270/) (2021)
- [Bias Mitigation for Machine Learning Classifiers: A Comprehensive Survey](https://mlsanity.com/paper/2207.07068/) (2022)
- [Human-Centered Explainable AI (XAI): From Algorithms to User Experiences](https://mlsanity.com/paper/2110.10790/) (2021)
- [Lessons from Archives: Strategies for Collecting Sociocultural Data in Machine Learning](https://mlsanity.com/paper/1912.10389/) (2019)
- [Re-imagining Algorithmic Fairness in India and Beyond](https://mlsanity.com/paper/2101.09995/) (2021)
- [The Measure and Mismeasure of Fairness](https://mlsanity.com/paper/1808.00023/) (2018)
- [Toward Gender-Inclusive Coreference Resolution](https://mlsanity.com/paper/1910.13913/) (2019)
- [Multi-Dimensional Gender Bias Classification](https://mlsanity.com/paper/2005.00614/) (2020)
- [In conversation with Artificial Intelligence: aligning language models with human values](https://mlsanity.com/paper/2209.00731/) (2022)
- [Don't Blame the Annotator: Bias Already Starts in the Annotation Instructions](https://mlsanity.com/paper/2205.00415/) (2022)
- [Putting Humans in the Natural Language Processing Loop: A Survey](https://mlsanity.com/paper/2103.04044/) (2021)
- [Are We Consistently Biased? Multidimensional Analysis of Biases in Distributional Word Vectors](https://mlsanity.com/paper/1904.11783/) (2019)
- [Choose Your Lenses: Flaws in Gender Bias Evaluation](https://mlsanity.com/paper/2210.11471/) (2022)
- [Gender Bias in BERT -- Measuring and Analysing Biases through Sentiment Rating in a Realistic Downstream Classification Task](https://mlsanity.com/paper/2306.15298/) (2023)
- [Examining Covert Gender Bias: A Case Study in Turkish and English Machine Translation Models](https://mlsanity.com/paper/2108.10379/) (2021)
- [Algorithmic Fairness and Structural Injustice: Insights from Feminist Political Philosophy](https://mlsanity.com/paper/2206.00945/) (2022)
- [Case Study: Predictive Fairness to Reduce Misdemeanor Recidivism Through Social Service Interventions](https://mlsanity.com/paper/2001.09233/) (2020)
- [Detecting Unintended Social Bias in Toxic Language Datasets](https://mlsanity.com/paper/2210.11762/) (2022)

## Abstract (arXiv)

Despite numerous efforts to mitigate their biases, ML systems continue to harm already-marginalized people. While predominant ML approaches assume bias can be removed and fair models can be created, we show that these are not always possible, nor desirable, goals. We reframe the problem of ML bias by creating models to identify biased language, drawing attention to a dataset's biases rather than trying to remove them. Then, through a workshop, we evaluated the models for a specific use case: workflows of information and heritage professionals. Our findings demonstrate the limitations of ML for identifying bias due to its contextual nature, the way in which approaches to mitigating it can simultaneously privilege and oppress different communities, and its inevitability. We demonstrate the need to expand ML approaches to bias and fairness, providing a mixed-methods approach to investigating the feasibility of removing bias or achieving fairness in a given ML use case.