# Disembodied Machine Learning: On the Illusion of Objectivity in NLP

arXiv: https://arxiv.org/abs/2101.11974 · HTML: https://mlsanity.com/paper/2101.11974/

## Key facts

- Cited by 37 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 41 references that are in the map.
- First posted on arXiv 28 Jan 2021.

## Builds on (most cited first, 19 of 19)

- [BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding](https://mlsanity.com/paper/1810.04805/) (2018)
- [Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddings](https://mlsanity.com/paper/1607.06520/) (2016)
- [Model Cards for Model Reporting](https://mlsanity.com/paper/1810.03993/) (2018)
- [Automated Hate Speech Detection and the Problem of Offensive Language](https://mlsanity.com/paper/1703.04009/) (2017)
- [A Reductions Approach to Fair Classification](https://mlsanity.com/paper/1803.02453/) (2018)
- [Men Also Like Shopping: Reducing Gender Bias Amplification using Corpus-level Constraints](https://mlsanity.com/paper/1707.09457/) (2017)
- [Lipstick on a Pig: Debiasing Methods Cover up Systematic Gender Biases in Word Embeddings But do not Remove Them](https://mlsanity.com/paper/1903.03862/) (2019)
- [Improving fairness in machine learning systems: What do industry practitioners need?](https://mlsanity.com/paper/1812.05239/) (2018)
- [Racial Bias in Hate Speech and Abusive Language Detection Datasets](https://mlsanity.com/paper/1905.12516/) (2019)
- [Demographic Dialectal Variation in Social Media: A Case Study of African-American English](https://mlsanity.com/paper/1608.08868/) (2016)
- [Predictive Biases in Natural Language Processing Models: A Conceptual Framework and Overview](https://mlsanity.com/paper/1912.11078/) (2019)
- [Assessing Social and Intersectional Biases in Contextualized Word Representations](https://mlsanity.com/paper/1911.01485/) (2019)
- [Getting Gender Right in Neural Machine Translation](https://mlsanity.com/paper/1909.05088/) (2019)
- [What's in a Name? Reducing Bias in Bios without Access to Protected Attributes](https://mlsanity.com/paper/1904.05233/) (2019)
- [Mapping Languages: The Corpus of Global Language Use](https://mlsanity.com/paper/2004.00798/) (2020)
- [POTs: Protective Optimization Technologies](https://mlsanity.com/paper/1806.02711/) (2018)
- [Multi-Task Learning for Mental Health using Social Media Text](https://mlsanity.com/paper/1712.03538/) (2017)
- [Context-Aware Embeddings for Automatic Art Analysis](https://mlsanity.com/paper/1904.04985/) (2019)
- [Par4Sim -- Adaptive Paraphrasing for Text Simplification](https://mlsanity.com/paper/1806.08309/) (2018)

## Built on by (most cited first, 30 of 37)

- [AI and the Everything in the Whole Wide World Benchmark](https://mlsanity.com/paper/2111.15366/) (2021)
- [MERLOT: Multimodal Neural Script Knowledge Models](https://mlsanity.com/paper/2106.02636/) (2021)
- [Annotators with Attitudes: How Annotator Beliefs And Identities Bias Toxic Language Detection](https://mlsanity.com/paper/2111.07997/) (2021)
- [Do All Languages Cost the Same? Tokenization in the Era of Commercial Language Models](https://mlsanity.com/paper/2305.13707/) (2023)
- [MERLOT Reserve: Neural Script Knowledge through Vision and Language and Sound](https://mlsanity.com/paper/2201.02639/) (2022)
- [Gender Bias in Machine Translation](https://mlsanity.com/paper/2104.06001/) (2021)
- [NLPositionality: Characterizing Design Biases of Datasets and Models](https://mlsanity.com/paper/2306.01943/) (2023)
- [Evaluating the Social Impact of Generative AI Systems in Systems and Society](https://mlsanity.com/paper/2306.05949/) (2023)
- [A Survey on Stance Detection for Mis- and Disinformation Identification](https://mlsanity.com/paper/2103.00242/) (2021)
- [A Survey of Race, Racism, and Anti-Racism in NLP](https://mlsanity.com/paper/2106.11410/) (2021)
- [The Perspectivist Paradigm Shift: Assumptions and Challenges of Capturing Human Labels](https://mlsanity.com/paper/2405.05860/) (2024)
- [Bootstrapping LLM-based Task-Oriented Dialogue Agents via Self-Talk](https://mlsanity.com/paper/2401.05033/) (2024)
- [Data Governance in the Age of Large-Scale Data-Driven Language Technology](https://mlsanity.com/paper/2206.03216/) (2022)
- [Cross-Domain Label-Adaptive Stance Detection](https://mlsanity.com/paper/2104.07467/) (2021)
- [Few-Shot Cross-Lingual Stance Detection with Sentiment-Based Pre-Training](https://mlsanity.com/paper/2109.06050/) (2021)
- [Factoring the Matrix of Domination: A Critical Review and Reimagination of Intersectionality in AI Fairness](https://mlsanity.com/paper/2303.17555/) (2023)
- [Reliability Testing for Natural Language Processing Systems](https://mlsanity.com/paper/2105.02590/) (2021)
- [Reusable Templates and Guides For Documenting Datasets and Models for Natural Language Processing and Generation: A Case Study of the HuggingFace and GEM Data and Model Cards](https://mlsanity.com/paper/2108.07374/) (2021)
- [Stop! In the Name of Flaws: Disentangling Personal Names and Sociodemographic Attributes in NLP](https://mlsanity.com/paper/2405.17159/) (2024)
- [Abuse is Contextual, What about NLP? The Role of Context in Abusive Language Annotation and Detection](https://mlsanity.com/paper/2103.14916/) (2021)
- [ACTOR: Active Learning with Annotator-specific Classification Heads to Embrace Human Label Variation](https://mlsanity.com/paper/2310.14979/) (2023)
- [Vote'n'Rank: Revision of Benchmarking with Social Choice Theory](https://mlsanity.com/paper/2210.05769/) (2022)
- [TAPE: Assessing Few-shot Russian Language Understanding](https://mlsanity.com/paper/2210.12813/) (2022)
- [D3CODE: Disentangling Disagreements in Data across Cultures on Offensiveness Detection and Evaluation](https://mlsanity.com/paper/2404.10857/) (2024)
- [Experimental Standards for Deep Learning in Natural Language Processing Research](https://mlsanity.com/paper/2204.06251/) (2022)
- [How Are LLMs Mitigating Stereotyping Harms? Learning from Search Engine Studies](https://mlsanity.com/paper/2407.11733/) (2024)
- [Discipline and Label: A WEIRD Genealogy and Social Theory of Data Annotation](https://mlsanity.com/paper/2402.06811/) (2024)
- [GREENER: Graph Neural Networks for News Media Profiling](https://mlsanity.com/paper/2211.05533/) (2022)
- [Back to the Future: On Potential Histories in NLP](https://mlsanity.com/paper/2210.06245/) (2022)
- [What if AI systems weren't chatbots?](https://mlsanity.com/paper/2605.07896/) (2026)

## Abstract (arXiv)

Machine Learning seeks to identify and encode bodies of knowledge within provided datasets. However, data encodes subjective content, which determines the possible outcomes of the models trained on it. Because such subjectivity enables marginalisation of parts of society, it is termed (social) `bias' and sought to be removed. In this paper, we contextualise this discourse of bias in the ML community against the subjective choices in the development process. Through a consideration of how choices in data and model development construct subjectivity, or biases that are represented in a model, we argue that addressing and mitigating biases is near-impossible. This is because both data and ML models are objects for which meaning is made in each step of the development pipeline, from data selection over annotation to model training and analysis. Accordingly, we find the prevalent discourse of bias limiting in its ability to address social marginalisation. We recommend to be conscientious of this, and to accept that de-biasing methods only correct for a fraction of biases.