# Intersectional Bias in Causal Language Models

arXiv: https://arxiv.org/abs/2107.07691 · HTML: https://mlsanity.com/paper/2107.07691/

## Key facts

- Cited by 24 papers in the map (data through 14 Oct 2026).
- Builds on 26 of its 55 references that are in the map.
- First posted on arXiv 16 Jul 2021.

## Builds on (most cited first, 26 of 26)

- [Attention Is All You Need](https://mlsanity.com/paper/1706.03762/) (2017)
- [Language Models are Few-Shot Learners](https://mlsanity.com/paper/2005.14165/) (2020)
- [Efficient Estimation of Word Representations in Vector Space](https://mlsanity.com/paper/1301.3781/) (2013)
- [DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter](https://mlsanity.com/paper/1910.01108/) (2019)
- [The Curious Case of Neural Text Degeneration](https://mlsanity.com/paper/1904.09751/) (2019)
- [The Pile: An 800GB Dataset of Diverse Text for Language Modeling](https://mlsanity.com/paper/2101.00027/) (2020)
- [Deep contextualized word representations](https://mlsanity.com/paper/1802.05365/) (2018)
- [Calibrate Before Use: Improving Few-Shot Performance of Language Models](https://mlsanity.com/paper/2102.09690/) (2021)
- [Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddings](https://mlsanity.com/paper/1607.06520/) (2016)
- [Hierarchical Neural Story Generation](https://mlsanity.com/paper/1805.04833/) (2018)
- [Beyond Accuracy: Behavioral Testing of NLP models with CheckList](https://mlsanity.com/paper/2005.04118/) (2020)
- [Gender Bias in Coreference Resolution: Evaluation and Debiasing Methods](https://mlsanity.com/paper/1804.06876/) (2018)
- [Gender Bias in Coreference Resolution](https://mlsanity.com/paper/1804.09301/) (2018)
- [Persistent Anti-Muslim Bias in Large Language Models](https://mlsanity.com/paper/2101.05783/) (2021)
- [Mitigating Gender Bias in Natural Language Processing: Literature Review](https://mlsanity.com/paper/1906.08976/) (2019)
- [Evaluating Gender Bias in Machine Translation](https://mlsanity.com/paper/1906.00591/) (2019)
- [Identifying and Reducing Gender Bias in Word-Level Language Models](https://mlsanity.com/paper/1904.03035/) (2019)
- [Gender Bias in Contextualized Word Embeddings](https://mlsanity.com/paper/1904.03310/) (2019)
- [Detecting Emergent Intersectional Biases: Contextualized Word Embeddings Contain a Distribution of Human-like Biases](https://mlsanity.com/paper/2006.03955/) (2020)
- [Reducing Sentiment Bias in Language Models via Counterfactual Evaluation](https://mlsanity.com/paper/1911.03064/) (2019)
- [Evaluating the Underlying Gender Bias in Contextualized Word Embeddings](https://mlsanity.com/paper/1904.08783/) (2019)
- [Gender-preserving Debiasing for Pre-trained Word Embeddings](https://mlsanity.com/paper/1906.00742/) (2019)
- [Fine-grained Sentiment Classification using BERT](https://mlsanity.com/paper/1910.03474/) (2019)
- [Toward Fairness in AI for People with Disabilities: A Research Roadmap](https://mlsanity.com/paper/1907.02227/) (2019)
- [Intersectional Bias in Hate Speech and Abusive Language Datasets](https://mlsanity.com/paper/2005.05921/) (2020)
- [AI Fairness for People with Disabilities: Point of View](https://mlsanity.com/paper/1811.10670/) (2018)

## Built on by (most cited first, 24 of 24)

- [BBQ: A Hand-Built Bias Benchmark for Question Answering](https://mlsanity.com/paper/2110.08193/) (2021)
- [Language Model Tokenizers Introduce Unfairness Between Languages](https://mlsanity.com/paper/2305.15425/) (2023)
- [Language Model Behavior: A Comprehensive Survey](https://mlsanity.com/paper/2303.11504/) (2023)
- [Datasheet for the Pile](https://mlsanity.com/paper/2201.07311/) (2022)
- [AD-AutoGPT: An Autonomous GPT for Alzheimer's Disease Infodemiology](https://mlsanity.com/paper/2306.10095/) (2023)
- [The Silicon Ceiling: Auditing GPT's Race and Gender Biases in Hiring](https://mlsanity.com/paper/2405.04412/) (2024)
- [Global Voices, Local Biases: Socio-Cultural Prejudices across Languages](https://mlsanity.com/paper/2310.17586/) (2023)
- [Evaluating Bias in LLMs for Job-Resume Matching: Gender, Race, and Education](https://mlsanity.com/paper/2503.19182/) (2025)
- [Towards Auditing Large Language Models: Improving Text-based Stereotype Detection](https://mlsanity.com/paper/2311.14126/) (2023)
- [Automating Thematic Analysis: How LLMs Analyse Controversial Topics](https://mlsanity.com/paper/2405.06919/) (2024)
- [Evaluating and Mitigating Social Bias for Large Language Models in Open-ended Settings](https://mlsanity.com/paper/2412.06134/) (2024)
- [CatRAG: Functor-Guided Structural Debiasing with Retrieval Augmentation for Fair LLMs](https://mlsanity.com/paper/2603.21524/) (2026)
- [Towards an Enhanced Understanding of Bias in Pre-trained Neural Language Models: A Survey with Special Emphasis on Affective Bias](https://mlsanity.com/paper/2204.10365/) (2022)
- [Exploring Social Bias in Downstream Applications of Text-to-Image Foundation Models](https://mlsanity.com/paper/2312.10065/) (2023)
- [Structured Like a Language Model: Analysing AI as an Automated Subject](https://mlsanity.com/paper/2212.05058/) (2022)
- [FairPy: A Toolkit for Evaluation of Prediction Biases and their Mitigation in Large Language Models](https://mlsanity.com/paper/2302.05508/) (2023)
- [Protected group bias and stereotypes in Large Language Models](https://mlsanity.com/paper/2403.14727/) (2024)
- [Stereotype Detection in LLMs: A Multiclass, Explainable, and Benchmark-Driven Approach](https://mlsanity.com/paper/2404.01768/) (2024)
- [Understanding Gender Bias in AI-Generated Product Descriptions](https://mlsanity.com/paper/2506.05390/) (2025)
- [Cognitive network science reveals bias in GPT-3, ChatGPT, and GPT-4 mirroring math anxiety in high-school students](https://mlsanity.com/paper/2305.18320/) (2023)
- [Selecting Shots for Demographic Fairness in Few-Shot Learning with Large Language Models](https://mlsanity.com/paper/2311.08472/) (2023)
- [From Bytes to Biases: Investigating the Cultural Self-Perception of Large Language Models](https://mlsanity.com/paper/2312.17256/) (2023)
- ["Accessibility people, you go work on that thing of yours over there": Addressing Disability Inclusion in AI Product Organizations](https://mlsanity.com/paper/2508.16607/) (2025)
- [Identifying Implicit Bias in LLM-based Chat AI Toward People with Intellectual Disabilities](https://mlsanity.com/paper/2607.26062/) (2026)

## Abstract (arXiv)

To examine whether intersectional bias can be observed in language generation, we examine GPT-2 and GPT-NEO models, ranging in size from 124 million to ~2.7 billion parameters. We conduct an experiment combining up to three social categories - gender, religion and disability - into unconditional or zero-shot prompts used to generate sentences that are then analysed for sentiment. Our results confirm earlier tests conducted with auto-regressive causal models, including the GPT family of models. We also illustrate why bias may be resistant to techniques that target single categories (e.g. gender, religion and race), as it can also manifest, in often subtle ways, in texts prompted by concatenated social categories. To address these difficulties, we suggest technical and community-based approaches need to combine to acknowledge and address complex and intersectional language model bias.