# The Fairness Collapse Phenomenon: Bias Amplification in Language Models Trained on Synthetic Data

arXiv: https://arxiv.org/abs/2608.04268 · HTML: https://mlsanity.com/paper/2608.04268/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in Model collapse from synthetic data, one of 13 directions in Synthetic data and self-improvement for LLMs.
- Builds on 26 of its 31 references that are in the map.
- First posted on arXiv 4 Aug 2026.

## Placed in

- Synthetic data and self-improvement for LLMs / [Model collapse from synthetic data](https://mlsanity.com/synth/model-collapse-from-synthetic-data/)

## Builds on (most cited first, 26 of 26)

- [Measuring Massive Multitask Language Understanding](https://mlsanity.com/paper/2009.03300/) (2020)
- [Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddings](https://mlsanity.com/paper/1607.06520/) (2016)
- [StereoSet: Measuring stereotypical bias in pretrained language models](https://mlsanity.com/paper/2004.09456/) (2020)
- [CrowS-Pairs: A Challenge Dataset for Measuring Social Biases in Masked Language Models](https://mlsanity.com/paper/2010.00133/) (2020)
- [Gender Bias in Coreference Resolution: Evaluation and Debiasing Methods](https://mlsanity.com/paper/1804.06876/) (2018)
- [The Woman Worked as a Babysitter: On Biases in Language Generation](https://mlsanity.com/paper/1909.01326/) (2019)
- [Null It Out: Guarding Protected Attributes by Iterative Nullspace Projection](https://mlsanity.com/paper/2004.07667/) (2020)
- [The Curse of Recursion: Training on Generated Data Makes Models Forget](https://mlsanity.com/paper/2305.17493/) (2023)
- [Bias in Bios: A Case Study of Semantic Representation Bias in a High-Stakes Setting](https://mlsanity.com/paper/1901.09451/) (2019)
- [Self-Diagnosis and Self-Debiasing: A Proposal for Reducing Corpus-Based Bias in NLP](https://mlsanity.com/paper/2103.00453/) (2021)
- [Self-Consuming Generative Models Go MAD](https://mlsanity.com/paper/2307.01850/) (2023)
- [Towards Debiasing Sentence Representations](https://mlsanity.com/paper/2007.08100/) (2020)
- [A Tale of Tails: Model Collapse as a Change of Scaling Laws](https://mlsanity.com/paper/2402.07043/) (2024)
- [MMTEB: Massive Multilingual Text Embedding Benchmark](https://mlsanity.com/paper/2502.13595/) (2025)
- [The Curious Decline of Linguistic Diversity: Training Language Models on Synthetic Text](https://mlsanity.com/paper/2311.09807/) (2023)
- [How Bad is Training on Synthetic Data? A Statistical Analysis of Language Model Collapse](https://mlsanity.com/paper/2404.05090/) (2024)
- [Collapse or Thrive? Perils and Promises of Synthetic Data in a Self-Generating World](https://mlsanity.com/paper/2410.16713/) (2024)
- [What's in a Name? Reducing Bias in Bios without Access to Protected Attributes](https://mlsanity.com/paper/1904.05233/) (2019)
- [Gender, Race, and Intersectional Bias in Resume Screening via Language Model Retrieval](https://mlsanity.com/paper/2407.20371/) (2024)
- [FairSteer: Inference Time Debiasing for LLMs with Dynamic Activation Steering](https://mlsanity.com/paper/2504.14492/) (2025)
- [Self-Consuming Generative Models with Curated Data Provably Optimize Human Preferences](https://mlsanity.com/paper/2407.09499/) (2024)
- [Social Bias Probing: Fairness Benchmarking for Language Models](https://mlsanity.com/paper/2311.09090/) (2023)
- [The Silicon Ceiling: Auditing GPT's Race and Gender Biases in Hiring](https://mlsanity.com/paper/2405.04412/) (2024)
- [How to Synthesize Text Data without Model Collapse?](https://mlsanity.com/paper/2412.14689/) (2024)
- [JobFair: A Framework for Benchmarking Gender Hiring Bias in Large Language Models](https://mlsanity.com/paper/2406.15484/) (2024)
- [The Impact of AI-Generated Text on the Internet](https://mlsanity.com/paper/2604.26965/) (2026)

## Abstract (arXiv)

Generative models trained on artificially generated data have been shown to exhibit model collapse, resulting in significant performance degradation. As synthetic content increasingly contaminates the training corpora of language models, this raises critical concerns about the use of open data in continued pretraining. Although previous work has demonstrated model collapse in language models, it remains unclear whether exposure to synthetic data amplifies or attenuates the social biases already present in pretrained models. Because language models are known to reproduce and amplify demographic stereotypes, recursive training on self-generated data may create a self-reinforcing feedback loop in which biased associations become progressively stronger across generations. We call this hypothesized phenomenon fairness collapse. In this work, we construct controlled training regimes in which models are repeatedly trained on synthetic data using the Bias in Bios dataset. Across experiments, we observe a consistent and concerning pattern: fairness degradation emerges before substantial degradation is reflected by standard language-modeling metrics. This result highlights a critical risk associated with synthetic data contamination in language model training: bias can increase silently before strong indicators of model collapse become apparent.