# A Systematic Study of Bias Amplification

arXiv: https://arxiv.org/abs/2201.11706 · HTML: https://mlsanity.com/paper/2201.11706/

## Key facts

- Cited by 65 papers in the map (data through 14 Oct 2026).
- Builds on 23 of its 46 references that are in the map.
- First posted on arXiv 27 Jan 2022.

## Builds on (most cited first, 23 of 23)

- [Deep Residual Learning for Image Recognition](https://mlsanity.com/paper/1512.03385/) (2015)
- [Learning Transferable Visual Models From Natural Language Supervision](https://mlsanity.com/paper/2103.00020/) (2021)
- [On Calibration of Modern Neural Networks](https://mlsanity.com/paper/1706.04599/) (2017)
- [Fashion-MNIST: a Novel Image Dataset for Benchmarking Machine Learning Algorithms](https://mlsanity.com/paper/1708.07747/) (2017)
- [Equality of Opportunity in Supervised Learning](https://mlsanity.com/paper/1610.02413/) (2016)
- [A Survey on Bias and Fairness in Machine Learning](https://mlsanity.com/paper/1908.09635/) (2019)
- [Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour](https://mlsanity.com/paper/1706.02677/) (2017)
- [Counterfactual Fairness](https://mlsanity.com/paper/1703.06856/) (2017)
- [Fair prediction with disparate impact: A study of bias in recidivism prediction instruments](https://mlsanity.com/paper/1610.07524/) (2016)
- [Men Also Like Shopping: Reducing Gender Bias Amplification using Corpus-level Constraints](https://mlsanity.com/paper/1707.09457/) (2017)
- [Fairness in Criminal Justice Risk Assessments: The State of the Art](https://mlsanity.com/paper/1703.09207/) (2017)
- [Women also Snowboard: Overcoming Bias in Captioning Models](https://mlsanity.com/paper/1803.09797/) (2018)
- [Queens are Powerful too: Mitigating Gender Bias in Dialogue Generation](https://mlsanity.com/paper/1911.03842/) (2019)
- [Does Object Recognition Work for Everyone?](https://mlsanity.com/paper/1906.02659/) (2019)
- [An Intersectional Definition of Fairness](https://mlsanity.com/paper/1807.08362/) (2018)
- [Fair Generative Modeling via Weak Supervision](https://mlsanity.com/paper/1910.12008/) (2019)
- [Understanding and Evaluating Racial Biases in Image Captioning](https://mlsanity.com/paper/2106.08503/) (2021)
- [Directional Bias Amplification](https://mlsanity.com/paper/2102.12594/) (2021)
- [Feature-Wise Bias Amplification](https://mlsanity.com/paper/1812.08999/) (2018)
- [Imperfect ImaGANation: Implications of GANs Exacerbating Biases on Facial Data Augmentation and Snapchat Selfie Lenses](https://mlsanity.com/paper/2001.09528/) (2020)
- [Mitigating Gender Bias Amplification in Distribution by Posterior Regularization](https://mlsanity.com/paper/2005.06251/) (2020)
- [Fairness On The Ground: Applying Algorithmic Fairness Approaches to Production Systems](https://mlsanity.com/paper/2103.06172/) (2021)
- [ConvNets and ImageNet Beyond Accuracy: Understanding Mistakes and Uncovering Biases](https://mlsanity.com/paper/1711.11443/) (2017)

## Built on by (most cited first, 30 of 65)

- [Pythia: A Suite for Analyzing Large Language Models Across Training and Scaling](https://mlsanity.com/paper/2304.01373/) (2023)
- [The Platonic Representation Hypothesis](https://mlsanity.com/paper/2405.07987/) (2024)
- [NLPositionality: Characterizing Design Biases of Datasets and Models](https://mlsanity.com/paper/2306.01943/) (2023)
- [Language Generation Models Can Cause Harm: So What Can We Do About It? An Actionable Survey](https://mlsanity.com/paper/2210.07700/) (2022)
- [The Bias Amplification Paradox in Text-to-Image Generation](https://mlsanity.com/paper/2308.00755/) (2023)
- [Data Feedback Loops: Model-driven Amplification of Dataset Biases](https://mlsanity.com/paper/2209.03942/) (2022)
- [The Homogenizing Effect of Large Language Models on Human Expression and Thought](https://mlsanity.com/paper/2508.01491/) (2025)
- [Vision-Language Models Performing Zero-Shot Tasks Exhibit Gender-based Disparities](https://mlsanity.com/paper/2301.11100/) (2023)
- [Training Unbiased Diffusion Models From Biased Dataset](https://mlsanity.com/paper/2403.01189/) (2024)
- [FairTune: Optimizing Parameter Efficient Fine Tuning for Fairness in Medical Image Analysis](https://mlsanity.com/paper/2310.05055/) (2023)
- [Men Also Do Laundry: Multi-Attribute Bias Amplification](https://mlsanity.com/paper/2210.11924/) (2022)
- [NASDM: Nuclei-Aware Semantic Histopathology Image Generation Using Diffusion Models](https://mlsanity.com/paper/2303.11477/) (2023)
- [Model-Agnostic Gender Debiased Image Captioning](https://mlsanity.com/paper/2304.03693/) (2023)
- [DataFinder: Scientific Dataset Recommendation from Natural Language Descriptions](https://mlsanity.com/paper/2305.16636/) (2023)
- [Harm Amplification in Text-to-Image Models](https://mlsanity.com/paper/2402.01787/) (2024)
- [On Catastrophic Inheritance of Large Foundation Models](https://mlsanity.com/paper/2402.01909/) (2024)
- [NLP for Maternal Healthcare: Perspectives and Guiding Principles in the Age of LLMs](https://mlsanity.com/paper/2312.11803/) (2023)
- [Auditing and Generating Synthetic Data with Controllable Trust Trade-offs](https://mlsanity.com/paper/2304.10819/) (2023)
- [Getting Sick After Seeing a Doctor? Diagnosing and Mitigating Knowledge Conflicts in Event Temporal Reasoning](https://mlsanity.com/paper/2305.14970/) (2023)
- [What Makes CLIP More Robust to Long-Tailed Pre-Training Data? A Controlled Study for Transferable Insights](https://mlsanity.com/paper/2405.21070/) (2024)
- [U-Fair: Uncertainty-based Multimodal Multitask Learning for Fairer Depression Detection](https://mlsanity.com/paper/2501.09687/) (2025)
- [Simplicity Bias Leads to Amplified Performance Disparities](https://mlsanity.com/paper/2212.06641/) (2022)
- [The Lock-in Hypothesis: Stagnation by Algorithm](https://mlsanity.com/paper/2506.06166/) (2025)
- [Automating Data Annotation under Strategic Human Agents: Risks and Potential Solutions](https://mlsanity.com/paper/2405.08027/) (2024)
- [Slight Corruption in Pre-training Data Makes Better Diffusion Models](https://mlsanity.com/paper/2405.20494/) (2024)
- [Quantifying and mitigating the impact of label errors on model disparity metrics](https://mlsanity.com/paper/2310.02533/) (2023)
- [Evaluating Bias and Fairness in Gender-Neutral Pretrained Vision-and-Language Models](https://mlsanity.com/paper/2310.17530/) (2023)
- [Intrinsic Bias is Predicted by Pretraining Data and Correlates with Downstream Performance in Vision-Language Encoders](https://mlsanity.com/paper/2502.07957/) (2025)
- [Who's Asking? Investigating Bias Through the Lens of Disability Framed Queries in LLMs](https://mlsanity.com/paper/2508.15831/) (2025)
- [Towards Reliable Assessments of Demographic Disparities in Multi-Label Image Classifiers](https://mlsanity.com/paper/2302.08572/) (2023)

## Abstract (arXiv)

Recent research suggests that predictions made by machine-learning models can amplify biases present in the training data. When a model amplifies bias, it makes certain predictions at a higher rate for some groups than expected based on training-data statistics. Mitigating such bias amplification requires a deep understanding of the mechanics in modern machine learning that give rise to that amplification. We perform the first systematic, controlled study into when and how bias amplification occurs. To enable this study, we design a simple image-classification problem in which we can tightly control (synthetic) biases. Our study of this problem reveals that the strength of bias amplification is correlated to measures such as model accuracy, model capacity, model overconfidence, and amount of training data. We also find that bias amplification can vary greatly during training. Finally, we find that bias amplification may depend on the difficulty of the classification task relative to the difficulty of recognizing group membership: bias amplification appears to occur primarily when it is easier to recognize group membership than class membership. Our results suggest best practices for training machine-learning models that we hope will help pave the way for the development of better mitigation strategies. Code can be found at https://github.com/facebookresearch/cv_bias_amplification.