mlsanityData to 14 Oct 2026

Fairness Feedback Loops: Training on Synthetic Data Amplifies Bias

arXiv 2403.07857 ↗First posted 12 Mar 2024cs.LGcited by 55

Not on the map. Why?

Abstract (arXiv)

Model-induced distribution shifts (MIDS) occur as previous model outputs pollute new model training sets over generations of models. This is known as model collapse in the case of generative models, and performative prediction or unfairness feedback loops for supervised models. When a model induces a distribution shift, it also encodes its mistakes, biases, and unfairnesses into the ground truth of its data ecosystem. We introduce a framework that allows us to track multiple MIDS over many generations, finding that they can lead to loss in performance, fairness, and minoritized group representation, even in initially unbiased datasets. Despite these negative consequences, we identify how models might be used for positive, intentional, interventions in their data ecosystems, providing redress for historical discrimination through a framework called algorithmic reparation (AR). We simulate AR interventions by curating representative training batches for stochastic gradient descent to demonstrate how AR can improve upon the unfairnesses of models and data ecosystems subject to other MIDS. Our work takes an important step towards identifying, mitigating, and taking accountability for the unfair feedback loops enabled by the idea that ML systems are inherently neutral and objective.

Key facts. Cited by 55 papers in the map (data through 14 Oct 2026). Builds on 28 of its 75 references that are in the map. First posted on arXiv 12 Mar 2024.

Builds on

28 of 75 references.

References · 28

  1. 2015Distilling the Knowledge in a Neural Networkcited by 10,324
  2. 2014Deep Learning Face Attributes in the Wildcited by 3,693
  3. 2016Equality of Opportunity in Supervised Learningcited by 2,007
  4. 2019Invariant Risk Minimizationcited by 1,854
  5. 2016Fair prediction with disparate impact: A study of bias in recidivism prediction instrumentscited by 672
  6. 2017Algorithmic decision making and the cost of fairnesscited by 403
  7. 2018Fairness Without Demographics in Repeated Loss Minimizationcited by 364
  8. 2023The Curse of Recursion: Training on Generated Data Makes Models Forgetcited by 356
  9. 2016Semi-supervised Knowledge Transfer for Deep Learning from Private Training Datacited by 348
  10. 2020Performative Predictioncited by 327
  11. 2023Self-Consuming Generative Models Go MADcited by 250
  12. 2019A Framework for Understanding Sources of Harm throughout the Machine Learning Life Cyclecited by 193
16 more
  1. 2019Lessons from Archives: Strategies for Collecting Sociocultural Data in Machine Learningcited by 116
  2. 2019Fair Generative Modeling via Weak Supervisioncited by 114
  3. 2020Synthetic Data -- Anonymisation Groundhog Daycited by 108
  4. 2023Artificial Artificial Artificial Intelligence: Crowd Workers Widely Use Large Language Models for Text Production Taskscited by 103
  5. 2017Runaway Feedback Loops in Predictive Policingcited by 97
  6. 2019Fairwashing: the risk of rationalizationcited by 81
  7. 2019Bias Correction of Learned Generative Models using Likelihood-Free Importance Weightingcited by 70
  8. 2022Data Feedback Loops: Model-driven Amplification of Dataset Biasescited by 60
  9. 2022Will Large-scale Generative Models Corrupt Future Datasets?cited by 59
  10. 2023Towards Understanding the Interplay of Generative Artificial Intelligence and the Internetcited by 54
  11. 2021Robin Hood and Matthew Effects: Differential Privacy Has Disparate Impact on Synthetic Datacited by 48
  12. 2019Fair Classification and Social Welfarecited by 40
  13. 2020Curriculum Labeling: Revisiting Pseudo-Labeling for Semi-Supervised Learningcited by 14
  14. 2021On the Fairness of Generative Adversarial Networks (GANs)cited by 14
  15. 2021Impact of classification difficulty on the weight matrices spectra in Deep Learning and application to early-stoppingcited by 12
  16. 2022Simplicity Bias Leads to Amplified Performance Disparitiescited by 11

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2024: 9; 2025: 27; 2026: 19.

55 papers.

Sort:

By area:

  1. 2024RewardBench: Evaluating Reward Models for Language Modelingcited by 499
  2. 2024RL on Incorrect Synthetic Data Scales the Efficiency of LLM Math Reasoning by Eight-Foldcited by 99
  3. 2024Self-Consuming Generative Models with Curated Data Provably Optimize Human Preferencescited by 39
  4. 2024Self-Improving Diffusion Models with Synthetic Datacited by 35
  5. 2025The Homogenizing Effect of Large Language Models on Human Expression and Thoughtcited by 35
  6. 2025Position: Model Collapse Does Not Mean What You Thinkcited by 33
  7. 2024WildFeedback: Aligning LLMs With In-situ User Interactions And Feedbackcited by 31
  8. 2025A Closer Look at Model Collapse: From a Generalization-to-Memorization Perspectivecited by 18
  9. 2025A Theoretical Perspective: How to Prevent Model Collapse in Self-consuming Training Loopscited by 16
  10. 2024Bias Amplification: Large Language Models as Increasingly Biased Mediacited by 12
  11. 2025Self-Consuming Generative Models with Adversarially Curated Datacited by 10
  12. 2025Modeling Annotator Disagreement with Demographic-Aware Experts and Synthetic Perspectivescited by 10
38 more
  1. 2025Value Drifts: Tracing Value Alignment During LLM Post-Trainingcited by 10
  2. 2024Automating Data Annotation under Strategic Human Agents: Risks and Potential Solutionscited by 9
  3. 2025Why Less is More (Sometimes): A Theory of Data Curationcited by 8
  4. 2025High-dimensional Analysis of Synthetic Data Selectioncited by 7
  5. 2025More of the Same: Persistent Representational Harms Under Increased Representationcited by 6
  6. 2025LLM Web Dynamics: Tracing Model Collapse in a Network of LLMscited by 6
  7. 2025BitMark: Watermarking Bitwise Autoregressive Image Generative Modelscited by 5
  8. 2025Synthetic Artifact Auditing: Tracing LLM-Generated Synthetic Data Usage in Downstream Applicationscited by 4
  9. 2025BiMarker: Enhancing Text Watermark Detection for Large Language Models with Bipolar Watermarkscited by 3
  10. 2025Synthetic Data Generation for Augmenting Small Samplescited by 3
  11. 2025Generative Artificial Intelligence for Internet of Things Computing: A Systematic Surveycited by 3
  12. 2025Synthetic Voice Data for Automatic Speech Recognition in African Languagescited by 3
  13. 2025Limits to AI Growth: The Ecological and Social Consequences of Scalingcited by 2
  14. 2025Convergence Dynamics and Stabilization Strategies of Co-Evolving Generative Modelscited by 2
  15. 2026Observations and Remedies for Large Language Model Bias in Self-Consuming Performative Loopcited by 2
  16. 2026When and How Human Curation Backfires: Preference Alignment under Multi-Model Self-Consuming Loopcited by 2
  17. 2025The erasure of intensive livestock farming in text-to-image generative AIcited by 1
  18. 2025Quantitative Auditing of AI Fairness with Differentially Private Synthetic Datacited by 1
  19. 2025Knowledge Collapse in LLMs: When Fluency Survives but Facts Fail under Recursive Synthetic Trainingcited by 1
  20. 2025Is Lying Only Sinful in Islam? Exploring Religious Bias in Multilingual Large Language Models Across Major Religionscited by 1
  21. 2026Position: the Stochastic Parrot in the Coal Mine. Model Collapse is a Threat to Low-Resource Communitiescited by 1
  22. 2026When Sample Selection Bias Precipitates Model Collapsecited by 1
  23. 2026Data Provenance for Image Auto-Regressive Generationcited by 1
  24. 2024Adjusting Pretrained Backbones for Performativitycited by 0
  25. 2024The Evolution and Future Perspectives of Artificial Intelligence Generated Contentcited by 0
  26. 2025FairT2I: Mitigating Social Bias in Text-to-Image Generation via Large Language Model-Assisted Detection and Attribute Rebalancingcited by 0
  27. 2025A Critical Review of Predominant Bias in Neural Networkscited by 0
  28. 2025Watermarking Needs Input Repetition Maskingcited by 0
  29. 2025TEDI: Trustworthy and Ethical Dataset Indicators to Analyze and Compare Dataset Documentationcited by 0
  30. 2026When Predictions Shape Reality: A Socio-Technical Synthesis of Performative Predictions in Machine Learningcited by 0
  31. 2026Should I use Synthetic Data for That? An Analysis of the Suitability of Synthetic Data for Data Sharing and Augmentationcited by 0
  32. 2026Data Kernel Perspective Space Performance Guarantees for Synthetic Data from Transformer Modelscited by 0
  33. 2026Fair Lung Disease Diagnosis from Chest CT via Gender-Adversarial Attention Multiple Instance Learningcited by 0
  34. 2026SERUM: Simple, Efficient, Robust, and Unifying Marking for Diffusion-based Image Generationcited by 0
  35. 2026Jump Start or False Start? A Theoretical and Empirical Evaluation of LLM-initialized Banditscited by 0
  36. 2026How Supply Chain Dependencies Complicate Bias Measurement and Accountability Attribution in AI Hiring Applicationscited by 0
  37. 2026Silent Failures in Federated Personalization of Foundation Modelscited by 0
  38. 2026Disparate Impact in Synthetic Data Generationcited by 0

50 most cited of 55 All 55 (JSON)

Similar, not cited

Close in wording, no citation either way

  1. 2026The Fairness Collapse Phenomenon: Bias Amplification in Language Models Trained on Synthetic Data
  2. 2025AIM-Fair: Advancing Algorithmic Fairness via Selectively Fine-Tuning Biased Models with Contextual Synthetic Data
  3. 2025Recursive Training Loops in LLMs: How training data properties modulate distribution shift in generated data?
  4. 2026Stay Fair! Ensuring Group Fairness in Diffusion Models Across Guidance Scales
  5. 2026Bias Redistribution in Visual Machine Unlearning: Does Forgetting One Group Harm Another?
  6. 2024Towards Harmless Rawlsian Fairness Regardless of Demographic Prior
  7. 2025Debiasing Diffusion Model: Enhancing Fairness through Latent Representation Learning in Stable Diffusion Model
  8. 2025On Optimal Steering to Achieve Exact Fairness