mlsanityData to 7 Oct 2026

Data Feedback Loops: Model-driven Amplification of Dataset Biases

arXiv 2209.03942 ↗First posted 8 Sep 2022cs.LG cs.AI cs.CLcited by 59

Not on the map. Why?

Abstract (arXiv)

Datasets scraped from the internet have been critical to the successes of large-scale machine learning. Yet, this very success puts the utility of future internet-derived datasets at potential risk, as model outputs begin to replace human annotations as a source of supervision. In this work, we first formalize a system where interactions with one model are recorded as history and scraped as training data in the future. We then analyze its stability over time by tracking changes to a test-time bias statistic (e.g. gender bias of model predictions). We find that the degree of bias amplification is closely linked to whether the model's outputs behave like samples from the training distribution, a behavior which we characterize and define as consistent calibration. Experiments in three conditional prediction scenarios - image classification, visual role-labeling, and language generation - demonstrate that models that exhibit a sampling-like behavior are more calibrated and thus more stable. Based on this insight, we propose an intervention to help calibrate and stabilize unstable feedback systems. Code is available at https://github.com/rtaori/data_feedback.

Key facts. Cited by 59 papers in the map (data through 7 Oct 2026). Builds on 50 of its 88 references that are in the map. First posted on arXiv 8 Sep 2022.

Builds on

50 of 88 references.

References · 50

  1. 2015Deep Residual Learning for Image Recognitioncited by 51,809
  2. 2014Adam: A Method for Stochastic Optimizationcited by 36,960
  3. 2021Learning Transferable Visual Models From Natural Language Supervisioncited by 33,456
  4. 2020Language Models are Few-Shot Learnerscited by 32,199
  5. 2020Denoising Diffusion Probabilistic Modelscited by 20,851
  6. 2019PyTorch: An Imperative Style, High-Performance Deep Learning Librarycited by 20,160
  7. 2022Photorealistic Text-to-Image Diffusion Models with Deep Language Understandingcited by 6,322
  8. 2019The Curious Case of Neural Text Degenerationcited by 2,881
  9. 2020The Pile: An 800GB Dataset of Diverse Text for Language Modelingcited by 2,346
  10. 2020Extracting Training Data from Large Language Modelscited by 2,144
  11. 2020RealToxicityPrompts: Evaluating Neural Toxic Degeneration in Language Modelscited by 1,235
  12. 2018Hierarchical Neural Story Generationcited by 1,182
38 more
  1. 2019The Woman Worked as a Babysitter: On Biases in Language Generationcited by 540
  2. 2021MAUVE: Measuring the Gap Between Neural Text and Human Text using Divergence Frontierscited by 466
  3. 2017Men Also Like Shopping: Reducing Gender Bias Amplification using Corpus-level Constraintscited by 420
  4. 2021BOLD: Dataset and Metrics for Measuring Biases in Open-Ended Language Generationcited by 410
  5. 2018Detecting and Correcting for Label Shift with Black Box Predictorscited by 364
  6. 2018Fairness Without Demographics in Repeated Loss Minimizationcited by 363
  7. 2021Explaining Neural Scaling Lawscited by 353
  8. 2020Performative Predictioncited by 318
  9. 2020A Brief Review of Domain Adaptationcited by 287
  10. 2021Annotators with Attitudes: How Annotator Beliefs And Identities Bias Toxic Language Detectioncited by 258
  11. 2015Strategic Classificationcited by 256
  12. 2019StegaStamp: Invisible Hyperlinks in Physical Photographscited by 256
  13. 2021Knowledge distillation: A good teacher is patient and consistentcited by 255
  14. 2019Billion-scale semi-supervised learning for image classificationcited by 232
  15. 2022A Contrastive Framework for Neural Text Generationcited by 224
  16. 2018A DIRT-T Approach to Unsupervised Domain Adaptationcited by 215
  17. 2019Queens are Powerful too: Mitigating Gender Bias in Dialogue Generationcited by 152
  18. 2020Understanding Self-Training for Gradual Domain Adaptationcited by 149
  19. 2021Are GAN generated images easy to detect? A critical analysis of the state-of-the-artcited by 139
  20. 2020Enabling Language Models to Fill in the Blankscited by 121
  21. 2020An Overview of Deep Semi-Supervised Learningcited by 121
  22. 2017How Algorithmic Confounding in Recommendation Systems Increases Homogeneity and Decreases Utilitycited by 116
  23. 2020Optimal Regularization Can Mitigate Double Descentcited by 107
  24. 2017Runaway Feedback Loops in Predictive Policingcited by 97
  25. 2021Outside the Echo Chamber: Optimizing the Performative Riskcited by 95
  26. 2020Performative Prediction in a Stateful Worldcited by 90
  27. 2021Anticipating Safety Issues in E2E Conversational AI: Framework and Toolingcited by 77
  28. 2021How to Learn when Data Reacts to Your Model: Performative Gradient Descentcited by 70
  29. 2019Degenerate Feedback Loops in Recommender Systemscited by 67
  30. 2022A Systematic Study of Bias Amplificationcited by 65
  31. 2021Designing Toxic Content Classification for a Diversity of Perspectivescited by 63
  32. 2019The Disparate Equilibria of Algorithmic Decision Making when Individuals Invest Rationallycited by 41
  33. 2020Distributional Generalization: A New Kind of Generalizationcited by 41
  34. 2018Feature-Wise Bias Amplificationcited by 31
  35. 2021The Rich Get Richer: Disparate Impact of Semi-Supervised Learningcited by 28
  36. 2021Scaling Laws for Deep Learningcited by 25
  37. 2017Deconvolving Feedback Loops in Recommender Systemscited by 17
  38. 2017Human Interaction with Recommendation Systemscited by 15

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2022: 1; 2023: 9; 2024: 19; 2025: 18; 2026: 12.

59 papers.

Sort:

By area:

  1. 2023Does Writing with Language Models Reduce Content Diversity?cited by 190
  2. 2024Is Model Collapse Inevitable? Breaking the Curse of Recursion by Accumulating Real and Synthetic Datacited by 143
  3. 2023Improving Multimodal Datasets with Image Captioningcited by 93
  4. 2023Leaving Reality to Imagination: Robust Classification via Generated Datasetscited by 91
  5. 2023Identifying and Mitigating the Security Risks of Generative AIcited by 90
  6. 2022Language Generation Models Can Cause Harm: So What Can We Do About It? An Actionable Surveycited by 80
  7. 2024Model Collapse Demystified: The Case of Regressioncited by 80
  8. 2024A Design Space for Intelligent and Interactive Writing Assistantscited by 76
  9. 2024Feedback Loops With Language Models Drive In-Context Reward Hackingcited by 70
  10. 2023Uncertainty in Natural Language Generation: From Theory to Applicationscited by 61
  11. 2023Discovering and Mitigating Visual Biases through Keyword Explanationcited by 58
  12. 2024Fairness Feedback Loops: Training on Synthetic Data Amplifies Biascited by 55
38 more
  1. 2023Performative Prediction: Past and Futurecited by 52
  2. 2023Prompting Multilingual Large Language Models to Generate Code-Mixed Texts: The Case of South East Asian Languagescited by 50
  3. 2024Collapse or Thrive? Perils and Promises of Synthetic Data in a Self-Generating Worldcited by 49
  4. 2025Position: Model Collapse Does Not Mean What You Thinkcited by 32
  5. 2024Would Deep Generative Models Amplify Bias in Future Models?cited by 24
  6. 2025Pre-training under infinite computecited by 24
  7. 2024Harm Amplification in Text-to-Image Modelscited by 14
  8. 2024Bias Amplification: Large Language Models as Increasingly Biased Mediacited by 12
  9. 2024Model Collapse in the Self-Consuming Chain of Diffusion Finetuning: A Novel Perspective from Quantitative Trait Modelingcited by 11
  10. 2025Revisiting the Predictability of Performative, Social Eventscited by 11
  11. 2025The Lock-in Hypothesis: Stagnation by Algorithmcited by 11
  12. 2024Automating Data Annotation under Strategic Human Agents: Risks and Potential Solutionscited by 9
  13. 2025Self-Consuming Generative Models with Adversarially Curated Datacited by 8
  14. 2025Why Less is More (Sometimes): A Theory of Data Curationcited by 8
  15. 2025From Five Dimensions to Many: Large Language Models as Precise and Interpretable Psychological Profilerscited by 5
  16. 2024Addressing Discretization-Induced Bias in Demographic Predictioncited by 4
  17. 2024Synthetic Context Generation for Question Generationcited by 4
  18. 2024The Limitations of Model Retraining in the Face of Performativitycited by 4
  19. 2024Theoretical Proof that Auto-regressive Language Models Collapse when Real-world Data is a Finite Setcited by 4
  20. 2025When Incentives Backfire, Data Stops Being Humancited by 4
  21. 2025Statistical Inference under Performativitycited by 4
  22. 2026The Stability of Online Algorithms in Performative Predictioncited by 4
  23. 2024Performative Debias with Fair-exposure Optimization Driven by Strategic Agents in Recommender Systemscited by 3
  24. 2024Identifying Predictions That Influence the Future: Detecting Performative Concept Drift in Data Streamscited by 3
  25. 2025FairSense: Long-Term Fairness Analysis of ML-Enabled Systemscited by 3
  26. 2025Stabilizing Self-Consuming Diffusion Models with Latent Space Filteringcited by 3
  27. 2026Tracing the Data Trail: A Survey of Data Provenance, Transparency and Traceability in LLMscited by 3
  28. 2023Predictive Minds: LLMs As Atypical Active Inference Agentscited by 2
  29. 2025The Strong, Weak and Benign Goodhart's law. An independence-free and paradigm-agnostic formalisationcited by 2
  30. 2024A Mathematical Model of the Hidden Feedback Loop Effect in Machine Learning Systemscited by 1
  31. 2025Convergence Dynamics and Stabilization Strategies of Co-Evolving Generative Modelscited by 1
  32. 2025Towards Reliable Time Series Forecasting under Future Uncertainty: Ambiguity and Novelty Rejection Mechanismscited by 1
  33. 2025Optimal Regularization for Performative Learningcited by 1
  34. 2026Retraining Seeks Stable Signalscited by 1
  35. 2024Adjusting Pretrained Backbones for Performativitycited by 0
  36. 2025A Critical Review of Predominant Bias in Neural Networkscited by 0
  37. 2025Active Data Sampling and Generation for Bias Remediationcited by 0
  38. 2025From Data-Driven to Purpose-Driven Artificial Intelligence: Systems Thinking for Data-Analytic Automation of Patient Carecited by 0

50 most cited of 59 All 59 (JSON)