# A Mathematical Model of the Hidden Feedback Loop Effect in Machine Learning Systems

arXiv: https://arxiv.org/abs/2405.02726 · HTML: https://mlsanity.com/paper/2405.02726/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Builds on 19 of its 58 references that are in the map.
- First posted on arXiv 4 May 2024.

## Builds on (most cited first, 19 of 19)

- [Trustworthy AI: From Principles to Practices](https://mlsanity.com/paper/2110.01167/) (2021)
- [Runaway Feedback Loops in Predictive Policing](https://mlsanity.com/paper/1706.09847/) (2017)
- [Unified Optimal Analysis of the (Stochastic) Gradient Method](https://mlsanity.com/paper/1907.04232/) (2019)
- [The relationship between trust in AI and trustworthy machine learning technologies](https://mlsanity.com/paper/1912.00782/) (2019)
- [Degenerate Feedback Loops in Recommender Systems](https://mlsanity.com/paper/1902.10730/) (2019)
- [Data Feedback Loops: Model-driven Amplification of Dataset Biases](https://mlsanity.com/paper/2209.03942/) (2022)
- [Socially Responsible AI Algorithms: Issues, Purposes, and Challenges](https://mlsanity.com/paper/2101.02032/) (2021)
- [High Probability Convergence of Stochastic Gradient Methods](https://mlsanity.com/paper/2302.14843/) (2023)
- [Software Engineering Challenges of Deep Learning](https://mlsanity.com/paper/1810.12034/) (2018)
- [Misplaced Trust: Measuring the Interference of Machine Learning in Human Decision-Making](https://mlsanity.com/paper/2005.10960/) (2020)
- [Deconvolving Feedback Loops in Recommender Systems](https://mlsanity.com/paper/1703.01049/) (2017)
- [Towards Guidelines for Assessing Qualities of Machine Learning Systems](https://mlsanity.com/paper/2008.11007/) (2020)
- [Convergence of sequences: a survey](https://mlsanity.com/paper/2111.11374/) (2021)
- [Practices for Engineering Trustworthy Machine Learning Applications](https://mlsanity.com/paper/2103.00964/) (2021)
- [The Challenges and Opportunities of Human-Centered AI for Trustworthy Robots and Autonomous Systems](https://mlsanity.com/paper/2105.04408/) (2021)
- [Requirements Engineering for Machine Learning: A Review and Reflection](https://mlsanity.com/paper/2210.00859/) (2022)
- [Iterated Function Systems: A Comprehensive Survey](https://mlsanity.com/paper/2211.14661/) (2022)
- [MLDev: Data Science Experiment Automation and Reproducibility Software](https://mlsanity.com/paper/2107.12322/) (2021)
- [Existence conditions for hidden feedback loops in online recommender systems](https://mlsanity.com/paper/2109.05278/) (2021)

## Built on by (most cited first, 1 of 1)

- [Collapse or Thrive? Perils and Promises of Synthetic Data in a Self-Generating World](https://mlsanity.com/paper/2410.16713/) (2024)

## Abstract (arXiv)

Widespread deployment of societal-scale machine learning systems necessitates a thorough understanding of the resulting long-term effects these systems have on their environment, including loss of trustworthiness, bias amplification, and violation of AI safety requirements. We introduce a repeated learning process to jointly describe several phenomena attributed to unintended hidden feedback loops, such as error amplification, induced concept drift, echo chambers and others. The process comprises the entire cycle of obtaining the data, training the predictive model, and delivering predictions to end-users within a single mathematical model. A distinctive feature of such repeated learning setting is that the state of the environment becomes causally dependent on the learner itself over time, thus violating the usual assumptions about the data distribution. We present a novel dynamical systems model of the repeated learning process and prove the limiting set of probability distributions for positive and negative feedback loop modes of the system operation. We conduct a series of computational experiments using an exemplary supervised learning problem on two synthetic data sets. The results of the experiments correspond to the theoretical predictions derived from the dynamical model. Our results demonstrate the feasibility of the proposed approach for studying the repeated learning processes in machine learning systems and open a range of opportunities for further research in the area.