# Accounting for AI and Users Shaping One Another: The Role of Mathematical Models

arXiv: https://arxiv.org/abs/2404.12366 · HTML: https://mlsanity.com/paper/2404.12366/

## Key facts

- Cited by 11 papers in the map (data through 14 Oct 2026).
- Builds on 60 of its 94 references that are in the map.
- First posted on arXiv 18 Apr 2024.

## Builds on (most cited first, 30 of 60)

- [Fair prediction with disparate impact: A study of bias in recidivism prediction instruments](https://mlsanity.com/paper/1610.07524/) (2016)
- [Inherent Trade-Offs in the Fair Determination of Risk Scores](https://mlsanity.com/paper/1609.05807/) (2016)
- [Performative Prediction](https://mlsanity.com/paper/2002.06673/) (2020)
- [Measurement and Fairness](https://mlsanity.com/paper/1912.05511/) (2019)
- [Strategic Classification](https://mlsanity.com/paper/1506.06980/) (2015)
- [Delayed Impact of Fair Machine Learning](https://mlsanity.com/paper/1803.04383/) (2018)
- [How Algorithmic Confounding in Recommendation Systems Increases Homogeneity and Decreases Utility](https://mlsanity.com/paper/1710.11214/) (2017)
- [RecSim: A Configurable Simulation Platform for Recommender Systems](https://mlsanity.com/paper/1909.04847/) (2019)
- [Feedback Loop and Bias Amplification in Recommender Systems](https://mlsanity.com/paper/2007.13019/) (2020)
- [How Do Classifiers Induce Agents To Invest Effort Strategically?](https://mlsanity.com/paper/1807.05307/) (2018)
- [Feedback Loops With Language Models Drive In-Context Reward Hacking](https://mlsanity.com/paper/2402.06627/) (2024)
- [Degenerate Feedback Loops in Recommender Systems](https://mlsanity.com/paper/1902.10730/) (2019)
- [RecoGym: A Reinforcement Learning Environment for the problem of Product Recommendation in Online Advertising](https://mlsanity.com/paper/1808.00720/) (2018)
- [Rotting Bandits](https://mlsanity.com/paper/1702.07274/) (2017)
- [Bayesian Incentive-Compatible Bandit Exploration](https://mlsanity.com/paper/1502.04147/) (2015)
- [Roles for Computing in Social Change](https://mlsanity.com/paper/1912.04883/) (2019)
- [Alternative Microfoundations for Strategic Classification](https://mlsanity.com/paper/2106.12705/) (2021)
- [Hidden Incentives for Auto-Induced Distributional Shift](https://mlsanity.com/paper/2009.09153/) (2020)
- [CIRS: Bursting Filter Bubbles by Counterfactual Interactive Recommender System](https://mlsanity.com/paper/2204.01266/) (2022)
- [Modeling Content Creator Incentives on Algorithm-Curated Platforms](https://mlsanity.com/paper/2206.13102/) (2022)
- [A Game-Theoretic Approach to Recommendation Systems with Strategic Content Providers](https://mlsanity.com/paper/1806.00955/) (2018)
- [Supply-Side Equilibria in Recommender Systems](https://mlsanity.com/paper/2206.13489/) (2022)
- [Optimizing Long-term Social Welfare in Recommender Systems: A Constrained Matching Approach](https://mlsanity.com/paper/2008.00104/) (2020)
- [Recovering Bandits](https://mlsanity.com/paper/1910.14354/) (2019)
- [The Challenge of Understanding What Users Want: Inconsistent Preferences and Engagement Optimization](https://mlsanity.com/paper/2202.11776/) (2022)
- [Estimating and Penalizing Induced Preference Shifts in Recommender Systems](https://mlsanity.com/paper/2204.11966/) (2022)
- [How Bad is Top-K Recommendation under Competing Content Creators?](https://mlsanity.com/paper/2302.01971/) (2023)
- [Stochastic Bandits with Delay-Dependent Payoffs](https://mlsanity.com/paper/1910.02757/) (2019)
- [Do Offline Metrics Predict Online Performance in Recommender Systems?](https://mlsanity.com/paper/2011.07931/) (2020)
- [Preference Dynamics Under Personalized Recommendations](https://mlsanity.com/paper/2205.13026/) (2022)

## Built on by (most cited first, 11 of 11)

- [Anatomy of a Machine Learning Ecosystem: 2 Million Models on Hugging Face](https://mlsanity.com/paper/2508.06811/) (2025)
- [AI-Assisted Decision Making with Human Learning](https://mlsanity.com/paper/2502.13062/) (2025)
- [Decision-Dependent Stochastic Optimization: The Role of Distribution Dynamics](https://mlsanity.com/paper/2503.07324/) (2025)
- [Strategic Linear Contextual Bandits](https://mlsanity.com/paper/2406.00551/) (2024)
- [The Backfiring Effect of Weak AI Safety Regulation](https://mlsanity.com/paper/2503.20848/) (2025)
- [Collective dynamics of strategic classification](https://mlsanity.com/paper/2508.09340/) (2025)
- [Collaborating with GenAI: Incentives and Replacements](https://mlsanity.com/paper/2508.20213/) (2025)
- [The Burden of Interactive Alignment with Inconsistent Preferences](https://mlsanity.com/paper/2510.16368/) (2025)
- [Evolutionary Prediction Games](https://mlsanity.com/paper/2503.03401/) (2025)
- [Data Sharing with a Generative AI Competitor](https://mlsanity.com/paper/2505.12386/) (2025)
- [Socially-Aware Recommender Systems Mitigate Opinion Clusterization](https://mlsanity.com/paper/2601.02412/) (2026)

## Abstract (arXiv)

As AI systems enter into a growing number of societal domains, these systems increasingly shape and are shaped by user preferences, opinions, and behaviors. However, the design of AI systems rarely accounts for how AI and users shape one another. In this position paper, we argue for the development of formal interaction models which mathematically specify how AI and users shape one another. Formal interaction models can be leveraged to (1) specify interactions for implementation, (2) monitor interactions through empirical analysis, (3) anticipate societal impacts via counterfactual analysis, and (4) control societal impacts via interventions. The design space of formal interaction models is vast, and model design requires careful consideration of factors such as style, granularity, mathematical complexity, and measurability. Using content recommender systems as a case study, we critically examine the nascent literature of formal interaction models with respect to these use-cases and design axes. More broadly, we call for the community to leverage formal interaction models when designing, evaluating, or auditing any AI system which interacts with users.