# Unintended Selection: Persistent Qualification Rate Disparities and Interventions arXiv: https://arxiv.org/abs/2111.01201 ยท HTML: https://mlsanity.com/paper/2111.01201/ ## Key facts - Cited by 18 papers in the map (data through 14 Oct 2026). - Builds on 22 of its 43 references that are in the map. - First posted on arXiv 1 Nov 2021. ## Builds on (most cited first, 22 of 22) - [Convex optimization](https://mlsanity.com/paper/2106.01946/) (2021) - [Equality of Opportunity in Supervised Learning](https://mlsanity.com/paper/1610.02413/) (2016) - [Counterfactual Fairness](https://mlsanity.com/paper/1703.06856/) (2017) - [Certifying and removing disparate impact](https://mlsanity.com/paper/1412.3756/) (2014) - [Fair prediction with disparate impact: A study of bias in recidivism prediction instruments](https://mlsanity.com/paper/1610.07524/) (2016) - [Inherent Trade-Offs in the Fair Determination of Risk Scores](https://mlsanity.com/paper/1609.05807/) (2016) - [Fairness Beyond Disparate Treatment & Disparate Impact: Learning Classification without Disparate Mistreatment](https://mlsanity.com/paper/1610.08452/) (2016) - [Delayed Impact of Fair Machine Learning](https://mlsanity.com/paper/1803.04383/) (2018) - [Fairness in Learning: Classic and Contextual Bandits](https://mlsanity.com/paper/1605.07139/) (2016) - [How Algorithmic Confounding in Recommendation Systems Increases Homogeneity and Decreases Utility](https://mlsanity.com/paper/1710.11214/) (2017) - [Runaway Feedback Loops in Predictive Policing](https://mlsanity.com/paper/1706.09847/) (2017) - [Fairness in Reinforcement Learning](https://mlsanity.com/paper/1611.03071/) (2016) - [The Disparate Effects of Strategic Manipulation](https://mlsanity.com/paper/1808.08646/) (2018) - [From Parity to Preference-based Notions of Fairness in Classification](https://mlsanity.com/paper/1707.00010/) (2017) - [How Do Fair Decisions Fare in Long-term Qualification?](https://mlsanity.com/paper/2010.11300/) (2020) - [On the Long-term Impact of Algorithmic Decision Policies: Effort Unfairness and Feature Segregation through Social Learning](https://mlsanity.com/paper/1903.01209/) (2019) - [A Short-term Intervention for Long-term Fairness in the Labor Market](https://mlsanity.com/paper/1712.00064/) (2017) - [The Disparate Equilibria of Algorithmic Decision Making when Individuals Invest Rationally](https://mlsanity.com/paper/1910.04123/) (2019) - [Group Retention when Using Machine Learning in Sequential Decision Making: the Interplay between User Dynamics and Fairness](https://mlsanity.com/paper/1905.00569/) (2019) - [From Fair Decision Making to Social Equality](https://mlsanity.com/paper/1812.02952/) (2018) - [Bandit Learning with Delayed Impact of Actions](https://mlsanity.com/paper/2002.10316/) (2020) - [Dynamic Modeling and Equilibria in Fair Decision Making](https://mlsanity.com/paper/1911.06837/) (2019) ## Built on by (most cited first, 18 of 18) - [Fairness Transferability Subject to Bounded Distribution Shift](https://mlsanity.com/paper/2206.00129/) (2022) - [Long-Term Fairness with Unknown Dynamics](https://mlsanity.com/paper/2304.09362/) (2023) - [What-is and How-to for Fairness in Machine Learning: A Survey, Reflection, and Perspective](https://mlsanity.com/paper/2206.04101/) (2022) - [Performative Federated Learning: A Solution to Model-Dependent and Heterogeneous Distribution Shifts](https://mlsanity.com/paper/2305.05090/) (2023) - [Non-linear Welfare-Aware Strategic Learning](https://mlsanity.com/paper/2405.01810/) (2024) - [Certifying Some Distributional Fairness with Subpopulation Decomposition](https://mlsanity.com/paper/2205.15494/) (2022) - [Addressing Polarization and Unfairness in Performative Prediction](https://mlsanity.com/paper/2406.16756/) (2024) - [Designing Long-term Group Fair Policies in Dynamical Systems](https://mlsanity.com/paper/2311.12447/) (2023) - [Automating Data Annotation under Strategic Human Agents: Risks and Potential Solutions](https://mlsanity.com/paper/2405.08027/) (2024) - [The Importance of Human-Labeled Data in the Era of LLMs](https://mlsanity.com/paper/2306.14910/) (2023) - [Learning under Imitative Strategic Behavior with Unforeseeable Outcomes](https://mlsanity.com/paper/2405.01797/) (2024) - [Algorithmic Decision-Making under Agents with Persistent Improvement](https://mlsanity.com/paper/2405.01807/) (2024) - [How Strategic Agents Respond: Comparing Analytical Models with LLM-Generated Responses in Strategic Classification](https://mlsanity.com/paper/2501.16355/) (2025) - [Generalization Error Bounds for Learning under Censored Feedback](https://mlsanity.com/paper/2404.09247/) (2024) - [Friends in Unexpected Places: Enhancing Local Fairness in Federated Learning through Clustering](https://mlsanity.com/paper/2407.19331/) (2024) - [Revealing Positive and Negative Role Models to Help People Make Good Decisions](https://mlsanity.com/paper/2603.02495/) (2026) - [Evolutionary Prediction Games](https://mlsanity.com/paper/2503.03401/) (2025) - [Adaptive Bounded Exploration and Intermediate Actions for Data Debiasing](https://mlsanity.com/paper/2504.08151/) (2025) ## Abstract (arXiv) Realistically -- and equitably -- modeling the dynamics of group-level disparities in machine learning remains an open problem. In particular, we desire models that do not suppose inherent differences between artificial groups of people -- but rather endogenize disparities by appeal to unequal initial conditions of insular subpopulations. In this paper, agents each have a real-valued feature X (e.g., credit score) informed by a "true" binary label Y representing qualification (e.g., for a loan). Each agent alternately (1) receives a binary classification label hatY (e.g., loan approval) from a Bayes-optimal machine learning classifier observing X and (2) may update their qualification Y by imitating successful strategies (e.g., seek a raise) within an isolated group G of agents to which they belong. We consider the disparity of qualification rates Pr(Y=1) between different groups and how this disparity changes subject to a sequence of Bayes-optimal classifiers repeatedly retrained on the global population. We model the evolving qualification rates of each subpopulation (group) using the replicator equation, which derives from a class of imitation processes. We show that differences in qualification rates between subpopulations can persist indefinitely for a set of non-trivial equilibrium states due to uniformed classifier deployments, even when groups are identical in all aspects except initial qualification densities. We next simulate the effects of commonly proposed fairness interventions on this dynamical system along with a new feedback control mechanism capable of permanently eliminating group-level qualification rate disparities. We conclude by discussing the limitations of our model and findings and by outlining potential future work.