Unintended Selection: Persistent Qualification Rate Disparities and Interventions

Realistically -- and equitably -- modeling the dynamics of group-level disparities in machine learning remains an open problem. In particular, we desire models that do not suppose inherent differences between artificial groups of people -- but rather endogenize disparities by appeal to unequal initial conditions of insular subpopulations. In this paper, agents each have a real-valued feature X (e.g., credit score) informed by a "true" binary label Y representing qualification (e.g., for a loan). Each agent alternately (1) receives a binary classification label hatY (e.g., loan approval) from a Bayes-optimal machine learning classifier observing X and (2) may update their qualification Y by imitating successful strategies (e.g., seek a raise) within an isolated group G of agents to which they belong. We consider the disparity of qualification rates Pr(Y=1) between different groups and how this disparity changes subject to a sequence of Bayes-optimal classifiers repeatedly retrained on the global population. We model the evolving qualification rates of each subpopulation (group) using the replicator equation, which derives from a class of imitation processes. We show that differences in qualification rates between subpopulations can persist indefinitely for a set of non-trivial equilibrium states due to uniformed classifier deployments, even when groups are identical in all aspects except initial qualification densities. We next simulate the effects of commonly proposed fairness interventions on this dynamical system along with a new feedback control mechanism capable of permanently eliminating group-level qualification rate disparities. We conclude by discussing the limitations of our model and findings and by outlining potential future work.