# Long-Term Fairness with Unknown Dynamics

arXiv: https://arxiv.org/abs/2304.09362 · HTML: https://mlsanity.com/paper/2304.09362/

## Key facts

- Cited by 27 papers in the map (data through 14 Oct 2026).
- Builds on 36 of its 60 references that are in the map.
- First posted on arXiv 19 Apr 2023.

## Builds on (most cited first, 30 of 36)

- [PyTorch: An Imperative Style, High-Performance Deep Learning Library](https://mlsanity.com/paper/1912.01703/) (2019)
- [Addressing Function Approximation Error in Actor-Critic Methods](https://mlsanity.com/paper/1802.09477/) (2018)
- [Equality of Opportunity in Supervised Learning](https://mlsanity.com/paper/1610.02413/) (2016)
- [Certifying and removing disparate impact](https://mlsanity.com/paper/1412.3756/) (2014)
- [Provably Efficient Reinforcement Learning with Linear Function Approximation](https://mlsanity.com/paper/1907.05388/) (2019)
- [Fairness Without Demographics in Repeated Loss Minimization](https://mlsanity.com/paper/1806.08010/) (2018)
- [Performative Prediction](https://mlsanity.com/paper/2002.06673/) (2020)
- [Modern Koopman Theory for Dynamical Systems](https://mlsanity.com/paper/2102.12086/) (2021)
- [Delayed Impact of Fair Machine Learning](https://mlsanity.com/paper/1803.04383/) (2018)
- [Fairness in Learning: Classic and Contextual Bandits](https://mlsanity.com/paper/1605.07139/) (2016)
- [Exploration-Exploitation in Constrained MDPs](https://mlsanity.com/paper/2003.02189/) (2020)
- [Provably Efficient Safe Exploration via Primal-Dual Policy Optimization](https://mlsanity.com/paper/2003.00534/) (2020)
- [CRPO: A New Approach for Safe Reinforcement Learning with Convergence Guarantee](https://mlsanity.com/paper/2011.05869/) (2020)
- [How Algorithmic Confounding in Recommendation Systems Increases Homogeneity and Decreases Utility](https://mlsanity.com/paper/1710.11214/) (2017)
- [Runaway Feedback Loops in Predictive Policing](https://mlsanity.com/paper/1706.09847/) (2017)
- [Fairness in Reinforcement Learning](https://mlsanity.com/paper/1611.03071/) (2016)
- [Learning Policies with Zero or Bounded Constraint Violation for Constrained MDPs](https://mlsanity.com/paper/2106.02684/) (2021)
- [The Disparate Effects of Strategic Manipulation](https://mlsanity.com/paper/1808.08646/) (2018)
- [How Do Fair Decisions Fare in Long-term Qualification?](https://mlsanity.com/paper/2010.11300/) (2020)
- [Achieving Zero Constraint Violation for Constrained Reinforcement Learning via Primal-Dual Approach](https://mlsanity.com/paper/2109.06332/) (2021)
- [On the Long-term Impact of Algorithmic Decision Policies: Effort Unfairness and Feature Segregation through Social Learning](https://mlsanity.com/paper/1903.01209/) (2019)
- [Constrained Upper Confidence Reinforcement Learning](https://mlsanity.com/paper/2001.09377/) (2020)
- [A Short-term Intervention for Long-term Fairness in the Labor Market](https://mlsanity.com/paper/1712.00064/) (2017)
- [Calibrated Fairness in Bandits](https://mlsanity.com/paper/1707.01875/) (2017)
- [The Disparate Equilibria of Algorithmic Decision Making when Individuals Invest Rationally](https://mlsanity.com/paper/1910.04123/) (2019)
- [From Fair Decision Making to Social Equality](https://mlsanity.com/paper/1812.02952/) (2018)
- [Constrained episodic reinforcement learning in concave-convex and knapsack settings](https://mlsanity.com/paper/2006.05051/) (2020)
- [A Sample-Efficient Algorithm for Episodic Finite-Horizon MDP with Constraints](https://mlsanity.com/paper/2009.11348/) (2020)
- [Provably Efficient Model-Free Constrained RL with Linear Function Approximation](https://mlsanity.com/paper/2206.11889/) (2022)
- [Fairness Transferability Subject to Bounded Distribution Shift](https://mlsanity.com/paper/2206.00129/) (2022)

## Built on by (most cited first, 27 of 27)

- [Trustworthy LLMs: a Survey and Guideline for Evaluating Large Language Models' Alignment](https://mlsanity.com/paper/2308.05374/) (2023)
- [Online Restless Multi-Armed Bandits with Long-Term Fairness Constraints](https://mlsanity.com/paper/2312.10303/) (2023)
- [Designing Long-term Group Fair Policies in Dynamical Systems](https://mlsanity.com/paper/2311.12447/) (2023)
- [The Importance of Human-Labeled Data in the Era of LLMs](https://mlsanity.com/paper/2306.14910/) (2023)
- [Algorithmic Decision-Making under Agents with Persistent Improvement](https://mlsanity.com/paper/2405.01807/) (2024)
- [Adapting Static Fairness to Sequential Decision-Making: Bias Mitigation Strategies towards Equal Long-term Benefit Rate](https://mlsanity.com/paper/2309.03426/) (2023)
- [Algorithmic Fairness in Performative Policy Learning: Escaping the Impossibility of Group Fairness](https://mlsanity.com/paper/2405.20447/) (2024)
- [Long-Term Fairness Inquiries and Pursuits in Machine Learning: A Survey of Notions, Methods, and Challenges](https://mlsanity.com/paper/2406.06736/) (2024)
- [Enhancing Group Fairness in Online Settings Using Oblique Decision Forests](https://mlsanity.com/paper/2310.11401/) (2023)
- [FairSense: Long-Term Fairness Analysis of ML-Enabled Systems](https://mlsanity.com/paper/2501.01665/) (2025)
- [Responsible Agentic AI Requires Explicit Provenance](https://mlsanity.com/paper/2605.17169/) (2026)
- [Long-term Fairness For Real-time Decision Making: A Constrained Online Optimization Approach](https://mlsanity.com/paper/2401.02552/) (2024)
- [A Causal Lens for Learning Long-term Fair Policies](https://mlsanity.com/paper/2506.11242/) (2025)
- [Underrepresentation, Label Bias, and Proxies: Towards Data Bias Profiles for the EU AI Act and Beyond](https://mlsanity.com/paper/2507.08866/) (2025)
- [Fairness under uncertainty in sequential decisions](https://mlsanity.com/paper/2604.21711/) (2026)
- [Fair Clustering: Critique, Caveats, and Future Directions](https://mlsanity.com/paper/2406.15960/) (2024)
- [Long-Term Fairness in Sequential Multi-Agent Selection with Positive Reinforcement](https://mlsanity.com/paper/2407.07350/) (2024)
- [Fairness in Reinforcement Learning with Bisimulation Metrics](https://mlsanity.com/paper/2412.17123/) (2024)
- [Fairness Incentives in Response to Unfair Dynamic Pricing](https://mlsanity.com/paper/2404.14620/) (2024)
- [MAFE: Enabling Equitable Algorithm Design in Multi-Agent Multi-Stage Decision-Making Systems](https://mlsanity.com/paper/2502.18534/) (2025)
- [Evolutionary Prediction Games](https://mlsanity.com/paper/2503.03401/) (2025)
- [Group Fairness in Multi-Task Reinforcement Learning](https://mlsanity.com/paper/2503.07817/) (2025)
- [How Predicted Links Influence Network Evolution: Disentangling Choice and Algorithmic Feedback in Dynamic Graphs](https://mlsanity.com/paper/2603.03945/) (2026)
- [Revisiting Fairness Impossibility with Endogenous Behavior](https://mlsanity.com/paper/2604.06378/) (2026)
- [Price of Fairness in Short-Term and Long-Term Algorithmic Selections](https://mlsanity.com/paper/2605.06227/) (2026)
- [Algorithmic Fairness Over Time: Advances & Prospects](https://mlsanity.com/paper/2610.15015/) (2026)
- [Bayesian Learning to Select Fair Classifiers](https://mlsanity.com/paper/2610.20514/) (2026)

## Abstract (arXiv)

While machine learning can myopically reinforce social inequalities, it may also be used to dynamically seek equitable outcomes. In this paper, we formalize long-term fairness in the context of online reinforcement learning. This formulation can accommodate dynamical control objectives, such as driving equity inherent in the state of a population, that cannot be incorporated into static formulations of fairness. We demonstrate that this framing allows an algorithm to adapt to unknown dynamics by sacrificing short-term incentives to drive a classifier-population system towards more desirable equilibria. For the proposed setting, we develop an algorithm that adapts recent work in online learning. We prove that this algorithm achieves simultaneous probabilistic bounds on cumulative loss and cumulative violations of fairness (as statistical regularities between demographic groups). We compare our proposed algorithm to the repeated retraining of myopic classifiers, as a baseline, and to a deep reinforcement learning algorithm that lacks safety guarantees. Our experiments model human populations according to evolutionary game theory and integrate real-world datasets.