# What Hides behind Unfairness? Exploring Dynamics Fairness in Reinforcement Learning

arXiv: https://arxiv.org/abs/2404.10942 · HTML: https://mlsanity.com/paper/2404.10942/

## Key facts

- Cited by 3 papers in the map (data through 14 Oct 2026).
- Placed in RL for games and combinatorial optimization, one of 15 directions in Deep reinforcement learning methods.
- Builds on 27 of its 44 references that are in the map.
- First posted on arXiv 16 Apr 2024.

## Placed in

- Deep reinforcement learning methods / [RL for games and combinatorial optimization](https://mlsanity.com/deeprl/rl-for-games-and-combinatorial-optimization/)

## Builds on (most cited first, 27 of 27)

- [Equality of Opportunity in Supervised Learning](https://mlsanity.com/paper/1610.02413/) (2016)
- [A Survey on Bias and Fairness in Machine Learning](https://mlsanity.com/paper/1908.09635/) (2019)
- [Counterfactual Fairness](https://mlsanity.com/paper/1703.06856/) (2017)
- [Deep Reinforcement Learning in a Handful of Trials using Probabilistic Dynamics Models](https://mlsanity.com/paper/1805.12114/) (2018)
- [Direct and Indirect Effects](https://mlsanity.com/paper/1301.2300/) (2013)
- [Delayed Impact of Fair Machine Learning](https://mlsanity.com/paper/1803.04383/) (2018)
- [Algorithmic Fairness](https://mlsanity.com/paper/2001.09784/) (2020)
- [Context-aware Dynamics Model for Generalization in Model-Based Reinforcement Learning](https://mlsanity.com/paper/2005.06800/) (2020)
- [Woulda, Coulda, Shoulda: Counterfactually-Guided Policy Search](https://mlsanity.com/paper/1811.06272/) (2018)
- [OmniSafe: An Infrastructure for Accelerating Safe Reinforcement Learning Research](https://mlsanity.com/paper/2305.09304/) (2023)
- [Achieving Fairness in the Stochastic Multi-armed Bandit Problem](https://mlsanity.com/paper/1907.10516/) (2019)
- [A causal framework for discovering and removing direct and indirect discrimination](https://mlsanity.com/paper/1611.07509/) (2016)
- [PC-Fairness: A Unified Framework for Measuring Causality-based Fairness](https://mlsanity.com/paper/1910.12586/) (2019)
- [How Do Fair Decisions Fare in Long-term Qualification?](https://mlsanity.com/paper/2010.11300/) (2020)
- [Algorithms for Fairness in Sequential Decision Making](https://mlsanity.com/paper/1901.08568/) (2019)
- [Learning Optimal Fair Policies](https://mlsanity.com/paper/1809.02244/) (2018)
- [Causal Modeling for Fairness in Dynamical Systems](https://mlsanity.com/paper/1909.09141/) (2019)
- [Causal Fairness Analysis](https://mlsanity.com/paper/2207.11385/) (2022)
- [From Fair Decision Making to Social Equality](https://mlsanity.com/paper/1812.02952/) (2018)
- [Pareto Conditioned Networks](https://mlsanity.com/paper/2204.05036/) (2022)
- [Causal Reinforcement Learning: A Survey](https://mlsanity.com/paper/2307.01452/) (2023)
- [A Relational Intervention Approach for Unsupervised Dynamics Generalization in Model-Based Reinforcement Learning](https://mlsanity.com/paper/2206.04551/) (2022)
- [Policy Optimization with Advantage Regularization for Long-Term Fairness in Decision Systems](https://mlsanity.com/paper/2210.12546/) (2022)
- [Achieving User-Side Fairness in Contextual Bandits](https://mlsanity.com/paper/2010.12102/) (2020)
- [Causal Fairness for Outcome Control](https://mlsanity.com/paper/2306.05066/) (2023)
- [Tier Balancing: Towards Dynamic Fairness over Underlying Causal Factors](https://mlsanity.com/paper/2301.08987/) (2023)
- [RISE: Robust Individualized Decision Learning with Sensitive Variables](https://mlsanity.com/paper/2211.06569/) (2022)

## Built on by (most cited first, 3 of 3)

- [Fairness in Reinforcement Learning with Bisimulation Metrics](https://mlsanity.com/paper/2412.17123/) (2024)
- [MAFE: Enabling Equitable Algorithm Design in Multi-Agent Multi-Stage Decision-Making Systems](https://mlsanity.com/paper/2502.18534/) (2025)
- [Long-term Fairness with Selective Labels](https://mlsanity.com/paper/2605.22291/) (2026)

## Abstract (arXiv)

In sequential decision-making problems involving sensitive attributes like race and gender, reinforcement learning (RL) agents must carefully consider long-term fairness while maximizing returns. Recent works have proposed many different types of fairness notions, but how unfairness arises in RL problems remains unclear. In this paper, we address this gap in the literature by investigating the sources of inequality through a causal lens. We first analyse the causal relationships governing the data generation process and decompose the effect of sensitive attributes on long-term well-being into distinct components. We then introduce a novel notion called dynamics fairness, which explicitly captures the inequality stemming from environmental dynamics, distinguishing it from those induced by decision-making or inherited from the past. This notion requires evaluating the expected changes in the next state and the reward induced by changing the value of the sensitive attribute while holding everything else constant. To quantitatively evaluate this counterfactual concept, we derive identification formulas that allow us to obtain reliable estimations from data. Extensive experiments demonstrate the effectiveness of the proposed techniques in explaining, detecting, and reducing inequality in reinforcement learning. We publicly release code at https://github.com/familyld/InsightFair.