# A Distribution Mapping Approach to Counterfactually Fair Reinforcement Learning

arXiv: https://arxiv.org/abs/2608.08743 · HTML: https://mlsanity.com/paper/2608.08743/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Placed in RL for games and combinatorial optimization, one of 15 directions in Deep reinforcement learning methods.
- Builds on 16 of its 32 references that are in the map.
- First posted on arXiv 9 Aug 2026.

## Placed in

- Deep reinforcement learning methods / [RL for games and combinatorial optimization](https://mlsanity.com/deeprl/rl-for-games-and-combinatorial-optimization/)

## Builds on (most cited first, 16 of 16)

- [Adam: A Method for Stochastic Optimization](https://mlsanity.com/paper/1412.6980/) (2014)
- [Counterfactual Fairness](https://mlsanity.com/paper/1703.06856/) (2017)
- [Information-Theoretic Considerations in Batch Reinforcement Learning](https://mlsanity.com/paper/1905.00360/) (2019)
- [Batch Policy Learning under Constraints](https://mlsanity.com/paper/1903.08738/) (2019)
- [Contextual Markov Decision Processes](https://mlsanity.com/paper/1502.02259/) (2015)
- [Delayed Impact of Fair Machine Learning](https://mlsanity.com/paper/1803.04383/) (2018)
- [Causal Modeling for Fairness in Dynamical Systems](https://mlsanity.com/paper/1909.09141/) (2019)
- [Fast Rates for the Regret of Offline Reinforcement Learning](https://mlsanity.com/paper/2102.00479/) (2021)
- [Quantile regression with deep ReLU Networks: Estimators and minimax rates](https://mlsanity.com/paper/2010.08236/) (2020)
- [Counterfactual Fairness with Partially Known Causal Graph](https://mlsanity.com/paper/2205.13972/) (2022)
- [Achieving Counterfactual Fairness for Causal Bandit](https://mlsanity.com/paper/2109.10458/) (2021)
- [Counterfactual fairness: removing direct effects through regularization](https://mlsanity.com/paper/2002.10774/) (2020)
- [A Complete Characterization of Linear Estimators for Offline Policy Evaluation](https://mlsanity.com/paper/2203.04236/) (2022)
- [Evaluating Dynamic Conditional Quantile Treatment Effects with Applications in Ridesharing](https://mlsanity.com/paper/2305.10187/) (2023)
- [Counterfactually Fair Reinforcement Learning via Sequential Data Preprocessing](https://mlsanity.com/paper/2501.06366/) (2025)
- [Double Fairness Policy Learning: Integrating Action Fairness and Outcome Fairness in Decision-making](https://mlsanity.com/paper/2601.19186/) (2026)

## Abstract (arXiv)

Reinforcement learning (RL) seeks to optimize sequential decisions to maximize population-level benefits over time. However, when deployed in high-stakes settings such as healthcare, RL decisions might systematically restrict some subpopulation's access to valuable services in a manner contrary to the values and goals of stakeholders. Counterfactual fairness (CF) offers a promising framework to address this problem based on causal reasoning. This paper develops a data preprocessing algorithm that, when used in tandem with policy learning, enables CF in RL. Our algorithm relies on a novel quantile distribution mapping method for sequentially estimating the counterfactual states and rewards in the data preprocessing step, subsuming common additivity assumptions used for counterfactual prediction as a special case. We theoretically prove that the per-step level of counterfactual unfairness and infinite-horizon suboptimality gap can be bounded under mild regularity conditions. We also empirically test our algorithm in numerical experiments as well as in application to a real-world interventional digital health dataset.