# Inference-Time Policy Alignment for Fair Reinforcement Learning

arXiv: https://arxiv.org/abs/2608.00175 · HTML: https://mlsanity.com/paper/2608.00175/

## Key facts

- No citing papers in the map yet (data through 14 Oct 2026).
- Builds on 44 of its 70 references that are in the map.
- First posted on arXiv 31 Jul 2026.

## Builds on (most cited first, 30 of 44)

- [Training language models to follow instructions with human feedback](https://mlsanity.com/paper/2203.02155/) (2022)
- [Proximal Policy Optimization Algorithms](https://mlsanity.com/paper/1707.06347/) (2017)
- [Direct Preference Optimization: Your Language Model is Secretly a Reward Model](https://mlsanity.com/paper/2305.18290/) (2023)
- [Deep reinforcement learning from human preferences](https://mlsanity.com/paper/1706.03741/) (2017)
- [Trust Region Policy Optimization](https://mlsanity.com/paper/1502.05477/) (2015)
- [Asynchronous Methods for Deep Reinforcement Learning](https://mlsanity.com/paper/1602.01783/) (2016)
- [Domain Randomization for Transferring Deep Neural Networks from Simulation to the Real World](https://mlsanity.com/paper/1703.06907/) (2017)
- [Inference-Time Intervention: Eliciting Truthful Answers from a Language Model](https://mlsanity.com/paper/2306.03341/) (2023)
- [Steering Language Models With Activation Engineering](https://mlsanity.com/paper/2308.10248/) (2023)
- [Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning Algorithm](https://mlsanity.com/paper/1712.01815/) (2017)
- [A Reductions Approach to Fair Classification](https://mlsanity.com/paper/1803.02453/) (2018)
- [A Survey of Reinforcement Learning from Human Feedback](https://mlsanity.com/paper/2312.14925/) (2023)
- [A Practical Guide to Multi-Objective Reinforcement Learning and Planning](https://mlsanity.com/paper/2103.09568/) (2021)
- [Delayed Impact of Fair Machine Learning](https://mlsanity.com/paper/1803.04383/) (2018)
- [Open Problems in Mechanistic Interpretability](https://mlsanity.com/paper/2501.16496/) (2025)
- [LLM Post-Training: A Deep Dive into Reasoning Large Language Models](https://mlsanity.com/paper/2502.21321/) (2025)
- [A Unified Approach to Quantifying Algorithmic Unfairness: Measuring Individual & Group Unfairness via Inequality Indices](https://mlsanity.com/paper/1807.00787/) (2018)
- [Fairness in Reinforcement Learning](https://mlsanity.com/paper/1611.03071/) (2016)
- [Decoding-Time Language Model Alignment with Multiple Objectives](https://mlsanity.com/paper/2406.18853/) (2024)
- [From Parity to Preference-based Notions of Fairness in Classification](https://mlsanity.com/paper/1707.00010/) (2017)
- [Learning Fair Policies in Multiobjective (Deep) Reinforcement Learning with Average and Discounted Rewards](https://mlsanity.com/paper/2008.07773/) (2020)
- [GenARM: Reward Guided Generation with Autoregressive Reward Model for Test-time Alignment](https://mlsanity.com/paper/2410.08193/) (2024)
- [Fairness Behind a Veil of Ignorance: A Welfare Analysis for Automated Decision Making](https://mlsanity.com/paper/1806.04959/) (2018)
- [Learning Fairness in Multi-Agent Systems](https://mlsanity.com/paper/1910.14472/) (2019)
- [Learning Fair Policies in Decentralized Cooperative Multi-Agent Reinforcement Learning](https://mlsanity.com/paper/2012.09421/) (2020)
- [Multi-property Steering of Large Language Models with Dynamic Activation Composition](https://mlsanity.com/paper/2406.17563/) (2024)
- [Fairness in Learning-Based Sequential Decision Algorithms: A Survey](https://mlsanity.com/paper/2001.04861/) (2020)
- [InfAlign: Inference-aware language model alignment](https://mlsanity.com/paper/2412.19792/) (2024)
- [Inference-Time Policy Adapters (IPA): Tailoring Extreme-Scale LMs without Fine-tuning](https://mlsanity.com/paper/2305.15065/) (2023)
- [Multi-objective Bandits: Optimizing the Generalized Gini Index](https://mlsanity.com/paper/1706.04933/) (2017)

## Abstract (arXiv)

Deep reinforcement learning (RL) agents achieve strong performance by optimizing scalar reward functions. However, once deployed, the policies of these RL agents are often rigid and costly to adapt to new performance criteria. For instance, an agent trained to maximize expected cumulative reward may not accommodate previously unknown stakeholder preferences. Existing approaches to achieve fairness, a type of preference, in RL typically assume that such preferences are known a priori and require complete retraining of the policy under a fairness-oriented metric. Inspired by inference-time alignment in large language models, we investigate the problem of steering a pretrained RL policy toward welfare-based fairness objectives at inference time without updating the base policy's parameters. We formalize inference-time fairness alignment as a policy shaping problem and propose a multiplicative policy shaping framework that adjusts action probabilities using action-dependent welfare scores, thus requiring no modification to the base policy. Our framework is general and compatible with any deep RL agent. Through extensive experiments across multiple domains, we demonstrate that inference-time policy shaping substantially improves welfare-based fairness objectives while preserving core task performance.