# Reward Transfer from Inverse Reinforcement Learning: A Coupled Minimax Approach

arXiv: https://arxiv.org/abs/2605.27834 · HTML: https://mlsanity.com/paper/2605.27834/

## Key facts

- Cited by 1 papers in the map (data through 14 Oct 2026).
- Placed in Inverse reinforcement learning, one of 13 directions in Offline RL and imitation learning from data.
- Builds on 4 of its 43 references that are in the map.
- First posted on arXiv 27 May 2026.

## Placed in

- Offline RL and imitation learning from data / [Inverse reinforcement learning](https://mlsanity.com/offlinerl/inverse-reinforcement-learning/)

## Builds on (most cited first, 4 of 4)

- [Reinforcement Learning and Control as Probabilistic Inference: Tutorial and Review](https://mlsanity.com/paper/1805.00909/) (2018)
- [On the Properties of the Softmax Function with Application in Game Theory and Reinforcement Learning](https://mlsanity.com/paper/1704.00805/) (2017)
- [Efficient Inference for Inverse Reinforcement Learning and Dynamic Discrete Choice Models](https://mlsanity.com/paper/2512.24407/) (2025)
- [Debiased Inference for High-Dimensional Regression Models Based on Profile M-Estimation](https://mlsanity.com/paper/2512.12003/) (2025)

## Built on by (most cited first, 1 of 1)

- [PHIRL: Aligning Learned Rewards with Task Progress for Inverse Reinforcement Learning](https://mlsanity.com/paper/2609.31855/) (2026)

## Abstract (arXiv)

We study the transfer of rewards learned using inverse reinforcement learning from expert demonstrations in one environment to reinforcement learning in a new, different environment. This arises naturally when demonstrations are collected in a controlled environment. We formulate the problem as a joint system of Bellman equations across the source and target environments and develop minimax estimators for the target soft-q-function. Whereas a sequential solution approach first estimates the source reward and then plugs it into the target control problem, a coupled approach solves the source and target system of equations jointly. We show that, in contrast to the sequential approach, the coupled approach removes the first-order influence of source Bellman residual error. We characterize the local behavior of each approach, develop finite-sample soft-q-function error bounds, and prove regret guarantees for the resulting soft-control policy. An empirical investigation using a sepsis simulator validates the theoretical comparison.