# Understanding Reward Ambiguity Through Optimal Transport Theory in Inverse Reinforcement Learning

arXiv: https://arxiv.org/abs/2310.12055 · HTML: https://mlsanity.com/paper/2310.12055/

## Key facts

- Cited by 7 papers in the map (data through 14 Oct 2026).
- Placed in Inverse reinforcement learning, one of 13 directions in Offline RL and imitation learning from data.
- Builds on 5 of its 18 references that are in the map.
- First posted on arXiv 18 Oct 2023.

## Placed in

- Offline RL and imitation learning from data / [Inverse reinforcement learning](https://mlsanity.com/offlinerl/inverse-reinforcement-learning/)

## Builds on (most cited first, 5 of 5)

- [Sinkhorn Distances: Lightspeed Computation of Optimal Transportation Distances](https://mlsanity.com/paper/1306.0895/) (2013)
- [Concrete Problems in AI Safety](https://mlsanity.com/paper/1606.06565/) (2016)
- [Generative Adversarial Imitation Learning](https://mlsanity.com/paper/1606.03476/) (2016)
- [Computational Optimal Transport](https://mlsanity.com/paper/1803.00567/) (2018)
- [Risk-Aware Reinforcement Learning through Optimal Transport Theory](https://mlsanity.com/paper/2309.06239/) (2023)

## Built on by (most cited first, 7 of 7)

- [Optimal Transport-Assisted Risk-Sensitive Q-Learning](https://mlsanity.com/paper/2406.11774/) (2024)
- [Optimal Transport-Guided Safety in Temporal Difference Reinforcement Learning](https://mlsanity.com/paper/2502.16328/) (2025)
- [Wasserstein Barycenter Soft Actor-Critic](https://mlsanity.com/paper/2506.10167/) (2025)
- [Can Optimal Transport Improve Federated Inverse Reinforcement Learning?](https://mlsanity.com/paper/2601.00309/) (2026)
- [Federated Distributional Reinforcement Learning with Distributional Critic Regularization](https://mlsanity.com/paper/2603.17820/) (2026)
- [SSL: Sweet Spot Learning for Differentiated Guidance in Agentic Optimization](https://mlsanity.com/paper/2601.22491/) (2026)
- [Optimal Transport Meets Reinforcement Learning: A Survey](https://mlsanity.com/paper/2610.01413/) (2026)

## Abstract (arXiv)

In inverse reinforcement learning (IRL), the central objective is to infer underlying reward functions from observed expert behaviors in a way that not only explains the given data but also generalizes to unseen scenarios. This ensures robustness against reward ambiguity where multiple reward functions can equally explain the same expert behaviors. While significant efforts have been made in addressing this issue, current methods often face challenges with high-dimensional problems and lack a geometric foundation. This paper harnesses the optimal transport (OT) theory to provide a fresh perspective on these challenges. By utilizing the Wasserstein distance from OT, we establish a geometric framework that allows for quantifying reward ambiguity and identifying a central representation or centroid of reward functions. These insights pave the way for robust IRL methodologies anchored in geometric interpretations, offering a structured approach to tackle reward ambiguity in high-dimensional settings.