mlsanityData to 7 Oct 2026

Understanding Reward Ambiguity Through Optimal Transport Theory in Inverse Reinforcement Learning

arXiv 2310.12055 ↗First posted 18 Oct 2023cs.LG cs.SY eess.SYcited by 6

On the map: Offline RL and imitation learning from data → Inverse reinforcement learning

Abstract (arXiv)

In inverse reinforcement learning (IRL), the central objective is to infer underlying reward functions from observed expert behaviors in a way that not only explains the given data but also generalizes to unseen scenarios. This ensures robustness against reward ambiguity where multiple reward functions can equally explain the same expert behaviors. While significant efforts have been made in addressing this issue, current methods often face challenges with high-dimensional problems and lack a geometric foundation. This paper harnesses the optimal transport (OT) theory to provide a fresh perspective on these challenges. By utilizing the Wasserstein distance from OT, we establish a geometric framework that allows for quantifying reward ambiguity and identifying a central representation or centroid of reward functions. These insights pave the way for robust IRL methodologies anchored in geometric interpretations, offering a structured approach to tackle reward ambiguity in high-dimensional settings.

Key facts. Cited by 6 papers in the map (data through 7 Oct 2026). Placed in Inverse reinforcement learning, one of 13 directions in Offline RL and imitation learning from data. Builds on 5 of its 18 references that are in the map. First posted on arXiv 18 Oct 2023.

Builds on

5 of 18 references.

References · 5

  1. 2013Sinkhorn Distances: Lightspeed Computation of Optimal Transportation Distancescited by 2,277
  2. 2016Concrete Problems in AI Safetycited by 1,582
  3. 2016Generative Adversarial Imitation Learningcited by 1,576
  4. 2018Computational Optimal Transportcited by 765
  5. 2023Risk-Aware Reinforcement Learning through Optimal Transport Theorycited by 7

Built on by

Cited by year (2026 to 7 Oct)

Citing papers per year: 2024: 1; 2025: 2; 2026: 3.

6 papers.

  1. 2024Optimal Transport-Assisted Risk-Sensitive Q-Learningcited by 3
  2. 2025Wasserstein Barycenter Soft Actor-Criticcited by 1
  3. 2026Can Optimal Transport Improve Federated Inverse Reinforcement Learning?cited by 1
  4. 2026Federated Distributional Reinforcement Learning with Distributional Critic Regularizationcited by 1
  5. 2025Optimal Transport-Guided Safety in Temporal Difference Reinforcement Learningcited by 0
  6. 2026SSL: Sweet Spot Learning for Differentiated Guidance in Agentic Optimizationcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2025Reward Compatibility: A Framework for Inverse RL
  2. 2026Inverse Reinforcement Learning without an Optimal Demonstrator: A Feasible Reward Set Approach
  3. 2020Efficient Exploration of Reward Functions in Inverse Reinforcement Learning via Bayesian Optimization
  4. 2023Towards Theoretical Understanding of Inverse Reinforcement Learning
  5. 2024Towards the Transferability of Rewards Recovered via Regularized Inverse Reinforcement Learning
  6. 2023A Novel Variational Lower Bound for Inverse Reinforcement Learning
  7. 2018A Survey of Inverse Reinforcement Learning: Challenges, Methods and Progress
  8. 2019Option Compatible Reward Inverse Reinforcement Learning