mlsanityData to 14 Oct 2026

Generalizing Behavior via Inverse Reinforcement Learning with Closed-Form Reward Centroids

arXiv 2509.12010 ↗First posted 15 Sep 2025cs.LG cs.AIcited by 1

On the map: Offline RL and imitation learning from data → Inverse reinforcement learning

also in 1 more
Abstract (arXiv)

We study the problem of generalizing an expert agent's behavior, provided through demonstrations, to new environments and/or additional constraints. Inverse Reinforcement Learning (IRL) offers a promising solution by seeking to recover the expert's underlying reward function, which, if used for planning in the new settings, would reproduce the desired behavior. However, IRL is inherently ill-posed: multiple reward functions, forming the so-called feasible set, can explain the same observed behavior. Since these rewards may induce different policies in the new setting, in the absence of additional information, a decision criterion is needed to select which policy to deploy. In this paper, we propose a novel, principled criterion that selects the "average" policy among those induced by the rewards in a certain bounded subset of the feasible set. Remarkably, we show that this policy can be obtained by planning with the reward centroid of that subset, for which we derive a closed-form expression. We then present a provably efficient algorithm for estimating this centroid using an offline dataset of expert demonstrations only. Finally, we conduct numerical simulations that illustrate the relationship between the expert's behavior and the behavior produced by our method.

Key facts. Cited by 1 paper in the map (data through 14 Oct 2026). Placed in Inverse reinforcement learning, one of 13 directions in Offline RL and imitation learning from data. Builds on 25 of its 55 references that are in the map. First posted on arXiv 15 Sep 2025.

Builds on

25 of 55 references.

Closest ancestors · 12

Same direction, newest first

  1. 2024Walking the Values in Bayesian Inverse Reinforcement Learning15 Jul
  2. 2024Towards the Transferability of Rewards Recovered via Regularized Inverse Reinforcement Learning3 Jun
  3. 2024Offline Inverse RL: New Solution Concepts and Provably Efficient Algorithms23 Feb
  4. 2024Inverse Reinforcement Learning with Sub-optimal Experts8 Jan
  5. 2023Identifiability and Generalizability in Constrained Inverse Reinforcement Learning1 Jun
  6. 2023Towards Theoretical Understanding of Inverse Reinforcement Learning25 Apr
  7. 2022Misspecification in Inverse Reinforcement Learning6 Dec
  8. 2022Identifiability and generalizability from multiple experts in Inverse Reinforcement Learning22 Sep
  9. 2021Identifiability in inverse reinforcement learning7 Jun
  10. 2020f-IRL: Inverse Reinforcement Learning via State Marginal Matching9 Nov
  11. 2020Robust Inverse Reinforcement Learning under Transition Dynamics Mismatch2 Jul
  12. 2018A Survey of Inverse Reinforcement Learning: Challenges, Methods and Progress18 Jun

Other references · 13

  1. 2021Convex optimizationcited by 2,515
  2. 2016Generative Adversarial Imitation Learningcited by 1,597
  3. 2019Solving Rubik's Cube with a Robot Handcited by 734
  4. 2017Reinforcement Learning with Deep Energy-Based Policiescited by 669
  5. 2017Learning Robust Rewards with Adversarial Inverse Reinforcement Learningcited by 490
  6. 2018An Algorithmic Perspective on Imitation Learningcited by 459
  7. 2016Guided Cost Learning: Deep Inverse Optimal Control via Policy Optimizationcited by 406
  8. 2021IQ-Learn: Inverse soft-Q Learning for Imitationcited by 193
  9. 2021Policy Finetuning: Bridging Sample-Efficient Offline and Online Reinforcement Learningcited by 158
  10. 2018Sim-to-Real Reinforcement Learning for Deformable Object Manipulationcited by 131
  11. 2022Invariance in Policy Optimisation and Partial Identifiability in Reward Learningcited by 46
  12. 2019Random Expert Distillation: Imitation Learning via Expert Policy Support Estimationcited by 42
1 more
  1. 2016Towards Resolving Unidentifiability in Inverse Reinforcement Learningcited by 18

Built on by

Cited by year (2026 to 14 Oct)

Citing papers per year: 2026: 1.

1 paper.

  1. 2026Inverse Reinforcement Learning without an Optimal Demonstrator: A Feasible Reward Set Approachcited by 0

Similar, not cited

Close in wording, no citation either way

  1. 2023BC-IRL: Learning Generalizable Reward Functions from Demonstrations
  2. 2021Versatile Inverse Reinforcement Learning via Cumulative Rewards
  3. 2020Inverse Reinforcement Learning from a Gradient-based Learner
  4. 2023Is Inverse Reinforcement Learning Harder than Standard Reinforcement Learning? A Theoretical Perspective
  5. 2024Inverse Reinforcement Learning with Multiple Planning Horizons
  6. 2025Reward Compatibility: A Framework for Inverse RL
  7. 2020Inverse Reinforcement Learning via Matching of Optimality Profiles
  8. 2025Inverse Reinforcement Learning via Convex Optimization